Skip to content

DataOps and Data Engineering

Operational principles for an Analytics Engineering consultancy β€” how we monitor, iterate, and keep data products running reliably. Grounded in the DataOps Manifesto and dbt best practices.

185|## 4. DataOps / Data Engineering Principles 186| 187|> "Data teams need the same engineering rigour that software teams adopted 20 years ago." β€” DataOps Manifesto 188| 189|DataOps applies software engineering principles to data pipelines. For an AE consultancy, this is the operational backbone that turns one-off models into production-grade data products. 190| 191|### 4.1 Version Control for Data 192| 193|Principle: All data transformation logic, configurations, and documentation live in version-controlled repositories. 194| 195|- Git is the system of record: Every SQL model, macro, test, and YAML configuration file is version-controlled. If it's not in Git, it doesn't exist in production. 196|- Branching strategy: Feature branches for model changes, main for production. Never push directly to main. Use pull requests for review, discussion, and approval. 197|- Data isn't version-controlled (but logic is): We version the logic that produces data, not the data itself. This distinction is fundamental β€” it's what makes dbt's approach work. 198| 199|### 4.2 CI/CD for Pipelines 200| 201|Principle: Automated testing and deployment ensure that only validated changes reach production. 202| 203|- CI pipeline (on every PR): 204| 1. dbt compile β€” syntax validation 205| 2. dbt test β€” against a staging/dev database 206| 3. dbt run β€” against a dev schema (optional for speed) 207| 4. Custom checks: breaking change detection, contract validation 208|- CD pipeline (on merge to main): 209| 1. dbt run β€” production execution 210| 2. dbt test β€” production validation 211| 3. dbt source freshness β€” SLA check 212| 4. Notification on failure (Slack, PagerDuty) 213|- Environment isolation: Dev, staging, and production environments must be fully isolated. Never test in production. Never mix dev schemas with prod tables. 214| 215|### 4.3 Testing Data Transformations 216| 217|Principle: Data transformations are tested with the same discipline as application code. 218| 219|- Four levels of dbt tests: 220| 1. Schema tests (unique, not_null, relationships, accepted_values) β€” basic correctness. 221| 2. Custom generic tests β€” reusable business rules (e.g., positive_amount, valid_date_range). 222| 3. Singular tests β€” ad-hoc SQL assertions for complex business logic. 223| 4. Reconciliation tests β€” compare mart aggregates to source totals. 224|- Test coverage > test count: 100 tests on 5 critical models is better than 5 tests on 100 marginal models. Prioritise tests on models that feed dashboards, financial reports, and operational decisions. 225|- Test in production: CI catches structural errors, but production data surfaces edge cases. Run dbt test in production after every dbt run. 226| 227|### 4.4 Observability 228| 229|Principle: Pipelines must be monitorable, measurable, and alertable β€” you must know when things break before your stakeholders do. 230| 231|- Three pillars of data observability: 232| - Volume: Row counts, record volumes, unexpected spikes or drops. 233| - Freshness: Is data arriving when expected? Source freshness and model freshness. 234| - Schema: Did columns change? Types shift? Tables appear or disappear? 235|- Instrument everything: Log dbt run results, test outcomes, model durations, and row counts. Push these to a monitoring layer (Datadog, Monte Carlo, Elementary, or custom dashboards). 236|- Alert on anomalies, not thresholds alone: Threshold alerts ("row count < 1000") are brittle. Anomaly detection ("row count deviates >3Οƒ from 7-day rolling average") is more robust. 237|- SLA breaches are incidents: Treat a missed freshness SLA with the same rigour as an application outage. Define escalation paths and resolution protocols. 238| 239|### 4.5 SLAs and SLOs 240| 241|Principle: Data products have explicit service-level agreements and objectives, just like software services. 242| 243|- SLA (Service Level Agreement): The commitment to consumers β€” "financial data refreshed by 07:00 UTC." Breaches trigger a defined response. 244|- SLO (Service Level Objective): The internal target β€” "99% of dbt runs complete within 45 minutes." This is what your team aims for. 245|- SLI (Service Level Indicator): The metric that measures the SLO β€” "actual median run time of the financial mart models." This is the data point. 246|- Define SLAs per data product: Different consumers need different guarantees. The CEO's morning dashboard has a higher SLA than a weekly marketing report. 247|- Make SLAs visible:* Publish SLA dashboards. Data consumers should see freshness status and SLA adherence without asking engineering. 248| 249|---

480|## Appendix B: Quick Reference β€” dbt Project Non-Negotiables 481| 482|Every dbt project delivered by this consultancy should meet these minimum standards: 483| 484|- [ ] Three-tier model structure (staging β†’ intermediate β†’ marts) 485|- [ ] Unique and not_null tests on every model's primary key 486|- [ ] Relationships tests on every foreign key 487|- [ ] Source freshness checks configured and monitored 488|- [ ] Every model documented with descriptions for model and critical columns 489|- [ ] CI pipeline running dbt compile, dbt test, and dbt run on PRs 490|- [ ] CD pipeline running dbt run and dbt test on merge 491|- [ ] Environment isolation (dev, staging, production) 492|- [ ] Branching strategy (feature branches, PR reviews, main = production) 493|- [ ] Consistent naming conventions (stg_, int_, fct_, dim_) 494|- [ ] Custom generic tests for business rules beyond schema checks 495|- [ ] Model contracts enforced on mart models 496|- [ ] Cost monitoring (Snowflake resource monitors or equivalent) 497|- [ ] Access control (RBAC roles, not individual grants) 498|

Part of [[Data-Principles-for-Analytics-Engineering]]. See also: [[Data-Quality-and-Governance]], [[Data-Architecture]], [[AE-Consultancy-Delivery]].

  • [[Data-Principles-for-Analytics-Engineering]]
  • [[Data-Quality-and-Governance]]
  • [[Data-Architecture]]
  • [[AE-Consultancy-Delivery]]
  • [[10-Opportunities-for-Additional-Value]]