Data contracts for data platforms — Quick Intro

INTROBLOCK · 01

DCNT · 7 MIN PREVIEW

Your pipeline passed schema checks. It still delivered 40% nulls in revenue.

Schema validation and value-level contracts are two different things — and most teams only enforce one. This course shows you how to block breaking changes before they merge, attach freshness and completeness SLOs to a contract, and generate dbt schema, GE suites, and Avro schemas from a single ODCS v3.1.0 YAML so they can never silently drift apart.

CONCEPTBLOCK · 02

What a data contract actually enforces

A data contract is not a README. It is a machine-readable YAML (ODCS v3.1.0, now a Linux Foundation standard) that simultaneously defines schema shape, value-level quality rules, SLOs for freshness and completeness, and the server location — all in one file. Tools like datacontract-cli 0.10.x, dbt 1.9, and Soda Core 3.x read that file directly; they do not trust a human to keep a separate doc in sync. The failure mode contracts prevent is silent drift: a producer renames `customer_id` to `cust_id`, dbt runs green on the producer side, and 47 downstream models throw a column-not-found error at 2 AM. With `enforced: true` in dbt 1.9, that rename is a build-time hard failure on the producer's own PR — before it ever merges.

KEY DISTINCTIONSchema contracts block structural breaks (column removed, type changed). Value-level contracts block semantic breaks (column present but 40% null). You need both layers — dbt enforced: true for the first, GX Core 1.x Checkpoints or SodaCL for the second.

WATCH OUTODCS v2.x (the original PayPal / AIDA User Group format) was superseded by v3.0.0 in October 2024. The section names and required fields changed. If your team's contract YAMLs predate October 2024, they need migration before datacontract-cli 0.10.x will lint them cleanly.

GOTCHAA contract with no slaProperties freshness threshold is a promise with no expiry date. Glassdoor's petabyte-scale case study (Feb 2025) identified missing freshness SLOs as the top cause of stale dashboards that nobody gets paged for.

DIAGRAMBLOCK · 03

One ODCS YAML → four enforcement layers

datacontract-cli 0.10.x exports one ODCS YAML into dbt schema.yml, a GE Expectation Suite, and Avro schema. All four layers enforce the same contract — none are hand-written independently.

CODEBLOCK · 04

Minimal ODCS v3.1.0 contract with schema + quality + SLA blocks

YAML

1# contract.yaml — ODCS v3.1.0 (Linux Foundation Bitol, Apache 2.0)

2apiVersion: v3.1.0

3kind: DataContract

4id: urn:datacontract:orders:v1

5info:

6 title: Orders

7 version: 1.0.0

8 owner: platform-team@example.com

10servers:

11 production:

12 type: postgres

13 host: db.example.com

14 database: warehouse

15 schema: public

17schema:

18 - name: orders

19 type: table

20 columns:

21 - name: order_id

22 type: string

23 required: true

24 unique: true

25 - name: revenue_usd

26 type: number

27 required: true

28 - name: created_at

29 type: timestamp

30 required: true

32quality:

33 - rule: no_nulls

34 column: revenue_usd

35 type: predefined

36 - rule: freshness

37 column: created_at

38 type: predefined

39 threshold: 2h

41slaProperties:

42 - property: freshness

43 value: 2

44 unit: h

45 - property: completeness

46 value: 99

47 unit: percent

Line 1: apiVersion must be v3.1.0 — not '3.1.0' (string, not float). Line 15: schema block now supports complex types (JSON, Avro) in v3.1.0. Line 28: quality block accepts SQL, text, or predefined rules. Line 37: slaProperties drives Soda Core checks.yml generation via datacontract-cli export --format sodacl.

CHEATSHEETBLOCK · 05

5 contract rules every data platform engineer knows in 2025

01One ODCS v3.1.0 YAML is the single source of truth — never hand-write dbt schema.yml and Avro separately.

02Set enforced: true in dbt 1.9 so column removal is a build-time hard failure, not a 2 AM page.

03Pick a Kafka Schema Registry compatibility mode (BACKWARD/FORWARD/FULL) before the first schema is registered — changing it later requires a full topic migration.

04Attach freshness and completeness thresholds in slaProperties — a contract without an SLO is a promise with no expiry date.

05Run buf breaking --against .git#branch=main in CI for every Protobuf service — field-number reuse silently corrupts binary-encoded messages already in flight.

06Block PRs at commit time with dbt-checkpoint 2.x check-model-has-contracts before dbt build even runs.

MINIGAME · RAPIDFIRETFBLOCK · 06

Quick check — true or false?

A dbt model with enforced: true will fail the build if a contracted column is removed.

CLAIM 1/6 · READY · scroll into view

LESSON COMPLETEBLOCK · 07

That's the trailer.

NEXTLesson 1 · ODCS v3 contract anatomy

WHAT YOU'LL WALK AWAY WITH

Real skills, real career delta.

Skills you'll gain

ODCS v3.1.0 contract authoringWorking
Write and lint ODCS v3.1.0 YAML contracts covering the fundamentals, schema, quality, slaProperties, and server blocks using datacontract-cli 0.10.x. Diff contracts across versions to identify breaking field changes before they reach downstream consumers.
dbt model contracts with enforced: trueProduction
Configure dbt 1.9 model contracts in schema.yml with enforced: true so that column removal or type changes cause dbt build to exit non-zero. Implement multi-version models with deprecation_date and cross-project ref() in a dbt Mesh producer-consumer setup on DuckDB.
dbt-checkpoint pre-commit hook configurationWorking
Wire dbt-checkpoint 2.x hooks (check-model-has-contracts, check-model-has-description, check-source-has-freshness-defined) into .pre-commit-config.yaml to block commits that introduce contract-free models before dbt build ever runs.
Confluent Schema Registry compatibility mode enforcementProduction
Register Avro schemas against Confluent Schema Registry 7.8 via its REST API and configure BACKWARD, FORWARD, and FULL compatibility modes per subject. Trigger HTTP 409 conflict responses in CI when a schema evolution violates the configured compatibility rule.
buf breaking for Protobuf CI gatesWorking
Write buf.yaml v2 configs with FILE and WIRE_JSON breaking rule sets and run buf breaking --against .git#branch=main in GitHub Actions to block PRs that remove fields or reuse field numbers in .proto files.
GX Core 1.x Checkpoint authoring and Airflow integrationWorking
Define Expectation Suites and Checkpoints using the GX Core 1.x FileDataContext API and attach them as Airflow 2.9 task gates that fail the DAG and produce Data Docs HTML when freshness, completeness, or value-range expectations are violated.
SodaCL checks for freshness, completeness, and accuracy SLOsWorking
Write SodaCL checks.yml files for Soda Core 3.x that enforce freshness thresholds, completeness percentages, and null-column accuracy rules against a Postgres data source. Map each SodaCL check field to the corresponding ODCS slaProperties key.
datacontract-cli code generation pipelineWorking
Use datacontract-cli 0.10.x export commands to generate dbt schema.yml, Great Expectations Expectation Suite JSON, and Avro .avsc files from a single ODCS v3.1.0 YAML. Enforce artifact freshness in CI by regenerating all outputs and failing if any committed file differs from the generated version.
Kafka schema evolution testing with Avro and Docker ComposeWorking
Stand up a Kafka plus Confluent Schema Registry 7.8 stack with Docker Compose and write Python producer scripts using confluent-kafka 2.x to register schemas and demonstrate the contrast between a BACKWARD-compatible additive field change and a breaking field deletion that returns HTTP 409.
Backstage catalog-info.yaml for data contract discoverabilityWorking
Author catalog-info.yaml entity descriptors that link ODCS contract YAML URLs as annotations and configure mkdocs-based TechDocs to render contract schema, SLO, and ownership metadata inside a Backstage instance so consumers can locate contract terms without contacting the producing team.

Career & income delta

Career moves

Title yourself credibly as 'Data Contract Engineer' or 'Analytics Platform Engineer' on LinkedIn and in job applications — a role explicitly called out in 2025 data platform job postings at companies running dbt Mesh, Kafka, and ODCS-based governance; the ODCS v3.1.0 Linux Foundation standard (Bitol project, May 2025) gives the credential a named, citable specification to anchor the title
Move from individual-contributor data engineer into a 'Data Platform Governance' or 'Data Mesh Lead' track by demonstrating ownership of breaking-change CI gates (buf breaking, dbt-checkpoint, Schema Registry compatibility) — the exact toolchain cited in Netflix's Unified Data Architecture (InfoQ, December 2025) and Glassdoor's petabyte-scale case study (February 2025)
Position for 'Staff Data Engineer' or 'Principal Analytics Engineer' roles that require cross-team contract negotiation skills — dbt Mesh producer-consumer deprecation workflows (deprecation_date, versioned models, hard-delete windows) are listed as required competencies in Staff-level analytics engineering job descriptions on LinkedIn as of Q1 2025
Enter the Kafka/streaming platform engineering market by adding Schema Registry compatibility enforcement (Confluent 7.8, Apicurio 3.1) and Avro/Protobuf contract CI to your portfolio — streaming platform engineer roles on ZipRecruiter and LinkedIn consistently list schema governance as a differentiating skill separating mid-level from senior Kafka engineers in 2025 postings

Income impact

Data Engineer (3–6 years, contract/governance specialization): Levels.fyi reported a median total compensation of $175,000–$220,000 at mid-size tech companies for data engineers with platform governance skills in the United States as of Q4 2024; ZipRecruiter listed the average annual salary for 'Senior Data Engineer' at $152,000 as of March 2025, with top-10-percentile earners at $185,000+
Analytics Engineer with dbt Mesh + contracts expertise: LinkedIn Workforce Insights (January 2025) showed analytics engineer roles requiring 'dbt model contracts' or 'data mesh' commanding a 12–18% salary premium over general analytics engineer postings in the same metro; Levels.fyi median for Analytics Engineer L4/L5 at companies like Airbnb, Stripe, and Databricks ranged $160,000–$210,000 total compensation as of Q4 2024
Kafka / Streaming Platform Engineer with Schema Registry governance: ZipRecruiter reported average annual pay for 'Kafka Engineer' at $145,000 as of March 2025, with senior roles specifying schema governance or Schema Registry administration averaging $165,000–$190,000; Levels.fyi data for streaming infrastructure engineers at FAANG-adjacent companies showed $200,000–$260,000 total compensation at senior IC levels as of Q4 2024
Data Platform Lead / Staff Engineer with data contract ownership: Levels.fyi Staff Data Engineer total compensation ranged $230,000–$310,000 at top-tier tech companies as of Q4 2024; LinkedIn Workforce Insights (January 2025) identified 'data contract' as one of the fastest-growing skill keywords in data platform job postings, appearing in 34% more senior IC and lead-level job descriptions in 2024 versus 2023

Market resilience

Schema evolution reasoning — understanding what constitutes a wire-breaking change (field removal, field-number reuse, type widening vs. narrowing) in Avro, Protobuf, and SQL is a transport-layer-agnostic skill that transfers across Kafka, gRPC, REST, and any future messaging substrate; the compatibility mode mental model (BACKWARD/FORWARD/FULL) predates and will outlast any specific registry implementation
Contract-as-code authoring and enforcement — the practice of expressing data guarantees as version-controlled YAML (ODCS, SodaCL, dbt schema.yml) and enforcing them in CI is a workflow pattern that survives tool churn; the underlying discipline of 'define the interface before the implementation' transfers to any future standard that replaces ODCS v3
Producer-consumer SLO decomposition — the ability to decompose a data quality promise into freshness, completeness, and accuracy thresholds, attach them to a named owner, and wire them to an alerting system is a systems-design skill independent of whether the enforcement layer is Soda Core, Great Expectations, Monte Carlo, or a future tool; this maps directly to SRE error-budget thinking applied to data
Breaking-change CI gate design — knowing how to insert a compatibility check between a schema change and a merge (buf breaking, Schema Registry HTTP 409, dbt contract violation exit code) is a CI/CD design pattern that transfers to any language, any schema format, and any pipeline orchestrator; the skill is 'where in the delivery pipeline do I enforce the contract' not 'which specific CLI flag do I pass'