Data Contracts: Preventing Silent Schema Breakage in Pipelines

On Monday morning the GMV dashboard shows revenue down 99%. No job failed. No alert fired. Over the weekend, the orders team “simplified” the amount_paise column to send rupees instead. Same name, same integer type, every value now a hundredth of what it should be. Data contracts exist for exactly this failure: a data contract is a versioned, machine-checked agreement between a data producer and its consumers about schema, types, units and allowed values. Without one, the producer had no way to know anyone depended on paise.

Why do schema changes break pipelines silently?

Because most of the pipeline is built to keep going. Loaders coerce types rather than reject them. SELECT * absorbs new columns without complaint. A renamed column becomes NULL in a left join, and a NULL revenue figure sums to a smaller number instead of an error. Each layer swaps a loud failure for a quiet wrong answer.

The worst case is the one above, where nothing structural changed at all. Type checks pass. Row counts match. Only the meaning moved, and meaning is the one thing a schema never records. The data quality checks most teams run look for nulls and duplicates, not a unit that shifted overnight.

A contract check has to cover both layers: structure and meaning. The script below runs both against one incoming batch.

1import pandas as pd
2
3CONTRACT = {
4    "order_id":     {"dtype": "int64",  "nullable": False},
5    "amount_paise": {"dtype": "int64",  "nullable": False, "min": 1_000},  # ₹10 floor
6    "city":         {"dtype": "object", "nullable": True},
7}
8
9def check(df, contract):
10    errors = []
11    for col, rule in contract.items():
12        if col not in df.columns:
13            errors.append(f"missing column: {col}"); continue
14        if str(df[col].dtype) != rule["dtype"]:
15            errors.append(f"{col}: dtype {df[col].dtype}, expected {rule['dtype']}")
16        if not rule["nullable"] and df[col].isna().any():
17            errors.append(f"{col}: nulls in non-nullable column")
18        if "min" in rule and (df[col] < rule["min"]).mean() > 0.01:
19            share = (df[col] < rule["min"]).mean()
20            errors.append(f"{col}: {share:.0%} of rows below {rule['min']}")
21    extra = sorted(set(df.columns) - set(contract))
22    return errors + ([f"undeclared columns: {extra}"] if extra else [])
23
24# upstream now sends rupees in the paise column
25batch = pd.DataFrame({"order_id": [1, 2, 3], "amount_paise": [499, 1250, 89],
26                      "city": ["Pune", "Bengaluru", None]})
27print(check(batch, CONTRACT))
28# ['amount_paise: 67% of rows below 1000']

Every structural check passes. Only the range rule fires, and that rule exists because someone wrote down what a plausible order value looks like. Here, a schema check alone would have shipped the bad batch.

How do data contracts stop silent breakage?

They move the check to the producer’s side of the boundary, before data ships. Five fixes, in the order they pay off:

1. Keep the contract in the producer’s repository. The team that can break the data owns the file that describes it, and a pull request that changes a column must change the contract in the same diff.
2. Enforce structure at the boundary. For Kafka topics, Confluent Schema Registry rejects incompatible schemas; its default mode is BACKWARD, per the schema evolution documentation. For warehouse models, dbt fails the build when output columns or types drift from a contract set to enforced: true.
3. Add semantic rules that types cannot express. Units, ranges, allowed values and null rates, as in the script above. This is where the paise-to-rupees change gets caught.
4. Version breaking changes. Publish the new shape alongside the old one, give consumers a dated migration window, then retire the old version. Never mutate a column’s meaning in place.
5. List the consumers. A contract that names its downstream dashboards and models tells the producer exactly whom a change will hurt.

Where should data contracts be checked?

At three points, each catching what the previous one missed. In the producer’s CI, a schema diff against the contract file blocks the merge before anything deploys. At ingestion, the semantic rules run on each batch, and a failing batch goes to a quarantine table rather than into the warehouse or the bin. In the warehouse build, dbt contracts guard the models that consumers query.

Quarantine matters more than it sounds. Dropping a bad batch hides the incident, and loading it corrupts every report downstream. Holding it aside keeps the evidence, and lets the producer replay a corrected batch once the fix ships. Write the source system and owner into every quarantined record, the same metadata you would track for any data source and collection process.

Frequently Asked Questions