The Dependency Graph: Lineage Before You Touch Production

The first question in a data incident is rarely what failed?

It is usually:

What else will be wrong if we change this?

Without lineage, the answer is a search through SQL files, dashboards, notebooks, and tribal memory. That search is slow, incomplete, and especially dangerous when the pipeline is already under pressure.

Lineage turns a table into a dependency graph. It shows which job produced a dataset, which datasets a job read, and which consumers depend on the output. More importantly, it gives an operator a bounded answer before a migration, backfill, or rollback begins.

This is the first post in the next phase of the Reliable Data Systems series: making the platform explain its own state.

A Table Is Not the Whole Story

Consider a small warehouse flow:

1
2
3
4
5
6
7

raw.payments
      ↓
stg_payments
      ↓
fct_revenue ───────→ finance_dashboard
      ↓
customer_ltv ──────→ churn_features ──────→ model_v7

If raw.payments.currency becomes null, the immediate failure may be in ingestion. The business impact is somewhere else:

• Finance sees incomplete revenue by currency.
• Customer lifetime value becomes incomparable across regions.
• A model feature changes distribution without a model deployment.

The graph makes the difference visible. A pipeline log can tell you that a task succeeded. Lineage can tell you which products are now untrustworthy.

Three Entities, Two Kinds of Lineage

The useful minimum is not a complicated catalog. It is a consistent vocabulary.

OpenLineage uses this Job/Run/Dataset model and records state transitions such as START, COMPLETE, FAIL, or ABORT. Extra context is attached as facets: schema, ownership, input statistics, data quality results, or a query plan.

That distinction creates two complementary views:

Design-Time Lineage

Design-time lineage is inferred before execution:

• fct_revenue depends on stg_payments
• A model selects amount and currency
• A dashboard queries fct_revenue

It is useful for planning and code review. It can be extracted from dbt manifests, SQL parsers, workflow definitions, and declared contracts.

Run-Time Lineage

Run-time lineage records what actually happened:

• Run abc123 read partition 2026-06-14
• It wrote 4.8 million rows
• The output schema changed from version 3 to version 4
• The run completed with a 2.1% null rate in currency

Run-time lineage is the evidence needed during an incident. A static dependency file cannot prove which partition was touched by a specific backfill.

A Small Lineage Event

An implementation does not need to begin with a full catalog. Emit a run event at the boundary of each meaningful data job.

The exact transport and integration can vary. The durable design decision is to record the same identifiers every time. If one tool calls a dataset prod.fct_revenue and another calls it warehouse.finance.fct_revenue, the graph will split into two imaginary tables.

Lineage Quality Has Its Own Tests

Lineage is not automatically trustworthy because it exists.

Test the graph like any other production output:

Start with table-level lineage. Add column-level lineage only where it changes a decision, such as regulated fields, financial metrics, or high-risk model features. A graph that is 80% complete and clearly marked is more useful than a graph that claims false precision.

The Impact Analysis Protocol

Before changing a production dataset:

1. Identify the exact dataset, field, partition, or event version being changed.
2. Traverse downstream consumers and classify them as critical, important, or informational.
3. Find an owner for every critical consumer.
4. Check the last successful run, data-quality result, and schema version.
5. Choose the migration or rollback window based on actual usage, not file search alone.
6. Publish the change with a link to the affected graph and a verification plan.

After the change, compare the observed graph and metrics with the predicted impact. If the change touched an unexpected consumer, the lineage system has found a platform defect—not just an application defect.

The Lineage Rule

Lineage is operational state, not decorative documentation.

It should be emitted by the jobs that know what they read and wrote, enriched with ownership and quality metadata, and used before anyone makes a risky change. The goal is not to draw a beautiful graph. The goal is to make the blast radius small enough to act on.

References: OpenLineage overview, OpenLineage API events, and OpenLineage facets.

Next: why a database copy is not a change stream, and what log-based CDC really guarantees.