Metadata Is Operational State: Building a Catalog People Can Trust

“We have a data catalog” can mean two very different things.

It can mean there is a searchable list of tables with descriptions written six months ago. Or it can mean an operator can answer, in a few seconds:

• What is this dataset for?
• Who owns it?
• When was it last updated?
• Can I trust it for this decision?
• What will break if it changes?

The second one is not a documentation project. It is an operational metadata system.

This is the fourth post in the current Data Engineering run. The previous post described the table state. Now we add the context that lets people use that state safely.

Four Metadata Planes

Metadata becomes useful when its sources and freshness are explicit.

No single system knows all four planes. A reliable catalog joins them by stable dataset identifiers.

1
2
3
4
5

schema registry ─┐
warehouse ────────┼──→ metadata service ──→ catalog / search / alerts
orchestrator ────┤
quality tests ────┤
lineage events ──┘

If a field is manually copied from a pipeline log into a wiki, it will become stale at the exact moment an incident needs it.

The Minimum Data Product Card

A catalog record should be small enough to maintain and rich enough to make a decision.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

dataset: warehouse.finance.daily_revenue
description: Daily captured revenue by business date and currency.
grain: one row per business_date and currency
owner:
  team: finance-data
  channel: "#data-finance"
  escalation: finance-data-oncall
classification: internal-financial
source: payments.captured_events
freshness:
  target: 60 minutes
  critical_after: 180 minutes
quality:
  required_checks:
    - business_date_not_future
    - currency_allowed
    - reconciliation_to_payment_ledger
lineage: warehouse.finance.daily_revenue
consumers:
  - finance_dashboard
  - monthly_close_export
contract_version: 3

The record describes the interface, not the implementation. A user should not need to know whether it is built with dbt, Spark, Flink, or a SQL script to understand whether it is appropriate for a decision.

Make Updates Event-Driven

The catalog is trustworthy when pipeline activity updates it automatically.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

job starts
   │
   ├── emit run START
   ├── publish input/output lineage
   ├── publish schema and row statistics
   ├── attach quality results
   └── emit run COMPLETE or FAIL
          │
          ▼
     catalog projection

A catalog does not need to store every raw event forever. It can project the latest operational state while keeping links to the underlying run, contract, and lineage records.

OpenLineage’s model is useful here because it separates the Job, Run, and Dataset entities and allows facets to carry additional metadata. A data-quality result can be attached to the run. A schema can be attached to a dataset. An ownership record can be associated with the job or dataset. The graph remains interoperable while the catalog adds a human-facing view.

Freshness Is a Claim With Evidence

Never show “fresh” as a static label.

Show the evidence:

1
2
3
4
5
6

Expected update: every 60 minutes
Last source event: 2026-07-06 09:41 UTC
Last successful publication: 2026-07-06 09:48 UTC
Current freshness: 22 minutes
Status: within target
Evidence: run-20260706-0948

Separate source freshness from warehouse freshness. A table can be loaded successfully while the source has stopped producing data. The distinction was central to the data observability vital signs article.

Trust Should Be a Scorecard, Not a Badge

A green “certified” badge tends to outlive the conditions that justified it. Use a scorecard with explicit evidence instead.

This avoids confusing “the table exists” with “the table is suitable for every use.”

The Five-Minute Discovery Workflow

When a consumer asks, “Can I use this table?” the catalog should support a repeatable path:

1. Read the dataset purpose and grain.
2. Check the owner and escalation route.
3. Inspect freshness and recent run evidence.
4. Review required quality checks and current failures.
5. Follow upstream lineage to verify source meaning.
6. Follow downstream lineage before proposing a schema change.
7. Confirm the contract version and deprecation policy.

If the user must open seven disconnected tools to do this, the information may exist but the system still has a discovery problem.

Ownership Is a Data Field

Ownership cannot be an optional profile photo or an implicit Slack search.

Require it at publication time. Block a production dataset from being marked “trusted” when it has no accountable team, no definition of freshness, or no recovery contact. The goal is not bureaucracy. It is to prevent the data platform from silently assigning every unknown problem to the central data team.

The Metadata Rule

Catalogs earn trust by showing current evidence, not by displaying more fields.

Ingest technical facts from systems, operational facts from runs, business meaning from contracts, and usage from lineage and query history. Make each claim timestamped, attributable, and easy to challenge.

References: OpenLineage overview, OpenLineage facets, and BigQuery INFORMATION_SCHEMA metadata views.

Next: how query shape, pipeline design, and ownership determine the warehouse bill.