Data Catalogue and Lineage for AI

A data catalogue helps a team discover and understand available data. Lineage explains how a particular value or document was produced and where it is used. An enterprise preparing AI needs both: a searchable inventory cannot explain a transformed value, while a technical pipeline diagram may say little about whether the underlying material is authorised for the proposed business purpose.

The readiness question is practical. Can the team identify the exact source, meaning, version, transformations and access conditions behind the evidence that a proposed AI service will consume? Can it determine what needs review when that source changes? If these questions cannot be answered for a bounded use case, a large organisation-wide inventory is not yet a substitute for usable evidence.

This guide is for data architects, governance teams and business owners establishing that foundation. It develops a minimum viable catalogue and lineage approach around a fictional supplier-reporting assistant. The example is illustrative, not a carbon-accounting method or a claim about a completed engagement. Use it within an enterprise data and AI readiness assessment to define what a controlled pilot may rely on and which missing relationships must be resolved first.

Decide what the catalogue must help someone do

Begin with a user question rather than a metadata collection campaign. An analyst might need to find an approved supplier reporting dataset for a particular entity and period. A reviewer might need to distinguish reported figures from independently verified figures. An engineer might need to know which transformation turns a source value into an application field. Those tasks determine the minimum metadata and the navigation between records.

Catalogue the asset at the level where ownership and allowed use can be stated coherently. A folder can contain several unrelated datasets. A database can contain tables with different access restrictions. Conversely, cataloguing every temporary file as a separate business asset can make discovery unmanageable. Distinguish the logical dataset, its available distributions or interfaces and the specific versions used in an assessment.

The W3C Data Catalog Vocabulary, DCAT 3, provides a standard vocabulary for describing datasets and data services. It can support interoperable metadata exchange. It does not supply the organisation’s business ownership decisions, verify the data or make a catalogue entry sufficient for AI readiness. Select a representation that supports the required decisions without pretending that metadata conformance settles them.

Define a small set of questions the catalogue must answer successfully before expanding its scope. A new team member should be able to identify an asset’s owner, meaning, available period, approved access route and known limitations. If the catalogue requires an informal meeting to interpret every field, it has captured names without making the data usable.

An original minimum catalogue record

Catalogue fields for a proposed supplier-reporting assistant

Field group	Example content	Readiness question
Identity and scope	Supplier-reported facility figures; named entity and reporting period	Does this describe the fact the assistant will discuss?
Ownership and authority	Procurement data owner; source-review responsibility; assertion status	Who can explain and approve the interpretation?
Versions and availability	Original document revision, ingestion time and supersession relationship	Which information was available at the relevant time?
Meaning and quality	Units, reporting boundary, missing values and unresolved corrections	Can values be compared without changing their meaning?
Access and permitted use	Approved users, source agreement and authorised processing route	May this pilot use and disclose the material?
Derivatives and dependencies	Extracted rows, normalised table, index and evaluation references	What depends on this source and may need updating?

Attach evidence to important metadata claims. “Owner confirmed” should identify the confirmation and its scope. “Updated daily” should distinguish a contractual schedule from observed delivery behaviour. “Approved for AI” needs a defined purpose, audience and processing route; it is too broad to be useful without them. Mark unknown fields explicitly so that a complete-looking form does not conceal missing decisions.

A catalogue can contain pointers to restricted evidence instead of exposing the evidence itself. Metadata may reveal confidential projects, sensitive dataset names or the existence of an investigation. Apply suitable access controls to the catalogue and its previews. Discovery rights and rights to read underlying records can differ, and both need a deliberate design.

Trace one assertion from the supplier to the proposed answer

Imagine an authorised supplier document reports 2.4 tonnes of carbon dioxide equivalent for one facility and one quarter. The proposed assistant will summarise supplier-reported information for an internal procurement reviewer. It will not certify emissions, infer a product footprint or rank suppliers for automatic award. These boundaries define what the evidence chain must preserve.

Extraction creates a row containing the reported value, unit, facility, period and source location. A normalisation step converts 2.4 tonnes into 2,400 kilograms using a factor of 1,000. The normalised field must say kilograms, while the original assertion remains available. If the number is changed but the unit label is not, the pipeline has altered meaning even though extraction and conversion jobs both report successful completion.

Now suppose someone joins that facility-quarter figure to a purchase record for 400 boxes and divides the value by the box count, producing 6 kilograms per box. The arithmetic is correct, but the interpretation is not established: the source did not allocate all facility emissions to those purchases. Lineage should make the join and derived calculation visible so that a reviewer can reject an unsupported product-level claim. A reproducible calculation can still answer the wrong question.

The final answer must retain the assertion type: supplier-reported, for the named facility and quarter, under the source’s stated boundary. A source citation should lead to the relevant evidence, not merely to the supplier’s homepage. The system also needs a way to represent a later correction without erasing the earlier assertion that was available to a previous decision.

Represent the chain with entities, activities and responsibility

The W3C PROV family describes provenance through identifiable entities, activities and responsible agents. That offers a useful conceptual structure for an original enterprise lineage design. In this example, the document and extracted table are entities; extraction and unit normalisation are activities; the supplier, data owner and processing service carry different responsibilities. Their roles should not be collapsed into one generic “creator” field.

Give each important source assertion a stable identity and identify the particular version being used. Record the transformation specification and relevant configuration, input references, output references and execution time. An engineer should be able to determine whether a normalised value was copied, converted, imputed, inferred or manually corrected. Those operations can have different evidential consequences even when their outputs share the same numeric type.

Keep technical execution and business approval distinct. A job identifier establishes which processing run produced a table; it does not show that procurement accepted the interpretation. A reviewer approval should state what was reviewed, the source version, the permitted use and any remaining limitations. If an approval covers only extraction accuracy, it should not be displayed as approval of the supplier’s underlying environmental statement.

Manual steps belong in the chain. Spreadsheet edits, copied email attachments and reviewer overrides often sit between well-instrumented systems. Record the relevant change and responsible role without collecting unnecessary personal content. If a material manual transformation cannot be reconstructed, mark the relationship as unverified rather than drawing a confident arrow over the gap.

Time and correction history are part of the evidence

Assume the supplier later corrects the quarterly figure from 2.4 tonnes to 2.1 tonnes on 16 May. A readiness dataset frozen on 1 May could not have used the correction at that time. A current summary may need the corrected value, while an investigation of an earlier decision needs to know what evidence was then available. One field called “latest value” cannot answer both questions.

Record at least the relevant business period, source publication or approval time, time received by the organisation and the time the value became available to the intended application. These can differ. Preserve amendment and supersession relationships, including the reason where it is known. Avoid replacing a historical row in a way that makes a later correction appear to have been available earlier.

Choose temporal behaviour from the use case. Current knowledge search may prefer an approved latest version with historical context. Predictive evaluation may require information available at a historical decision time. Rebuilding an evaluation dataset from today’s corrected tables can silently change the experiment. The catalogue should expose the available history and its limitations before the team promises historical reconstruction.

State what is genuinely reproducible. Retaining a document identifier without its content version may not be enough if the file is overwritten. Retaining configuration without the relevant source snapshot may reproduce the procedure but not the evidence. Some model outputs are variable; reproducible source selection and transformations should not be confused with a guarantee of identical generated wording on every run.

For retrieval systems, include the relationship between a source version, extracted text, document segments, embedding configuration and index build. A vector identifier alone is not a readable source citation. The service needs a controlled way to recover the relevant evidence and its applicability metadata. If the source is reprocessed with different segmentation, preserve enough identity information to distinguish a genuinely revised assertion from a changed technical representation of the same assertion.

For predictive datasets, record the transformation’s input period and availability assumptions alongside its technical version. An aggregation can use a legitimate field yet include records that arrived after the intended prediction time. A lineage graph showing the correct tables would not expose that issue unless the temporal filter is represented. Include the query or an inspectable specification where appropriate, and test it on cases around the boundary. This turns lineage into an explanation of what information was actually available, not merely a list of connected systems.

Use two directions of lineage for different decisions

Backward tracing asks where an answer, feature or data field came from. It is useful for checking a questionable value, confirming a unit conversion or identifying the source version behind a response. Forward tracing asks which datasets, indexes, reports or tests depend on a source. It is useful when a supplier withdraws a statement, a licence changes or a correction requires reassessment.

The two directions need an explicit boundary. A source can feed a warehouse table, an extracted-text store, a search index and a manually exported analysis. Capturing the warehouse path alone does not establish complete downstream coverage. Identify known consumers, their owners and any unmanaged exports. Where the organisation cannot enumerate them, record that frontier of knowledge and define a proportionate control on further distribution.

For a readiness pilot, exercise both directions on a few materially different paths. Trace an answer back to its assertion, then introduce an authorised synthetic source revision and identify the derivatives that should change. Check whether notifications and rebuild tasks reach the right owners. This is evidence about the tested paths; it does not certify that every unregistered consumer has been discovered.

Deletion, retention and access changes also travel through derivatives. A document being removed from its original folder does not prove that extracted text or an embedding index no longer contains its information. Coordinate these tests with permissions and provenance readiness. The appropriate handling depends on the applicable purpose, rights and retention obligations, not on a universal assumption that every copy should always be deleted immediately.

Measure lineage coverage against a declared inventory

Define what counts as a complete path before reporting coverage. For this assistant, a complete path might require the exact source revision, extraction location, unit rule, record linkage and final assertion status. A path that stops at the warehouse table is partial. A pipeline diagram inferred from naming conventions is not equivalent to observed or otherwise verified transformation evidence.

Classify the declared paths as supported, partial, unverified or outside the assessed boundary. Explain the consequence of each gap. A missing transformation for an unused optional description may have little effect on the pilot. A missing conversion rule for a figure presented to decision-makers can be a blocking condition. Counting every path equally can conceal that difference, so keep critical dependencies visible.

Do not report “complete lineage” without specifying the inventory it refers to. New data sources, manual exports and changed jobs can alter the denominator. Maintain a discovery process and require new consumers to register their dependency where appropriate. Tool-generated coverage measures need interpretation: a successful connector scan can establish that metadata was collected without establishing that its business meaning or manual transformations are known.

Use representative quality checks where a population estimate is needed, and retain separate targeted checks for difficult or high-consequence paths. A catalogue record’s completeness and a value’s correctness are different measurements. Report both if the readiness decision depends on them, with the applicable source snapshot and review limitations.

A practical readiness exercise before choosing a catalogue platform

1. Select one proposed AI task and state the audience, source families and permitted outputs.
2. Create catalogue records for the smallest set of assets needed to support that task, preserving unknown fields.
3. Choose one ordinary assertion, one corrected assertion and one manually transformed assertion.
4. Trace each assertion backward to its versioned source and forward to its known derivatives.
5. Ask a business specialist to explain units, boundaries and allowed interpretations using the recorded evidence.
6. Introduce a controlled source correction and confirm which records, evaluation cases and indexes require review.
7. Ask a receiving team to repeat the investigation without undocumented help from the original analyst.

Capture the time and evidence required for the exercise, but avoid turning speed into the only success measure. A fast answer that overlooks a reporting boundary is not useful. The result should identify missing metadata, untraceable transformations, unclear responsibilities and unavailable history. Those findings provide concrete requirements for tooling and operating procedures rather than a speculative feature wish list.

A lightweight register can be sufficient for a narrow assessment if it is controlled and maintained. A larger portfolio may require automated discovery, connectors, lineage collection and access workflows. Evaluate a platform against the actual troublesome paths, including manual transformations and source revisions. A polished demo on a clean dataset cannot establish how the tool will behave on the organisation’s heterogeneous evidence.

Make stewardship and change handling part of acceptance

Assign responsibility for business metadata separately from pipeline operation. The engineering team may keep a connector running, while the data owner decides whether the reporting boundary has changed. A supplier manager may obtain a corrected statement, while a governance role determines its permitted use. Define the handoff between them and the evidence that confirms each responsibility was fulfilled.

Set review triggers for new sources, changed schemas, unit conversions, revised source agreements and additional consumers. Where an automated check detects a change, specify who interprets its consequence. An alert that no one owns does not maintain readiness. Conversely, a manual process can remain effective if its responsibilities, evidence and response times suit the bounded use case and are tested.

Put unresolved dependencies into a remediation backlog with acceptance evidence. “Improve lineage” is too broad. “Preserve original and corrected supplier assertions with separate publication and ingestion times, then demonstrate both current and historical retrieval” gives the team an inspectable outcome. Keep the original defect and its affected decisions attached to the item so that closure does not lose the reason for the work.

The final readiness conclusion should identify the assets, source versions, traced paths, permitted uses and remaining limitations. It may support a controlled internal summarisation pilot while excluding product-footprint comparisons that lack an allocation basis. For a readiness assessment discussion about data catalogue and lineage, bring one proposed output, its source records and a known transformation problem. A short trace that explains a real decision is a stronger starting point than an inventory whose most important relationships remain assumed.