AI Agent for Data Analysis

A data analysis agent can produce a correct-looking chart from the wrong denominator. It can also run valid SQL that duplicates every transaction after a join. The essential question is whether the analysis preserves the business meaning of the data from the user’s request to the final explanation.

This guide is for analytics leaders, operations analysts and data product owners considering natural-language access to business data. An agent can clarify questions, select approved datasets, prepare queries, execute authorised analysis and explain results. It needs explicit metric definitions and access controls; fluency is not a substitute for either.

Agree a metric contract before allowing open-ended questions

A metric contract defines the population, grain, numerator, denominator, time basis, exclusions and owner. Grain means what one row represents: a batch, an inspection, an order line or a customer. Without it, a natural request such as “show yield by plant” can support several incompatible calculations.

Use approved semantic definitions where they exist. If the organisation has conflicting definitions, the agent should expose the difference and ask the metric owner to resolve it. It should not silently choose whichever definition produces a more attractive chart. Record the definition version with each analysis.

Distinguish an exploratory question from an operational decision. An analyst may inspect a hypothesis using a provisional dataset, but a production allocation or performance assessment needs the agreed measure and review. Keep the scope of the result visible so a tentative finding does not become an official KPI through repeated forwarding.

Worked analysis: manufacturing yield

Illustrative dataset: Batch A contains 100 tested units, of which 90 pass. Batch B contains 10 tested units, of which 5 pass. The batch yields are 90% and 50%. Their simple average is 70%, while the combined unit yield is 95 ÷ 110 × 100 = approximately 86.36%. Both calculations are mathematically valid; they answer different questions.

If the user asks what share of tested units passed, the combined calculation is appropriate. If the user asks for the average of batch-level yields, the unweighted measure may be the intended one. The agent should clarify or use the approved metric, not assume that the familiar word “average” settles the issue.

Now add a defect table with several defect rows for one tested unit. Joining it directly to a unit table and counting rows can inflate the denominator. The agent must understand the relationship and aggregate at the intended grain before joining, or use a verified distinct identifier as the metric requires. A query that executes successfully can still answer the wrong question.

A further complication is retesting. Does the measure count first-pass results, final disposition or every test attempt? These definitions can produce different conclusions about process quality. The manufacturing owner should approve the choice. The agent records the decision and keeps retest analysis separate rather than mixing attempts with physical units.

A copyable analysis contract

Illustrative unit-yield metric contract

Contract field	Example definition
Question	What share of tested units passed first inspection in the selected period?
Grain	One physical unit with one eligible first-inspection outcome
Numerator	Eligible units with a passing first-inspection result
Denominator	All eligible units with a recorded first-inspection outcome
Time basis	First-inspection timestamp in the approved reporting timezone
Exclusions	Documented non-production test records and incomplete outcomes; counts disclosed
Owner	Manufacturing quality owner approves definition and changes

This contract does not claim that first-pass yield is always the right measure. It makes one choice testable. A different question needs a different contract. Keep missing outcomes visible: excluding them can be justified for a defined measure, but the excluded count may itself signal an operational problem.

For an approved batch summary with one row per batch and compatible counts, the arithmetic could be expressed as the following illustrative SQL. The view, filters and permissions would need to be defined for the actual environment.

SELECT
  100.0 * SUM(first_pass_units)
        / NULLIF(SUM(first_tested_units), 0) AS unit_yield_pct
FROM approved_batch_quality_summary
WHERE inspection_date >= :period_start
  AND inspection_date < :period_end;

The half-open time interval avoids counting the same boundary twice when periods are adjacent. A zero denominator returns no calculated rate in this example; the explanation should say that there were no eligible tested units, not report a 0% yield. Those are different business states.

Separate question planning from query execution

Ask the agent to produce a short analysis plan: metric definition, approved source, filters, joins, intended output and known limitations. For ambiguous or consequential work, an analyst reviews that plan before execution. Simple recurring questions can use pre-approved query patterns with constrained parameters.

Execute through a restricted identity with permitted datasets, row and column access, timeouts and resource limits. Do not give a natural-language interface unrestricted production database access. A read-only operation can still reveal confidential information or consume excessive resources, so access and workload controls are both necessary.

Validate generated queries before running them. Check referenced objects, prohibited operations, join cardinality, parameter handling and expected result size. Use a controlled execution environment for code-based analysis. The security architecture should enforce these boundaries even when the model produces an unexpected request.

Make the dataset state reproducible

Store the query or transformation, parameter values, metric version, source refresh state and execution time. If multiple queries need a consistent database view, use an appropriate snapshot strategy. PostgreSQL’s transaction isolation documentation describes how Repeatable Read uses a transaction snapshot; the correct choice still depends on the actual database and workload.

For exported data, preserve an approved version or checksum and relevant provenance. If the same question returns a different answer later, the team should be able to distinguish changed data from changed logic. A conversational history alone does not preserve the dataset that produced a result.

Explain freshness in business terms. A dashboard refreshed today may still contain source events only through yesterday. Late-arriving inspection results can alter a period’s yield. When results feed a recurring management report, agree whether the period is provisional, frozen or subject to restatement.

Do not turn association into a cause

An agent may discover that one shift has lower observed yield. That does not establish that the shift caused the difference. Product mix, equipment, maintenance, input material and missing records may differ. Ask for disaggregation and relevant alternative explanations before recommending an operational change.

Keep descriptive, predictive and causal analysis distinct. A chart describes observed data. A predictive model needs out-of-sample validation for a defined target. A causal claim requires an appropriate design and assumptions. For an agent coordinating predictive retention work, the churn prediction guide shows why model evaluation and intervention effectiveness are separate.

When the user asks a question the available data cannot answer, say what is missing and propose a feasible next analysis. Do not generate a confident explanation from industry stereotypes. A valuable analytical result can be “the data cannot distinguish these explanations because the relevant variable was not recorded”.

Show missing outcomes as a separate measure

Illustrative extension: suppose another 20 eligible units entered inspection but have no recorded outcome at the data cut. The observed-result yield remains 95 ÷ 110 = approximately 86.36% under the contract above. Outcome completeness is a different measure: 110 ÷ 130 = approximately 84.62%. Present both, with the 20 unresolved outcomes visible.

Counting the missing outcomes as failures would produce 95 ÷ 130 = approximately 73.08%, but that would answer a different question and assert a treatment the evidence does not justify. Dropping them without disclosure would also hide an important limitation. The metric owner should decide whether the operational report can proceed, needs a provisional label or must wait for the missing results.

Preserve the same distinction in charts and exports. A missing observation is not automatically a zero, and an unavailable period should not be drawn as a genuine performance collapse. Give the reader the population, coverage and exclusions alongside the headline rate so the visual cannot silently change the analytical meaning.

Evaluate with adversarial business questions

Build a test set where the tempting answer is wrong: averages with unequal denominators, one-to-many joins, duplicate identifiers, changed categories, currency mixtures, missing timezones and zero eligible populations. Include questions about data the user is not entitled to see. The agent should decline or narrow access without revealing restricted values in its explanation.

Have analysts compare results with independently reviewed calculations, not only expected wording. Check row counts before and after joins and reconcile totals to approved reports where definitions match. A discrepancy should lead to investigation; it should not be “fixed” by changing a filter until the numbers agree.

Measure numerical correctness, definition adherence, unsupported causal statements, access-control outcomes and time to a verifiable answer. Record when the agent appropriately asks for clarification. Treating every clarification as a failure encourages silent assumptions. The evaluation framework provides a structure for these checks.

Begin with a bounded analytical service

Choose a small set of high-value questions with reliable data and named metric owners. Let the agent become useful within that boundary before widening the dataset catalogue. Assign responsibility for changes in schemas, definitions and permissions; a query that was correct last quarter may no longer represent the business.

For custom AI agent development, agree the data access, semantic definitions, review model and execution controls. Request information about a data analysis agent with three real questions, the approved metric definitions and a redacted example where two reports disagree. That disagreement is often the best starting point for designing trustworthy analysis.