Data Quality in Representative Samples

A data-quality assessment is useful only when its results describe the data the proposed AI service will actually use. A clean export selected by a supplier, a folder of recent documents or a set of easy examples may help a team understand the format. None automatically represents the intended operating population. The assessment needs a defensible connection between the records inspected and the readiness decision being made.

This matters when a Saudi enterprise combines structured systems, Arabic documents, English reference material and manually maintained files. The easiest records to obtain may have different defects from the records that dominate real work. A high average can also conceal a small but essential source family that the proposed use cannot operate without. Sample design should make those differences visible.

The method below is intended for data owners, analysts and operations teams preparing an enterprise data and AI readiness assessment. It provides an original worked example, a sampling-plan specimen and an interpretation method. The numbers illustrate statistical reasoning; they are not measurements of an actual company, a recommended universal sample size or a claim that a particular AI application is safe to deploy.

Start with the unit that can make a business decision wrong

Choose a unit of analysis before counting records. A document can contain several products, a product can have multiple compatible variants and an order can contain several lines. Testing one page per file may miss the line that determines the customer’s answer. Testing every line from a small number of files may create the appearance of a large sample while providing little independent coverage of suppliers or document formats.

For a spare-parts catalogue, the critical unit might be a proposed product–vehicle-variant–model-year relationship. That is different from a product description or a PDF page. A readable description does not prove the fitment relationship. The assessment must identify the approved reference against which that relationship can be checked, with technical compatibility decisions retained by qualified business specialists.

Write the target population in operational terms: all relationships that the intended pilot may retrieve at a specified snapshot, for specified product families and channels. Identify excluded records and why they are excluded. If the future service will answer questions about archived products, excluding them because they are difficult to export changes the population and therefore the meaning of the results.

The NIST guidance on defining a sampling plan connects sampling with explicit measurement goals, selection procedures and responsibilities. Apply that discipline to the business unit of analysis. The sample is a way to answer a stated question; it is not a ceremonial number of files collected before a project begins.

Construct a frame that exposes missing populations

The sampling frame is the operational list from which records can be selected. Reconcile it with the systems, source owners and business processes that define the intended population. A database query can return valid rows while omitting records stored in an inaccessible branch system. A folder listing can miss scanned attachments. The frame itself therefore needs evidence, not just an impressive row count.

Record the snapshot date, extraction rule, unique key, eligible count and known omissions. Compare totals with independent operational counts where available, investigating differences rather than forcing them to agree. Duplicate rows should be distinguished from legitimate repeated relationships. If the same fitment relationship appears in several source files, decide whether the unit is the relationship, each assertion about it or each delivered answer opportunity.

Representativeness is always relative to a population and purpose. A sample representative of catalogue relationships is not automatically representative of customer enquiries. Frequently requested parts may dominate service traffic while rare variants dominate the record count. If the readiness question concerns expected customer exposure, design a separate traffic-based view or appropriate weighting rather than silently borrowing catalogue weights.

When the frame is incomplete, state what can and cannot be concluded. It may support a pilot restricted to one verified source family. It cannot support a statement about every source the enterprise owns. Put missing access or incomplete inventories into the data remediation backlog, with a route to rebuilding the frame before expanding the intended use.

Worked sample: three source families with different defect rates

Suppose a fictional distributor has 10,000 eligible fitment relationships. The assessed snapshot contains 6,000 from structured manufacturer feeds, 3,000 from dealer spreadsheets and 1,000 from imported supplier PDFs. Assume each relationship belongs to exactly one source family for this exercise, the frame is complete and records are randomly selected within each family. These assumptions matter to the calculation.

The team deliberately inspects a larger fraction of the smaller, less standardised sources. It selects 120 structured-feed records, 120 spreadsheet records and 60 PDF-derived records. A predefined rule marks a relationship as unusable for the proposed answer when the model-year match cannot be supported by the approved reference. The review identifies 3, 12 and 12 unusable relationships respectively.

Illustrative stratified sample of catalogue relationships

Source family	Population	Reviewed	Unusable	Observed rate	Population weight
Structured manufacturer feed	6,000	120	3	2.5%	60%
Dealer spreadsheet	3,000	120	12	10%	30%
Supplier PDF import	1,000	60	12	20%	10%

Pooling the sample gives 27 unusable relationships out of 300, or 9%. That describes the selected sample. The weighted estimate for the defined catalogue population is 60% multiplied by 2.5%, plus 30% multiplied by 10%, plus 10% multiplied by 20%: 6.5%. The difference arises because the sample intentionally overrepresents the smaller source families. Neither number is a universal model error rate.

The weighted result is an estimate, not an exact count of all defects. Its uncertainty depends on the selection design and variation within each stratum. Relationships derived from the same supplier document may share errors, so treating every relationship as independent can understate uncertainty. If selection occurs by document or supplier, preserve that clustering in the analysis and obtain appropriate statistical support for the inference.

Define the defect before the reviewer sees the record

A useful quality rule states what must be checked, against which reference, using which tolerance and with what result categories. “Looks correct” is not reproducible. For the fitment example, a rule could require an identifiable product, an approved variant mapping, an explicit model-year interval and supporting reference evidence. It should also explain how to handle conflicting references or a product outside the authorised catalogue scope.

Separate an observed incorrect value from an unsupported value. Imagine that further investigation of the 27 unusable sampled relationships identifies eight incorrect mappings and nineteen relationships with insufficient evidence to settle the mapping. Both groups may be unsuitable for the proposed automated answer, but the second group is not proof of technical incompatibility. This distinction changes remediation and prevents the assessment from overstating what it discovered.

Use an explicit review outcome for unresolved cases. Do not replace them with the reviewer’s best guess or remove them from the denominator because the answer was difficult. Where a quality rule concerns usable evidence, an unresolved relationship may fail that readiness criterion. The report should still preserve the reason: missing evidence, conflicting evidence or an established error. Decision-makers need both usability and evidential status.

Check agreement between reviewers on a deliberate subset containing ambiguous cases. Discuss disagreements in the interpretation of the rule before expanding review. If specialists apply different definitions of a product family, the problem may be source authority and business meaning, rather than poor individual review performance. Keep the original review outcomes when a later adjudication changes the final label.

Measure completeness and correctness at the right level

Field completeness is often easy to measure and easy to overinterpret. Suppose the same 300 sampled relationships each have eight assessed fields: 2,400 field positions. If 24 are empty, field completeness is 99%. That can coexist with the 27 unsupported or incorrect fitment relationships. A filled field can contain the wrong value, and several missing fields can belong to one relationship. The two measures have different units and cannot be substituted for one another.

Report record usability separately from individual quality dimensions. Identify whether a missing model-year boundary blocks the intended answer while an absent optional description does not. Keep a critical-field result visible instead of allowing abundant low-consequence fields to dominate the average. The purpose is to understand the consequences of defects for the proposed workflow, not to maximise the headline percentage.

Define normalisation rules without erasing source evidence. Arabic and Western digits, spacing variations or alternative product labels may be reconcilable. But removing punctuation from a product identifier can collapse two valid variants. Expanding an abbreviation can introduce a technical assumption. Preserve the raw value and the proposed normalised value, with the mapping rule and a way to review cases that do not match confidently.

Quality also changes with time. A supported fitment relationship may become outdated after a catalogue correction. Assess whether the pilot requires the latest state, the state when an order was placed or both. A snapshot that passes today does not prove that updates will remain timely. Use catalogue and lineage records to connect the measured version to its upstream source and update mechanism.

Add difficult cases without corrupting the estimate

Representative sampling and targeted challenge testing serve different purposes. The probability sample estimates specified characteristics of its population under the design assumptions. A challenge set explores suspected failure mechanisms: merged table cells, conflicting year ranges, uncommon Arabic abbreviations or documents with a revised appendix. Deliberately selecting such cases is useful precisely because they may not reflect ordinary frequency.

Keep the two sets separately identified throughout extraction, review and reporting. Do not add thirty difficult PDF cases to the random sample and recalculate an overall “representative” percentage. Do not remove hard cases from the probability sample because similar cases already appear in the challenge set. Record the selection mechanism so that a later reviewer can tell which conclusions are descriptive and which are inferential.

Challenge findings can change the sampling plan. If a defect appears tied to a particular supplier format, define that subgroup and collect further evidence using a documented follow-up design. Preserve the first result and explain the adaptation. Repeatedly selecting more cases until a desired percentage appears creates a biased story; it does not strengthen readiness evidence.

Arabic source defects need their own diagnostic path. If a field appears absent because its text was missed during extraction, investigate Arabic OCR and document quality. If the value is present but cannot be found for a relevant question, the issue may be retrieval. Distinguishing these mechanisms directs effort toward the actual cause instead of treating every failed answer as a training-data shortage.

A sampling-plan specimen that another team can execute

Original plan fields for a readiness quality review

Plan field	What to record	Execution check
Decision and unit	Proposed answer scope and one eligible relationship	Reviewers count the same object
Frame and exclusions	Snapshot, extraction rule, keys and missing sources	Counts reconcile or differences are explained
Selection design	Strata, selection method, sample allocation and replacement rule	Selected records can be reproduced
Measurement rule	Approved reference, result categories and adjudication	Ambiguous cases have a defined route
Analysis	Denominators, weights, clustering and uncertainty approach	The stated estimate follows the design
Decision boundary	Proposed tolerances, mandatory constraints and permitted pilot scope	The result can support a specific next action

Include the person responsible for obtaining each source and the conditions under which sampled evidence may be viewed. Minimise personal or commercially sensitive material where it is not needed for the quality question. If access restrictions make review impossible, retain the selected record as an access limitation and apply a documented handling rule. Replacing it with an easy record without recording the change can distort the sample.

Choose sample size from the decision. The team may need enough precision for an overall estimate, enough coverage of a rare source family or sufficient evidence to investigate a severe defect. These are different requirements. Consider expected defect frequency, desired uncertainty, subgroup analysis, clustering, review cost and the consequences of a wrong readiness decision. A fixed number promoted as suitable for every enterprise has no defensible statistical basis.

Retain the selected identifiers and the selection rule in an authorised evidence location. Reproducibility should not depend on someone remembering which spreadsheet filter was active. Where identifiers are masked, preserve a controlled reconciliation mechanism so that a later correction can be traced to the right source without exposing unnecessary information to every reviewer.

Show uncertainty before declaring a pass

Specify the uncertainty approach before looking at the final labels. A confidence interval for a population proportion needs assumptions about selection and dependence; a standard formula for independent equally selected records may not fit a stratified or clustered design. Retain the information required to analyse the actual sample, including stratum counts and selection probabilities where relevant. If the available review budget cannot support the precision required by the decision, say so and narrow the conclusion. Displaying a percentage with extra decimal places does not create more evidence.

Zero observed instances of a severe defect should be interpreted within the sample’s reach. The defect may be rare, concentrated in an excluded source or associated with a workflow the sample did not cover. Use a deliberately designed search for the mechanism where the consequences justify it, while keeping that search separate from prevalence estimation. An exhaustive check of one small critical population can sometimes answer a particular question more directly than a large mixed sample. It still says nothing about populations outside the checked boundary.

A simple sensitivity calculation can expose the importance of unavailable evidence. In a separate hypothetical selection of 100 records, suppose 80 can be reviewed and 8 are unusable. The observed reviewed-only rate is 10%. Across the 100 selected records, the unusable fraction could range from 8% if all 20 unresolved records are usable to 28% if all 20 are unusable. Those are bounds for this selected set under the stated alternatives, not a confidence interval for the wider population. They make the missing evidence visible without pretending that its quality is known.

Agree how a close result will be handled before assessment begins. Options include collecting additional evidence under a planned design, restricting the source population, changing the task or retaining stronger expert review. Avoid repeated informal sampling until the result crosses a preferred threshold. Also record whether the business tolerance concerns a data defect, an unsupported AI answer or a realised consequence. These are different events. Data-quality evidence can inform the next evaluation, but it cannot establish the performance of an application that has not yet been tested in its intended workflow.

Convert results into a constrained next step

Do not turn the illustrative 6.5% estimate into an automatic launch rule. The business must determine the consequence of an unsupported fitment answer, the role of expert review and the effectiveness of withholding uncertain cases. A pilot limited to verified structured sources could have different prerequisites from a customer-facing service using every imported PDF. Record the population and operating boundary attached to each proposal.

Where a subgroup performs poorly, investigate whether the cause is missing source evidence, extraction defects, stale mappings or inconsistent definitions. Correcting the 27 observed records alone does not establish that the underlying population has improved. After systemic remediation, inspect fresh evidence under a suitable design and compare results on a consistent basis. Preserve the initial measurement as the baseline rather than overwriting it.

Use a readiness scorecard to present quality evidence alongside its confidence and scope, while keeping mandatory conditions separate. The useful output is a decision such as “prepare a restricted internal evaluation on these approved source families after these mapping issues close,” with named owners and acceptance evidence. Bring the proposed population, source counts and a few difficult records to a readiness assessment discussion about representative data quality.