Rethinking Data Quality for AI-Driven Systems: Evidence from Practitioner Interviews

1 Introduction

• Traditional data quality remains broader than error absence, including fitness for use and dimensions such as accuracy, completeness, consistency, timeliness, accessibility, and provenance.Source: §1 · p. 2
• Data in AI-driven systems shapes model behavior, evaluation, runtime context, and whether systems can be lawfully released or operated.Source: §1 · p. 3
• Practitioners’ experiences of provenance, synthetic data, documentation, and governance together across the AI lifecycle remain less understood.The study asks how practitioners define and assess quality, what challenges arise across development, and how organizational and technical factors shape trust.Source: §1 · p. 3
• Data trust means justified confidence that data suits a specific AI use and that its origin, transformations, ownership, permitted use, and behavioral influence can be examined.It is one part of AI system trustworthiness, alongside model design, human oversight, interfaces, and organizational controls.Source: §1 · p. 3
• The study interprets six practitioner-grounded themes and five explanatory mechanisms through lifecycle assurance: evidence that data artifacts fit specific AI claims.The framing spans training, validation, certification, deployment, and agent action.Source: §1 · p. 3

2.1 Background

• Data quality is fitness for use, so a dataset may suit one consumer, task, or decision but not another.Traditional assessment also considers context-dependent judgments alongside measurable properties.Source: §2.1 · p. 3
• Provenance and governance extend quality concerns from record contents to data origins, transformations, ownership, and organizational accountability.Source: §2.1 · p. 4
• Machine learning embeds data in system behavior and creates hidden dependencies, feedback loops, and downstream effects across collection, training, deployment, and monitoring.Upstream decisions can affect model behavior, evaluation, and operational reliability.Source: §2.1 · p. 4

2.2 Related Work

• Related work documents data cascades and advocates attention to data, but prior practitioner evidence also found quality often lacked a shared definition and relied on ad hoc inspection.The study extends the evidence-based view to pseudo-labels, synthetic data, model judgments, agent context, and legal provenance.Source: §2.2 · p. 4
• Research documents model-evaluator bias, recursive-training degradation, and difficulty establishing training-data licensing, while their connected treatment as lifecycle trust concerns remains less known.The study synthesizes its integrated practitioner account as evidence that data artifacts fit specific AI claims.Source: §2.2 · p. 4

3 Research Method

• The purposive sample included 16 practitioners from nine organizations, selected for direct AI development, deployment, or data-provision experience across varied organizational relationships with AI.The sample was weighted toward safety-critical perception-and-control AI and broadened with foundation-model, decentralized and federated AI, data-service, industrial-automation, and consultancy participants.Source: §3 · pp. 4–5

3.1 Data Collection

• Semi-structured interviews lasting 45–60 minutes were conducted in English between April and June 2026, remotely or on site.WhisperX transcripts were manually checked against recordings; AI was used only for transcription and speaker separation.Source: §3.1 · pp. 5–6

3.2 Data Analysis

• Reflexive thematic analysis followed Braun and Clarke’s six phases, using inductive coding and iterative review to construct themes without predefined quality categories.Coding and theme construction were performed by the authors.Source: §3.2 · p. 6, §3.1 · p. 6
• A second interpretive step compared the six themes to identify five explanatory mechanisms, while a supplementary table linked extracts and codes to themes and mechanisms.The counts of extracts document analytical grounding, not frequency; the authors did not calculate inter-coder agreement.Source: §3.2 · p. 6

T1: Behavior-Level Traceability

• Practitioners traced model behavior through versions, data distributions, sources, and labels; end-to-end systems made attribution costly and indirect, with organizational gaps leaving no owner for the full chain.Diagnosis relied on experimentation and A/B testing when models offered no reasoning trace.Source: T1: Behavior-Level Traceability · p. 7

T2: Models as Data-Quality Judges

• Model-based assessment scaled beyond manual review, but annotation-trained assessors inherit blind spots; model-generated assessor data can sever external ground truth, keeping human checks essential.An unusually high inference loss flagged suspect annotation sequences for rework before the next training cycle.Source: T2: Models as Data-Quality Judges · pp. 7–8

T3: Agent Context and Memory as a New Data Object

• Retrieved context and retained memory became quality objects, but accumulated memory could corrupt decisions and remained unmanaged because no role owned its provenance, freshness, or contamination.Only three participants discussed agent memory in depth, as agentic systems were still emerging in participating organizations.Source: T3: Agent Context and Memory as a New… · p. 8

T4: Authenticity of Generated Data

• Authenticity became stage-specific: generated and pseudo-labeled data raised concerns about observed-world fit, yet synthetic data could outperform unreliable measurements or cover rare cases.Practitioners described different acceptance thresholds for training and validation data.Source: T4: Authenticity of Generated Data · p. 9

T5: Lawful Governance and Data Rights as a Training-Data Gate

• Foundation-model teams treated lawful status as a training-data gate, while deletion demands raised the need for provable unlearning; automotive participants prioritized calibration and representativeness over GDPR.Unclear copyright, license, opt-out, or personal-data status could make technically useful data unacceptable for training.Source: T5: Lawful Governance and Data Rights… · pp. 9–10

T6: Representativeness as Behavioral Coverage

• Safety coverage can fail structurally: teams may lack the 10% most important crashes, which is 95% of fatal accidents, because impact destroys collection hardware.Representativeness was framed as coverage of situations in which the system must behave safely, including rare or structurally unrecordable cases.Source: T6: Representativeness as Behavioral… · p. 10

5.1 What AI Changes About Data Quality

• Data-quality defects can reappear as behavioral errors, weak validation confidence, or reduced trust, becoming harder to attribute after models or AI-generated artifacts mediate them.Practitioners linked trust to ownership, traceability, and governance, including whether data's use was permitted.Source: §5.1 · p. 11
• AI shifts data quality from a dataset property to lifecycle assurance: evidence that artifacts are fit for the specific claims they support.Traditional dimensions remain necessary, but their interpretation changes when data is embedded in model behavior, assessed by models, or constrained by legal rights.Source: §5.1 · pp. 10–11

5.2 Why the Themes Recur: Five Foundational Explanatory Mechanisms

• Four mechanisms recur: data is reused beyond its original purpose, scale limits inspection, models absorb data into behavior, and AI creates new data artifacts.These are second-order interpretations across the themes, with links to reduced trust presented as analytical rather than causal.Source: §5.2 · pp. 11–12, Figure 1 · p. 12

F5. Organizational ownership of data quality is fragmented (the organizational

• Fragmented ownership leaves the evidence chain between data sources, transformations, models, and checks without an end-to-end owner, helping explain traceability loss and unmanaged AI artifacts.The authors interpret this mechanism, alongside the other four, as weakening trust in links between sources, behavior, permitted use, and ownership.Source: F5. Organizational ownership of data… · pp. 12–13

5.3 Implications: Lifecycle Assurance as Evidence Infrastructure

• Lifecycle assurance links each data artifact to evidence matched to its lifecycle claim, including artifacts generated by the system that re-enter earlier stages.The conceptual model organizes participant-described artifacts into five lifecycle layers and associated evidence.Source: §5.3 · p. 13, Figure 2 · p. 14
• The implications are claim-specific assurance, first-class authenticity and provenance metadata, and accountability across the data-model-decision chain.The proposed evidence infrastructure makes uncertainty explicit and supports trusting data only within the claims it can support.Source: §5.3 · pp. 13–14

• The automotive-weighted sample may limit transferability to other domains, despite providing access to mature practices in safety-critical settings.Recruitment through professional networks and referrals may also have favored practitioners with similar backgrounds.Source: §6 · p. 14
• The first author conducted interpretive coding; reflexivity relied on notes, dataset comparison, and author discussion rather than inter-coder agreement.Transcripts were manually checked against audio, though errors could remain.Source: §6 · p. 14
• Lifecycle assurance is an interpretive framing from this study and requires evaluation across organizations and domains.Source: §6 · p. 14

• Interviews with 16 practitioners across nine organizations yielded six themes and five mechanisms, synthesized into lifecycle assurance for evidence that artifacts fit AI claims.The framing makes uncertainty explicit, reviewable, and accountable across the data-model-decision chain.Source: §7 · p. 15
• Managing AI data quality increasingly requires evidence justifying how data is used, transformed, and trusted throughout the lifecycle.Source: §7 · p. 15
• Future work should evaluate lifecycle assurance in industrial studies, operationalize artifact-level evidence in pipelines, and examine effects across domains.Source: §7 · p. 15