PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Data Cleaning Without a Clean Reference: An Experimental Study of Capabilities and Trade-offs

Hadi Fadlallah

arXiv:2608.14765v1cs.AIcs.DB

TL;DR

Reference-free data cleaning must distinguish genuine errors from rare but valid observations without a trusted clean reference. This study evaluates an evidence-grounded agent framework across configurations and finds that added capabilities create trade-offs rather than consistent overall improvement.

  • Problem

    Reference-free cleaning lacks a trusted reference for distinguishing genuine errors from rare but valid observations, making agent capability effects an open empirical question.

  • Method

    The study evaluates seven baseline and progressive agent configurations across three datasets, two settings, and 126 runs using profiling, tools, evidence controls, and conservative repair.

  • Results

    No configuration dominated across detection, repair, safety, evidence grounding, reproducibility, and cost; the deterministic baseline achieved the strongest synthetic detection result.

  • Takeaways & Limitations

    Capability composition should be selected according to the cleaning objective, available evidence, and consequences of incorrect modification.

  • Takeaways & Limitations

    Synthetic corruption may be more regular, visible, or reversible than natural errors, while original-data outputs were not manually adjudicated for accuracy.

Abstract

from arXiv · show

Data cleaning without a trusted clean reference is challenging because unusual values may represent either genuine errors or valid observations. This paper studies how different agent capabilities affect reference-free data cleaning and proposes an evidence-grounded framework that combines structured context, profiling, LLM reasoning, executable checks, controlled evidence retrieval, source ranking, citation alignment, conservative repair, reversible scripts, and provenance logging. Seven configurations are evaluated across financial, clinical, and environmental-monitoring datasets using controlled synthetic corruption and original-data descriptive analysis, resulting in 126 completed runs. The evaluation includes two comparison baselines and a progressive LLM-based sequence that adds executable tools, evidence retrieval, evidence controls, and conservative repair. In the synthetic evaluation, the deterministic profiling baseline achieved the highest detection F1-score of 0.561. Among the LLM-based configurations, the full conservative configuration achieved the highest F1-score of 0.421, but no configuration performed best across all evaluation criteria. The source-ranked configurations achieved the lowest unsupported-rule rates, while decision-level citation alignment remained weak. The full conservative configuration produced no unsafe or unnecessary modifications, although these rates were already zero before the conservative policy was added, and it performed no direct repairs. Overall, the results show that additional capabilities introduce trade-offs among detection, repair, evidence grounding, conservative behaviour, reproducibility, and operational cost rather than producing consistent improvements. The study provides a structured framework and empirical methodology for evaluating these trade-offs in reference-free agentic data cleaning.

1 Introduction · 2 Background and Related Work

The paper frames reference-free data cleaning as contextual, evidence-grounded decision-making rather than direct transformation. It connects classical data-quality, cleaning, retrieval, provenance, and agent research to an intentionally cautious study of how capabilities affect observable behavior under uncertainty.

  • 1 Introduction: LLMs and agentic systems can infer semantics, propose rules, generate scripts, explain anomalies, and coordinate workflows using reasoning, tools, and external evidence.Related work includes LLM-supported cleaning, automated pipelines, retrieval-supported repair, contextual rule generation, and semantic table profiling.
  • 1 Introduction: The proposed framework treats reference-free cleaning as evidence-grounded decision-making: agents detect candidate issues, gather evidence, construct requirements, classify repair risk, and generate reversible transformations only when justified.Ambiguous cases are preserved or escalated, while the framework combines profiling, controlled retrieval, source ranking, citation alignment, conservative repair, reversibility, and provenance.
  • 1 Introduction: The study evaluates two comparison baselines and a progressive A2–A6 ablation to measure how contextual knowledge, tools, evidence, and decision classes change cleaning behavior.Its three stated contributions are formalising these components, proposing the evidence-grounded framework, and reporting an exploratory capability ablation.
  • 1 Introduction: The paper does not assume that more agent capabilities improve cleaning and does not treat original-data outputs as if they had hidden ground truth.Instead, it examines observable changes under controlled conditions while respecting uncertainty in the reference-free setting.
  • 2 Background and Related Work: Data quality depends on the consumer, intended use, and represented entities across dimensions including accuracy, completeness, consistency, timeliness, believability, and interpretability.Context-aware assessment therefore selects rules with respect to usage context rather than treating quality as a single correctness property.
  • 2 Background and Related Work: Classical profiling, matching, and anomaly-detection systems show that unusual observations are deviations from expectations, not automatically errors.These systems motivate specialised logic for distributions, types, missingness, duplicates, dependencies, and entity resolution.
  • 2 Background and Related Work: Attribution and provenance research motivates checking whether cited sources support generated claims and recording the origins of data, rules, tools, and evidence.Retrieval-based approaches such as KATARA and RetClean support cleaning decisions when table content alone is insufficient.
  • 2 Background and Related Work: Reference-free cleaning must distinguish detecting a possible issue from modifying a value, because repair requires stronger evidence when unusual values may still be valid.Evidence grounding must be operationalised: retrieving a source or producing a citation alone does not establish source grounding.

3 Research Questions and Hypotheses · 4 Problem Definition

The study examines how agent configurations and added capabilities affect reference-free cleaning, including evidence controls, conservative repair, and operational cost. It defines cleaning as context-supported decisions over raw data when no trusted clean reference is available, with repairs constrained by evidence, checkability, reversibility, and policy.

  • 3 Research Questions and Hypotheses: The study asks how configurations and tool capabilities affect cleaning, whether evidence controls reduce unsupported rules or unsafe repairs, whether conservative repair preserves valid signals, and what costs added capabilities introduce.These questions are organized as RQ1–RQ5.
  • 3 Research Questions and Hypotheses: The hypotheses predict improved structural and syntactic detection from profiling and executable validation, fewer unsupported rules from source controls, and safer behavior from conservative repair.The hypotheses also allow that citation support may remain incomplete and added capabilities may introduce trade-offs.
  • 4 Problem Definition: In the reference-free setting, the raw dataset X is interpreted with available schema, metadata, sample records, profiling summaries, documentation, and other structured cleaning context because clean reference X∗ is unavailable.The context includes source, structural, semantic, temporal, use, risk, policy, and provenance characteristics.
  • 4 Problem Definition: The decision-support set K may include dataset context, retrieved evidence E_ext, and tool outputs T from parsing, duplicate, unit, temporal, and consistency checks.The experiment used a controlled local evidence corpus rather than online retrieval, and not every K component is required for every task.
  • 4 Problem Definition: The agent produces a decision set C in which each cleaning decision selects among preserve, human-review, or reject-repair outcomes.The notation separates the raw dataset X from the decisions C.
  • 4 Problem Definition: A repair is permitted only when evidence is sufficient, the rule is operationally checkable, the transformation is reversible, and the conservative policy is not violated.The formulation distinguishes a cleaning requirement from the transformation ultimately applied.
  • 4 Problem Definition: The framework requires evidence-grounded repairs, restraint toward rare or extreme values, executable validation where reliable, ranked and decision-aligned sources, provenance-preserving reversibility, and empirical capability evaluation.Unusual values should not be modified solely because they are unusual.

5 Knowledge and Decision Taxonomies

The framework organizes reference-free cleaning around taxonomies for required outputs, contextual knowledge, tools, evidence, and conservative decisions. Its implementation distinguishes measured capabilities from framework concepts and referral mechanisms, emphasizing auditable, non-automatic repair decisions.

  • Taxonomy framework: Five taxonomies define required outputs, supporting knowledge, tool capabilities, decision evidence, and conservative action classes.They synthesize prior work spanning data quality, profiling, retrieval, attribution, provenance, constraint-based repair, probabilistic cleaning, and anomaly detection.
  • Experimental operationalization: Table 1 maps taxonomy components to the ablation while distinguishing directly measured categories from partially implemented context and logging or framework-level referral mechanisms.Decision labels are logged across configurations, but human review is implemented only as a referral class and no external expert adjudication of original-data cases was performed.
  • Required outputs: The required-output taxonomy separates candidate issues, requirements, rules, decisions, repairs, provenance, and escalation so detected anomalies are not automatically treated as repairable errors.Provenance and escalation are first-class outputs, enabling reviewers to inspect the issue, inferred requirement, supporting evidence, chosen action, and any changed value.
  • Prior knowledge: Prior knowledge includes schema, semantic, temporal, entity, risk, policy, and provenance context, but the experiment does not evaluate these categories independently.The taxonomy is broader than schema metadata because interpreting financial, clinical, or environmental values may require production, usage, and risk context.
  • Tool capabilities: Tools handle deterministic verification such as parsing, type validation, duplicate detection, and script execution, while the agent interprets outputs using domain context and repair risk.Tool outputs are treated as evidence records rather than unverified suggestions, preserving a verification role distinct from LLM judgement.
  • Evidence: The evidence taxonomy makes authority, relevance, specificity, consistency, and temporal validity explicit without claiming a universal evidence-quality formula.Different domains may weight these dimensions differently, with clinical and regulatory settings potentially requiring stronger authority and specificity.

6 Methodology: Evidence-Grounded Agentic Cleaning Framework

The framework combines profiling, evidence acquisition, executable validation, conservative decision-making, reversible repair, and provenance logging for reference-free data cleaning. It separates dataset-derived from retrieved evidence and links supported requirements and rules to auditable actions.

  • Framework workflow: The multi-stage process covers issue detection, evidence acquisition, requirement generation, rule construction, decision classification, reversible repair execution, and provenance logging.It begins with the raw dataset, schema, metadata, and sample records, then uses profiling and issue detection to identify candidate quality problems.
  • Evidence sources: The framework distinguishes dataset-derived evidence from retrieved evidence, using both to support requirements and rules for evaluating candidate cleaning decisions.Dataset-derived evidence includes profiling, missingness, duplicates, temporal behavior, distributions, and executable validation results; retrieved evidence includes documentation, standards, manuals, and domain references.
  • Framework workflow: The five-step workflow constructs ΓX, profiles data, proposes contextualized issues, invokes deterministic checks, creates evidence-backed rules, and classifies decisions conservatively.Evidence is assessed by authority, relevance, specificity, consistency, and temporal validity.
  • Conservative repair: Safe repair requires a clear issue, evidence-supported rules, and reversible transformations; uncertain or high-risk cases may receive conditional repair, flagging, preservation, review, or rejection.The conservative repair policy can reduce direct repair when evidence is partial or conflicting.
  • Execution and provenance: Executable checks and reversible scripts separate agent recommendations from operational effects, preserving original values and producing logs that support auditability and reproducibility.Evidence records are linked to requirements and actions so reviewers can distinguish supported repairs from unsupported ones.

7 Experimental Design and Implementation

The experiment evaluates seven configurations across three datasets and two run modes, with 126 completed runs using repeated dataset–mode–configuration combinations. It combines controlled synthetic corruption for ground-truth evaluation with descriptive-only analysis of unmodified original data.

  • Experimental scope: Seven configurations were evaluated across financial, clinical, and environmental-monitoring datasets in three repeated runs for each dataset–mode–configuration combination, yielding 126 completed runs.A0 and A1 were comparison baselines; A2–A6 formed the progressive LLM-based ablation.
  • Run modes: Synthetic-corruption runs injected known errors and plausible valid-signal cases for ground-truth detection, repair, and preservation metrics, whereas original-data results were descriptive only.The original datasets lacked a trusted clean reference or manual adjudication in the verified outputs.
  • Configurations: A0 was an LLM baseline without profiling, executable tools, evidence retrieval, or conservative controls; A1 was deterministic profiling and checking; A2–A6 progressively added capabilities.The direct comparisons were A2→A3 for executable tools, A3→A4 for evidence retrieval, A4→A5 for source-ranking and citation alignment, and A5→A6 for conservative repair.
  • Implementation: LLM-based configurations used openai/gpt-oss-20b through OpenRouter with temperature 0.2 and a 3000-token maximum, while A1 was deterministic and used no LLM.Model settings, retry logic, output constraints, and postprocessing were fixed across LLM-based configurations.
  • Implementation: All configurations produced a fixed JSON schema covering issues, requirements, validation rules, decisions, evidence, provenance, and audit indicators, enabling comparable postprocessing.Malformed responses were normalized or JSON-repaired by the saved pipeline, with those events retained in execution records.

8 Evaluation Protocol

The evaluation separates cleaning effectiveness from safety, evidence grounding, cost, and reproducibility, using synthetic corruption for measurable effectiveness and original-data runs for descriptive analysis. Comparisons focus on observed patterns, audit indicators, and reproducibility rather than statistical significance.

  • Evaluation design: Synthetic-corruption runs evaluate detection, repair, and valid-signal preservation because injected ground truth is available.Original-data runs are interpreted descriptively because no trusted clean reference or manual adjudication is available.
  • Evaluation design: Execution indicators, including retries, JSON repair, schema normalization, and reversible-script execution, are reported as an audit rather than cleaning-effectiveness measures.
  • Configuration comparisons: The progressive capability analysis compares A2 versus A3 for executable Python validation, A3 versus A4 for controlled evidence retrieval, A4 versus A5 for source ranking and citation alignment, and A5 versus A6 for conservative repair.A0 and A1 are contrasting LLM and deterministic baselines, while A2 is the profiling-aware LLM starting point.
  • Interpretation and limitations: No p-values, confidence intervals, or statistical significance tests are reported because they were not part of the completed analysis.The evaluation instead focuses on observed means, execution-audit indicators, exact reproducibility across repeated runs, and directional hypothesis patterns.
  • Interpretation and limitations: Original-data results are kept separate from the controlled synthetic evaluation because original-data issues may represent tru

9 Results

The 126 completed runs reveal configuration trade-offs across synthetic detection, repair and safety, evidence grounding, descriptive behavior, operational cost, and reproducibility. No configuration consistently dominated these criteria, and original-data results describe behavior rather than accuracy.

  • Execution validity and reproducibility: 126 successful runs produced schema-valid final outputs after normalization, with JSON repair used in 30 runs and postprocessing changing 31 saved outputs.No agent was rerun and no new cleaning decisions were introduced during postprocessing.
  • Synthetic-corruption performance: 0.561 was A1’s detection F1-score, the highest among configurations; among LLM-based configurations, A6 achieved the highest F1-score at 0.421, while A4 had the highest precision at 0.928.A1’s precision was 0.687 and recall was 0.484; A0’s F1-score was 0.013.
  • Repair and safety: 0.770 was A1’s safe-repair rate and 1.000 its valid-signal preservation rate, but its repair-match rate was 0.000 and it produced unsafe repairs and unnecessary modifications.Only A3 achieved a non-zero repair-match rate, at 0.028.
  • Repair and safety: 0 unsafe or unnecessary modifications were produced by A6, which made no repair-class decisions and therefore had zero safe repair and zero repair match.A6’s behavior is interpreted as conservative non-repair rather than successful direct repair.
  • Dataset-level variation: 0.405 was EHR’s mean synthetic detection F1-score across configurations and repetitions, compared with 0.380 for German Credit and 0.280 for Radiation.The experiment does not isolate which dataset characteristics caused these differences.
  • Evidence grounding: 0.037 was the unsupported-rule rate for both A5 and A6, the lowest reported, despite their lower synthetic evidence coverage of 0.630 and 0.519; citation alignment reached only 0.036 for A5 and 0.041 for A6.Evidence coverage was 0.870 for A1, 0.778 for A4, and 0.722 for A2, showing that coverage and unsupported-rule rates did not move together.
  • Operational cost: 77.466 seconds and 12916.222 tokens were A6’s highest mean runtime and token use across all runs, rising to 138.938 seconds and 14803.000 tokens in original-data runs.A1 was fastest because it used deterministic profiling and no LLM calls, while runtime did not increase monotonically as capabilities were added.

10 Discussion

Adding capabilities changed different aspects of reference-free cleaning without producing monotonic overall improvement or a configuration that dominated across evaluation criteria. The findings instead support evaluating profiling, executable validation, evidence controls, conservative policies, and abstention as separate capabilities, while distinguishing synthetic accuracy from original-data behaviour.

  • Capability trade-offs: A1 achieved the strongest synthetic detection result, while the schema-only LLM baseline A0 performed poorly, showing that more complete LLM configurations did not guarantee better detection.Capability additions changed system behaviour without producing monotonic improvement in overall cleaning performance.
  • Capability trade-offs: 0.307 to 0.402: Moving from A2 to A3 increased synthetic detection F1-score, but repair-match rate remained 0.028 despite successful reversible-script execution.Executable tools improved validation of suspected problems without resolving the correct replacement-value problem.
  • Evidence grounding: 0.037: A5 and A6 achieved the lowest synthetic unsupported-rule rates, while decision-level citation-alignment scores remained 0.036 and 0.041.The results distinguish making evidence available from grounding individual decisions in that evidence.
  • Conservative behaviour: 0: A6 made no unsafe or unnecessary modifications, but A5 had already achieved zero on both indicators; A6 instead produced no repair-class decisions.The A5–A6 comparison does not clearly support H3 on these safety measures and shifted behaviour toward abstention and escalation.
  • Operational and dataset effects: A6 had the highest overall runtime and token consumption, whereas A1 was considerably less expensive; effectiveness also varied by dataset, with mean synthetic detection F1-score highest for EHR at 0.405.Mean synthetic detection F1-score was 0.380 for German Credit and 0.280 for Radiation, and original-data runs assessed behaviour rather than objective accuracy.
  • Overall interpretation: No configuration dominated across detection, repair, safety, evidence grounding, reproducibility, and cost, supporting capability-level evaluation rather than treating systems as monolithic agents.The hypotheses received different levels of directional support: H1 and H2 were consistent with selected findings, H3 was not clearly supported, and H4 was broadly consistent.

11 Threats to Validity

The findings are exploratory and should be interpreted cautiously because validity depends on the operational metrics, controlled evidence setting, synthetic corruption, limited datasets and models, reproducibility conditions, and researcher-defined implementation choices.

  • Construct validity: The metrics approximate safe reference-free cleaning rather than capturing every downstream use; original-data metrics describe behavior, evidence use, cost, and reproducibility, not accuracy.Synthetic precision, recall, F1-score, expected-repair match, and valid-signal preservation are meaningful only when injected ground truth is available.
  • Evidence and citation validity: Controlled local retrieval and automated scoring limit claims about independent source-quality verification and external evidence grounding.Low citation-alignment values and frequent citation of profiling or executable-check evidence further constrain those claims.
  • Internal validity: Synthetic corruption may be more regular, visible, or reversible than natural errors, while unadjudicated original-data outputs cannot support detection or repair accuracy claims.A0 and A1 are contrasting baselines rather than a single-factor ablation because they differ in reasoning mechanism and profiling capabilities.
  • External validity: Results may not generalise beyond finance, healthcare, and environmental monitoring, especially when ten supplied records cannot represent long temporal patterns, rare events, or entity-level constraints.Generalisability across models and providers also remains unresolved because all LLM-based runs used one LLM and provider setting.
  • Reproducibility and temporal validity: The local corpus improves reproducibility but limits evidence freshness, while model updates, routing, moderation, and infrastructure changes threaten temporal validity.The experiment records model configuration, provider, inference parameters, retry behavior, responses, run metadata, and postprocessed metrics for auditing.
  • Implementation and evaluation bias: Implementation and evaluation may be biased by bugs, hidden assumptions, and researcher judgement in scripts, corruption scenarios, taxonomies, policies, metrics, and evidence-ranking weights.Saved artifacts, execution logs, output audits, and explicit scoring logic reduce—but do not eliminate—these risks.

12 Conclusion

The study presents an evidence-grounded framework for reference-free data cleaning and evaluates seven configurations across 126 successful runs. Results show capability trade-offs rather than consistent improvements, with no configuration dominating detection, repair, safety, evidence grounding, reproducibility, and operational cost.

  • Study scope and framework: 126 successful runs evaluated the framework across three datasets, two evaluation settings, and seven configurations.The framework combines structured context, profiling, LLM reasoning, executable checks, controlled evidence retrieval, source ranking, citation alignment, conservative repair, reversible scripts, and provenance logging.
  • Empirical findings: The deterministic profiling baseline achieved the highest synthetic detection F1-score, while executable tools improved detection within the progressive LLM-based sequence.The results did not show consistent improvements across all metrics.
  • Evidence grounding: Source-ranking and citation controls were associated with low unsupported-rule rates, but decision-level citation alignment remained weak.The source-ranked configurations achieved the lowest unsupported-rule rates, while citation alignment remained weak.
  • Conservative repair and cost: The full conservative configuration avoided unsafe and unnecessary modifications, although those values were already zero in A5, and A6 performed no successful direct repairs.The full conservative configuration also introduced the highest overall runtime and token use.
  • Future work: Future work should broaden datasets, models, and evidence providers while improving evidence alignment, repair evaluation, semantic reproducibility, expert review, and controlled knowledge-category ablations.The proposed ablations would independently add or remove schema, temporal, entity, domain, policy, task, and risk knowledge.

Declarations

The author reports no conflicts of interest, no new human-participant recruitment or data collection, and limited AI assistance for manuscript preparation rather than experiments or results.

  • The author declares no conflicts of interest.
  • The study used publicly available datasets and did not recruit human participants or collect new human-subject data.No additional informed consent was sought for this secondary analysis.
  • AI tools assisted with language editing and manuscript organisation, while the author verified the scientific content and remained responsible for it.AI tools were not used to execute experiments or generate experimental data or results.

Data and Code Availability

The study uses publicly available datasets and provides a blinded repository containing the materials needed to review the reported experiments.

  • The datasets are publicly available from sources cited in the manuscript.
  • A blinded repository provides the source code, experiment scripts, configuration files, controlled evidence corpus, and saved experimental artefacts.The repository supports review of the materials used to produce and inspect the reported results.
Loading 2608.14765v1…