Source-linked AI summary
Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems
Srimonti Dutta, Akshata Kishore Moharir
TL;DR
Answer accuracy alone cannot establish that an LLM data agent performed the requested structured computation, because correct answers may rest on invalid traces. The paper defines Trace Integrity, operationalizes it with execution contracts, and introduces CAIT Rate; its proof-of-concept shows these reliability signals diverge.
Problem
Answer-only evaluation does not reliably show whether a benchmark-correct structured-data answer is supported by the computation the user requested.
Method
The paper defines Trace Integrity and operationalizes it through execution contracts that bind user intent to schema elements, operator plans, assumptions, queries, verification status, and final answers.
Results
Answer accuracy, Trace Integrity Pass Rate, and CAIT Rate were distinct signals across Direct SQL, Operation Summary + SQL, and Contract-First SQL on BIRD Mini-Dev.
Takeaways & Limitations
LLM data agents should be evaluated by whether their returned answers are backed by computation that can be inspected, replayed, and audited.
Takeaways & Limitations
The proof-of-concept uses 100 stratified BIRD Mini-Dev examples, one model, and fixed prompting and execution, so its absolute rates are not stable rankings of models or prompt formats.
Abstract
from arXiv · showhide
Answer accuracy is an insufficient reliability signal for LLM data agents. In structured-data tasks, a benchmark-correct answer can be produced by an invalid trace. This paper introduces Trace Integrity, a deployment reliability criterion for evaluating whether the computation recorded behind an answer is explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable. We identify the Structure Gap as the deployment failure mode that makes Trace Integrity necessary: natural-language reasoning and free-form rationales do not reliably specify the operator-level programs required by real-world systems. We operationalize Trace Integrity with execution contracts, structured artifacts that bind user intent to schema elements, operator plans, assumptions, executable queries, verification status, and final-answer linkage. We also introduce CAIT (Correct Answer / Invalid Trace) Rate, which measures how often answer-only evaluation counts computationally unsupported outputs as successes. In an empirical demonstration on BIRD Mini-Dev, Direct SQL, Operation Summary + SQL, and Contract-First SQL achieve answer accuracies of 20%, 22%, and 24%, while their Trace Integrity Pass Rates are 39%, 43%, and 40% and their CAIT Rates remain high at 55%, 59.1%, and 45.8%, showing that answer accuracy, trace validity, and silent-failure risk are distinct evaluation signals. Real-world LLM data agents should, therefore, be evaluated not only by whether their outputs match a reference answer, but by whether those outputs are backed by auditable computation.
1 Introduction
LLM data agents can produce benchmark-correct answers from computations that are not faithful, inspectable, or supported by the recorded trace. The paper introduces Trace Integrity, execution contracts, and CAIT Rate to evaluate this deployment risk beyond answer accuracy.
- Answer-only evaluation cannot distinguish a faithful computation from an accidental success involving wrong filters, joins, aggregations, time windows, grouping keys, or schema bindings.
- Operational users need an inspectable artifact connecting schema elements, filters, joins, aggregations, assumptions, and execution results to the answer.
- Trace Integrity requires the recorded computation to be explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable.
- CAIT Rate measures the fraction of correct answers backed by invalid computations, exposing computationally unsupported successes that answer-only evaluation counts.
- The Structure Gap is the failure of natural-language reasoning to specify the operator-level programs required by structured-data tasks.
- 20.0%, 22.0%, and 24.0% answer accuracy coexisted with 39.0%, 43.0%, and 40.0% Trace Integrity Pass Rates and 55.0%, 59.1%, and 45.8% CAIT Rates across the three methods.The methods were Direct SQL, Operation Summary + SQL, and Contract-First SQL, respectively, on 100 BIRD Mini-Dev examples.
2 The Trace Integrity Problem
Structured-data reliability depends on translating natural-language intent into explicit operator commitments rather than plausible explanations alone. The central hidden failure is a correct answer whose invalid trace receives credit under answer-only evaluation.
- Structured-data questions require explicit commitments about included records, entity alignment, measures, grouping, temporal windows, and ordering.For average revenue per active customer, both numerator and denominator must be defined; “last quarter” must resolve to a time interval.
- The Structure Gap is the mismatch between natural-language reasoning and operator-level computation such as filtering, joining, grouping, aggregation, ranking, temporal restriction, and schema binding.
- Plausible reasoning can omit a join, change an average into a sum, apply a filter to the wrong table, or use a semantically different grouping field.
- Silent failures look like normal completions when cleanly executing queries or rationales conceal omitted filters or incorrect grouping keys.
- Answer accuracy collapses faithful successes, visible structural failures, data or execution issues, ambiguities, and correct answers with invalid traces into one outcome.
- CAIT Rate asks what fraction of correct answers are backed by invalid computations, while Trace Integrity evaluates whether the recorded computation is checkable, replayable, and auditable.
3 Trace Integrity as an Audit Contract
Trace Integrity treats the recorded computation behind a structured-data answer as an auditable artifact rather than private reasoning or a free-text explanation. Execution contracts bind intent to schema, operators, queries, verification, and answer linkage so validators can check and reviewers can inspect the computation.
- Trace Integrity dimensions: Trace Integrity requires traces to be explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable.These dimensions define whether the recorded computation can be checked, replayed, and inspected.
- Execution contracts: An execution contract records user intent, schema elements, operator plans, assumptions, execution queries, verification status, and final answers.Its purpose is to expose computational commitments to validators, auditors, and downstream answer generators without exposing hidden reasoning.
- Execution contracts: A compact contract can specify tables, join keys, filters, grouping, metrics, operator sequence, and verification status before execution.The example includes a customer-revenue join, active and non-trial filters, last-quarter scope, regional grouping, a per-customer metric, sorting, and top-1 selection.
- Validation: Validators can check schema existence, required filters, join keys, metric denominators, and whether the final answer follows from the executed result.These checks connect the contract’s declared computation to the executable query and final output.
- Validation: Contract validation can fail on schema bindings, operators, assumptions, permissions, contract-query consistency, or answer-trace consistency before a fluent answer is presented.Specific failure modes can be reviewed by humans to decide whether an answer should be shown, blocked, or escalated.
- The Isolation Principle: The Isolation Principle has agents specify intended computation before value-level data access, separating planning from execution by default.Planning from the request, schema, metadata, and policy context reduces retrospective completion of underspecified plans after seeing result values.
4 Empirical Demonstration: Measuring Hidden Trace Failures
On 100 BIRD Mini-Dev examples, the evaluation compares answer correctness with execution and trace-level validity across three prompting conditions. The results show that correct answers can remain computationally unsupported, while trace diagnostics expose failures hidden by answer-only evaluation.
- Evaluation setup: The proof-of-concept evaluates Direct SQL, Operation Summary + SQL, and Contract-First SQL on 100 BIRD Mini-Dev examples.Gold and generated SQL are executed against the same database, and generated traces are compared with normalized gold traces.
- Metrics and validation: CAIT Rate measures correct answers whose traces fail integrity, identifying computationally unsupported successes counted by answer-only evaluation.Ncorrect∩invalid counts correct final answers with failed traces, while Ncorrect counts all correct final answers.
- Results: 20.0%, 22.0%, and 24.0% are the Answer Accuracies for Direct SQL, Operation Summary + SQL, and Contract-First SQL, respectively.These values are reported alongside trace-level metrics rather than treated as sufficient evidence of valid computation.
- Results: 39.0%, 43.0%, and 40.0% are the Trace Integrity Pass Rates for Direct SQL, Operation Summary + SQL, and Contract-First SQL, respectively.Operation Summary + SQL has the highest Trace Integrity Pass Rate among the three conditions.
- Results: 55.0%, 59.1%, and 45.8% are the CAIT Rates for Direct SQL, Operation Summary + SQL, and Contract-First SQL, respectively.Contract-First SQL has the lowest CAIT Rate, whereas Operation Summary + SQL has the highest.
- Failure analysis: Across 300 predictions, 51 queries fail to execute, while validation also localizes operator, schema, contract-query, and answer-trace mismatches.These diagnostics distinguish visible execution failures from trace failures that answer-only evaluation can hide.
5 Deployment Implications and Conclusion
Trace Integrity is presented as a deployment criterion and artifact for making data-agent computations inspectable, replayable, and useful in operational review. The proof-of-concept shows that answer accuracy, trace validity, and silent-failure risk are distinct properties.
- Deployment Implications: Execution contracts can store answers, validate computations, flag answer-trace disagreements, and turn failed traces into regression tests.The same artifact supports analyst and compliance review, debugging, incident response, and schema-drift monitoring.
- Conclusion: The proof-of-concept shows that answer accuracy, Trace Integrity Pass Rate, and CAIT Rate do not measure the same property.
- Conclusion: Correct answers can be supported by invalid computations that answer-only evaluation does not expose.The result motivates evaluating whether outputs are backed by auditable computation, not only whether they match a reference answer.
- Deployment Implications: LLM data agents should leave behind computations that can be inspected, replayed, challenged, and converted into regression objects when they fail.This distinction is framed as important for settings where structured-data answers influence decisions.
6 Limitations
The study is a scoped proof-of-concept rather than a comprehensive benchmark, and its deterministic validator does not establish full semantic equivalence. Execution failures are also distinct from the silent failures targeted by CAIT.
- Scope: The experiment uses 100 stratified BIRD Mini-Dev examples, one model, and a fixed prompting and execution setup.Its absolute rates should not be interpreted as stable rankings of models or prompt formats.
- Validation: Deterministic operator-level checks make evaluation reproducible and allow failures to be attributed to concrete trace issues.Examples include missing joins, wrong aggregations, invalid schema references, grouping errors, and answer-trace mismatches.
- Validation: Structural validation is not full semantic-equivalence checking, so semantically valid SQL rewrites may be penalized against a normalized gold trace.BIRD Mini-Dev can admit multiple valid SQL programs for the same question.
- Interpretation: CAIT Rate is a diagnostic signal of computational support rather than a final semantic judgment on every individual query.
- Interpretation: Execution failures contribute meaningfully to trace-integrity failures but are distinct from the silent-failure case targeted by CAIT.The central claim is most directly supported when execution succeeds but the recorded computation remains invalid.
7 Ethical Considerations
The paper frames Trace Integrity as an ethical and technical safeguard against unsupported computational outputs in decision-support settings. It also limits the artifact’s role by preserving governance, privacy, and human oversight boundaries.
- Data Use: The study uses benchmark questions, schemas, generated SQL, execution results, and derived trace artifacts, without private user data or newly collected personally identifying information.
- Risks: Fluent or benchmark-correct answers can conceal invalid computations, creating a risk that users trust unsupported results in real-world decision-support settings.Trace-level evaluation makes joins, filters, aggregations, assumptions, execution results, and answer-trace consistency available for inspection.
- Governance: Trace artifacts create governance responsibilities and should record computational commitments rather than hidden chain-of-thought or unrestricted data snapshots.
- Governance: A valid trace supports responsible review but does not determine whether underlying data are sufficient for a downstream decision.High-stakes uses should combine trace artifacts with data governance, human review, and institutional oversight.