PaperlayerView exact revision on arXiv

Source-linked AI summary

Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation

Jingjing Liu, Dishu Yang, Hongbo Qin, Hansong Zhang

arXiv:2610.15797v2cs.AIcs.SE

TL;DR

Plan-based DataFrame generation can carry a mistaken interpretation across plans, implementations, and tests, so passing tests may reflect shared assertions rather than task fidelity. The paper compares three pipelines and audits prompt-derived obligations across generated artifacts. Test-passing suites still failed reference agreement on tasks with reproduced references, leading the authors to argue for acceptance evidence tied to the task contract.

  • Problem

    Plan, program, and tests may share an incomplete interpretation, making test agreement insufficient to establish that generated code matches the task.

    Source: §I · p. 1

  • Method

    The study compares three complete pipelines on TPC-DS-derived tasks, auditing prompt-derived obligations in plans, PySpark programs, and tests while reserving hidden reference outputs for post-hoc evaluation.

    Source: §I · p. 1

  • Results

    All 77 test-passing suites on the 28 tasks with SQL-reproduced stored references remained reference mismatches.

    Source: §I · p. 1

  • Takeaways & Limitations

    Acceptance assertions should derive from explicit task contracts and remain fixed during revision, with requirement-linked checks and discriminating fixtures.

    Source: B. Verification Implications · p. 6

    Feedback, grounding, budgets, and editing permissions vary together, precluding an isolated audit effect; unretained pre-adjudication labels prevent reconstructing human agreement.

    Source: C. Scope and Validity · p. 6

Abstract

from arXiv · show

Planning and revision are intended to bring generated code closer to the task specification. Yet a mistaken interpretation can persist across the plan, implementation, and tests - a pattern we call interpretive lock-in. We investigate this risk on 30 DataFrame tasks derived from TPC Benchmark DS, comparing Frozen One-Pass, Test-Guided Revision, and PlanAudit-Guided Revision. Test-pass counts rose from 22/30 to 29/30 and 30/30, yet no generated solution matched its stored reference output. Evaluator controls returned the expected outcome in 57/57 cases. Same-engine SQL re-execution reproduced 28/30 references; all 77 test-passing runs on these tasks still failed reference agreement. PlanAudit-Guided Revision improved code scores on 3/30 tasks and test coverage on 11/30, but the median paired change was zero for both measures. Failure analysis revealed tests that omitted requirements, adopted candidate-defined schemas, used nondiscriminating fixtures, or repeated the implementation's incorrect logic. Fresh generation on ten targeted tasks produced 0/30 reference matches and 25/25 false passes among test-passing runs. These findings are diagnostic rather than causal: the pipelines differ in feedback, grounding, call budgets, and editing permissions, and the study uses one model and one primary generation per eligible task. Because pre-adjudication human labels were not retained, inter-rater agreement cannot be reconstructed and the mechanism analyses remain exploratory. Overall, increasingly successful self-generated tests did not establish semantic correctness, even after excluding unreproduced references.

I. INTRODUCTION

Plan-based DataFrame code generation can carry one incomplete interpretation across the plan, program, and tests. This study examines that risk across three pipelines and defines interpretive lock-in as shared omissions or mistaken requirements that persist through revision.

I. INTRODUCTION

  • DataFrame behavior depends on table choice, joins, filters, grouping grain, metrics, schema, and ordering, so a shared incomplete interpretation can persist across artifacts.Passing tests establish agreement with assertions under that interpretation, not that it matches the task.Source: §I · p. 1
  • Interpretive lock-in is the persistence of shared omissions or mistaken requirements through revision, not a claim about internal model cognition.It requires evidence that the same mistake survives revision; a false pass alone is insufficient.Source: §I · p. 1
  • The study compares three complete pipelines on 30 TPC-DS-derived tasks and examines reference agreement, within-task obligation coverage, and tests passing despite reference mismatch.Its contributions include semantic-obligation retention as a plan-mode measure, a prompt-derived audit pipeline, and a paired, single-realization comparison.Source: §I · p. 1
  • On 28 tasks whose stored references were reproduced through SQL re-execution, all 77 test-passing suites remained reference mismatches.The study also uses positive and negative evaluator controls and sensitivity analyses to separate the verification gap from unresolved reference provenance.Source: §I · p. 1

II. RELATED WORK

Prior work addresses planning, intent clarification, generated-test reliability, and semantic evaluation through several approaches. The study evaluates DataFrame tasks derived from TPC-DS and audits prompt-derived obligations in generated artifacts.

II. RELATED WORK

  • Self-planning separates planning from implementation, while TiCoder uses interactive judgments on generated tests to clarify user intent.This study instead tracks requirements across a model-generated plan, implementation, and test suite without user clarification during revision.Source: §II · pp. 1–2
  • Generated-test reliability work addresses debugging, test selection [5], or training against self-collusion [6], whereas this study traces missed requirements across artifacts and compares suites with a withheld SQL reference.Source: §II · p. 2
  • Semantic evaluation includes API constraints, database test suites, consistency checks, and requirement-linked tests and repairs [10]; PlanAudit-DF audits natural-language obligations without formal guarantees.It distinguishes prompt-inferable requirements from hidden output contracts and reserves SQL-reference comparison for post-revision evaluation.Source: §II · p. 2

A. Benchmark and Task Construction

  • Thirty eligible TPC-DS-derived tasks were retained from 103 query variants, with the same SF=1 database used across conditions.Eligibility required a stateless solve function, unmodified execution on Spark 4.1.2, and permitted DataFrame API use; 73 tasks were excluded.Source: A. Benchmark and Task Construction · p. 2
  • Prompts specified inputs, joins, filters, aggregations, schema, ordering, and requested artifacts without revealing SQL; prompts and reference outputs were frozen before generation.The authors derived 334 reference obligations and independently labeled coverage and failure mechanisms after runs froze, but did not retain pre-adjudication labels for agreement analysis.Source: A. Benchmark and Task Construction · p. 2

B. Conditions and Comparative Scope

The three conditions share an initial generation, while differing in feedback, grounding, budgets, and revision permissions. These are pipeline-level comparisons rather than isolated tests of one component.

B. Conditions and Comparative Scope

  • Each condition starts with one generation call and reuses its artifacts; protocol-level call budgets exclude pytest and Spark executions.Grounding, feedback, and revision permissions define the pipeline-level comparison.Source: B. Conditions and Comparative Scope · p. 2
  • Frozen One-Pass produces all artifacts from the task and input schemas, then runs tests and a non-evaluative smoke check without revision.Test-Guided Revision receives pytest failures and may revise code plus syntax, import, or fixture additions, but cannot add gold-derived assertions and leaves the plan frozen.Source: B. Conditions and Comparative Scope · p. 2
  • PlanAudit-Guided Revision uses up to eight calls for obligation extraction, assessment, schema grounding, artifact audit, and repair recommendations before revision.Its audit checks the one-pass plan, code, and tests for coverage or contradiction; revision can run for at most two cycles and stops when tests pass.Source: B. Conditions and Comparative Scope · p. 2

C. Agent Information Isolation

Each condition uses a separate directory and reset context, with reference materials and evaluation results withheld from the audit and revision stages. These controls are procedural, and access compliance was not verified.

C. Agent Information Isolation

  • A preflight check excludes reference SQL and outputs, human labels, scores, and other-condition artifacts, except the shared one-pass snapshot; audit and revision artifacts freeze before external evaluation.Operating-system enforcement and access compliance were not verified.Source: C. Agent Information Isolation · p. 3

D. Execution Protocol

The protocol specifies the generation environment and artifact controls, then defines reference agreement, obligation scores, and paired analyses. Robustness checks vary reference comparison and task cohorts; results are descriptive.

D. Execution Protocol

  • Generation used Claude Code with reported model settings and a Spark/PySpark environment, but invocation counts and token usage were unavailable.No cost or efficiency comparison was made.Source: D. Execution Protocol · p. 3
  • Materials are available from the corresponding author upon reasonable request, including run exports, scripts, reference reports, and a source-file registry.The validation report documents provenance, implementation checks, and remaining artifact gaps.Source: D. Execution Protocol · p. 3

E. Evaluation and Validation

  • Reference agreement requires matching column names, types, order, row counts, and row multisets, while requested row order is not evaluated.A false pass is a successful pytest run without reference agreement.Source: E. Evaluation and Validation · p. 3
  • Obligation scores weight partial plan and test coverage at 0.5, while execution failures receive zero code score.Means use all 30 task scores per condition; a self-confirming omission requires an absent or contradicted plan, violated code, and uncovered test obligation.Source: E. Evaluation and Validation · p. 3
  • Paired changes subtract each task's inherited one-pass score, and the analyses report counts, medians, and quartiles without hypothesis tests or population-rate intervals.Binary outcomes come from execution-comparison records, while scores come from final obligation labels.Source: E. Evaluation and Validation · p. 3
  • Robustness checks vary numeric tolerances and comparison normalization, and separately exclude q11 or retain only SQL-reproduced tasks.The post-hoc PlanAudit ladder applies exact comparison, alias normalization, numeric casting, then row sorting.Source: E. Evaluation and Validation · p. 3

IV. RESULTS

Across 30 tasks per condition, test passing increased while reference agreement remained absent. Evaluator controls returned expected outcomes, but paired score changes were generally zero and internal scores did not imply reference agreement.

A. Evaluator Validation

  • 57/57 evaluator controls returned expected outcomes, including 36 acceptances and 21 rejections, with all 24 perturbations marked verified.Row permutations were accepted, while numeric and null mutations failed multiset comparison.Source: A. Evaluator Validation · pp. 3–4, Table II · p. 4

B. Reference Agreement and Internal Metrics

  • 0/30 candidates matched references in every condition, while test-pass counts rose from 22/30 to 29/30 and 30/30.Every passing suite was a false pass; execution succeeded in 28/30, 29/30, and 29/30 runs.Source: Table III · p. 4, B. Reference Agreement and Internal… · p. 4, Figure 1 · p. 4
  • All paired median score changes were zero, and only PlanAudit test coverage had a nonzero IQR, at 8.33 percentage points.Mean gains reflect improvements in a subset rather than across most tasks.Source: B. Reference Agreement and Internal… · p. 4, Table IV · p. 4
  • Excluding q11, PlanAudit gains remained on 3/29 plans, 2/29 code scores, and 10/29 test scores, with zero median changes.Its mean code gain fell from 4.28 to 3.28 percentage points.Source: B. Reference Agreement and Internal… · p. 4
  • PlanAudit’s primary failure categories included alias mismatch (15), structural schema (4), row count (3), precision (3), values (3), column order (1), and runtime (1).These surface categories are not mutually exclusive root causes; 11 hidden-reference contract mismatches concerned prompt-inferable requirements.Source: B. Reference Agreement and Internal… · p. 4
  • PlanAudit’s 4.28-point code gain spanned q4, q11, and q2, while test coverage rose 6.45 points without recovering reference agreement.Test-Guided Revision’s 2.33-point code-mean rise came entirely from q2’s recovery from execution failure.Source: B. Reference Agreement and Internal… · p. 4
  • Among runs with at least 90% test coverage, 11, 12, and 17 lacked reference agreement across the three conditions.High plan semantic ratios also did not imply agreement: 17, 17, and 19 runs lacked it.Source: B. Reference Agreement and Internal… · p. 4, Figure 1 · p. 4

C. SQL Re-execution and Reference Sensitivity

SQL re-execution reproduced 28 of 30 study references, leaving two references unresolved. Restricting analyses to reproduced tasks did not remove the separation between test passing and reference agreement.

C. SQL Re-execution and Reference Sensitivity

  • The q2 reference has 53 rows versus 2,513 on re-execution, and the q35 outputs have differing schemas despite both containing 100 rows.Both references are attributed to task-specific construction whose code is unavailable, leaving provenance unresolved.Source: C. SQL Re-execution and Reference… · p. 4
  • Of 33 reference-inventory SQL queries, all executed successfully and 31 outputs matched the stored schema and multiset; three matches were outside the primary study, leaving 28/30 reproduced study references. All 77 passing suites remained false passes.In the restricted cohort, there were 28/28 passing suites and 0/28 reference matches in every condition.Source: C. SQL Re-execution and Reference… · pp. 4–5
  • On the 28-task subset, mean code scores were 82.21%, 82.21%, and 84.30%, while mean test coverage was 81.34%, 85.09%, and 88.25%.PlanAudit improved 4/28 plan, 2/28 code, and 11/28 test scores; paired medians remained zero.Source: C. SQL Re-execution and Reference… · p. 5
  • After additionally excluding q11, both revision pipelines passed every suite without a reference match, and all paired medians remained zero.PlanAudit improved 3 plan, 1 code, and 10 test scores; its sole remaining code improvement was q4.Source: C. SQL Re-execution and Reference… · p. 5

D. Failure Mechanisms

The failure analysis distinguishes missed obligation coverage from tests that pass despite defects. It identifies omissions and weak assertions, with case studies showing requirements absent from plans or tests and incorrect logic accepted by suites.

D. Failure Mechanisms

  • Among violated obligations in false-pass runs, no test covered 19/39 Frozen One-Pass cases and 24/52 PlanAudit-Guided cases.The remaining 20/39 and 28/52 had tests that passed despite the defect; these totals concern different run sets, not paired defects introduced by revision.Source: D. Failure Mechanisms · p. 5, Table VI · p. 5
  • Missing checks included untested planned obligations and omissions spanning plan, code, and tests, while inadequate checks included candidate-defined schemas and nondiscriminating fixtures.Some assertions repeated the implementation’s incorrect logic.Source: D. Failure Mechanisms · p. 5
  • In q33, all 11 tests passed in each condition although aggregation obligation O8 was absent from the plan, violated in code, and untested.Other cases included q14a’s alias mismatch, q4’s incorrect price column, and q11’s untested cross-channel alignment requirement.Source: D. Failure Mechanisms · p. 5

E. Normalization and Fresh-Generation Replication

Fresh generation produced no reference matches, while every test-passing run remained a false pass. Normalization and case analyses also exposed persistent value, schema, formula, and join errors.

E. Normalization and Fresh-Generation Replication

  • 0/30 outputs matched references across any fresh-generation condition or query.The targeted replication generated new artifact triples for ten selected tasks.Source: E. Normalization and Fresh-Generation… · p. 5
  • 27/30 outputs failed full normalization; only q13 and q17 were genuine alias-only mismatches, while q11’s apparent recovery reflected run drift.All three precision cases retained value differences after casting.Source: E. Normalization and Fresh-Generation… · p. 5
  • 25/25 test-passing runs were false passes, and the other five runs also failed reference agreement.Failures included omitted channel labels, incorrect rollup NULL semantics, and nondiscriminating price-formula fixtures.Source: E. Normalization and Fresh-Generation… · p. 5
  • Structural errors survived complete suites, and a PlanAudit revision changed an outer join to an inner join while all 21 tests passed.The change reduced the code semantic ratio from 50% to 40%.Source: E. Normalization and Fresh-Generation… · pp. 5–6

V. DISCUSSION

Revision improved executability and obligation retention without reference agreement, with gains concentrated in a minority of tasks. The discussion proposes contract-derived, fixed assertions and fixtures that discriminate among plausible interpretations.

A. Interpretive Lock-In Across Revisions

  • Revision improved executability and obligation retention without reference agreement; median paired changes were zero, locating gains in a minority of tasks.In targeted cases, revisions repaired execution errors or a join fixture while retaining semantic mismatches; q5 also gained a join error.Source: A. Interpretive Lock-In Across Revisions · p. 6
  • Both sensitivity restrictions preserved the test/reference gap despite smaller code gains, including after excluding references with unresolved construction provenance.The central verification finding therefore does not depend on those two references.Source: A. Interpretive Lock-In Across Revisions · p. 6

B. Verification Implications

  • The authors propose deriving acceptance assertions from explicit task contracts before candidate inspection and holding them fixed during revision.Linking each assertion to a requirement exposes missing checks.Source: B. Verification Implications · p. 6
  • Fixtures should discriminate alternatives using unmatched join keys, unequal price fields, and multiple groups; hidden tests, differential execution, and metamorphic relations offer complementary evidence.Contracts should distinguish schema, null behavior, grouping, channel labels, and ordering from value assertions.Source: B. Verification Implications · p. 6

C. Scope and Validity

The conclusion is that stronger test passing and obligation retention did not establish reference agreement, even on sensitivity subsets. The authors identify limits to validity and propose matched, prospective comparisons with independent validation.

C. Scope and Validity

  • Comparator controls and SQL re-execution support comparator behavior and repeatability, not independent semantic correctness; shared execution errors require independent implementations or hand-checked assertions.Pipeline feedback, grounding, budgets, and editing permissions vary together, precluding an isolated audit-effect estimate.Source: C. Scope and Validity · p. 6
  • Generality is limited by one model, one primary generation per eligible task, one database, and one environment; targeted replication establishes recurrence, not population robustness.Eligibility was not independently double-coded.Source: C. Scope and Validity · p. 6

D. Next Comparisons

  • Future studies should cross feedback, grounding, and editing permissions at matched call budgets, with fixed assertions, independent validators, prospective sampling, and reference-positive candidates.Preregistered obligation rubrics and retained pre-adjudication labels would enable human-agreement assessment.Source: D. Next Comparisons · p. 6

E. Conclusion

  • Higher test passing and obligation retention did not recover reference agreement, including on the 28-task and 27-task sensitivity subsets.The authors conclude that acceptance should require evidence tied to the task contract, not only assertions repeating the candidate’s interpretation.Source: E. Conclusion · p. 6

Related papers

Loading 2610.15797v2…