PaperlayerView exact revision on arXiv

Source-linked AI summary

Leakage-Resistant Evaluation of AI-Generated Database Specifications in Specification-Driven Migration

Oleg Grynets, Ivan Fedorchuk, Vasyl Lyashkevych

arXiv:2610.12778v1cs.SEcs.DBcs.LO

TL;DR

Evaluating generated specifications against their source remains distinct from evaluating whether the evaluator can locate known defects under controlled conditions. LRFB introduces a leakage-resistant benchmark and tests LLM judges against trivial baselines. Judges matched a specification-length baseline on binary detection but supplied category and evidence information the shortcut cannot provide.

  • Problem

    Existing evaluation approaches provide limited testing of evaluators against known specification defects, especially for omissions and under leakage-resistant conditions.

    Source: G. Research Gap · p. 3

  • Method

    LRFB uses a sealed benchmark of labelled source–specification pairs to compare LLM judges with trivial baselines and score detection, categories, affected items, and evidence.

    Source: §I · p. 1, B. Dataset Construction · p. 5, E. Measures · p. 6

  • Results

    On 247 held-out records, judges’ balanced accuracy ranged from 0.81 to 0.89 versus 0.88 for length; categories appeared in 84%–92% and evidence overlapped in 92%–93%.

    Source: §VIII · p. 10

  • Takeaways & Limitations

    The intermediate specification should be treated as a generation contract only after its evaluator has survived rigorous validation.

    Source: §VIII · p. 10

    The silver-standard synthetic benchmark lacks exhaustive independent expert verification, and its retained length confound prevents interpreting detection as semantic reasoning.

    Source: §VII · p. 9

Abstract

from arXiv · show

This paper presents a leakage-resistant framework and benchmark (LRFB) for measuring evaluators of AI-generated Oracle PL/SQL specifications. The benchmark contains 494 labelled source-specification pairs across 19 quality metrics grouped into seven dimensions; each metric contains 26 records divided into development and held-out subsets of 13 records each, with three clean and ten defective specifications per subset. Ground truth is mechanically sealed from inference and checked with canary and label-permutation tests. Three LLM judges were evaluated with three runs per record and compared against two trivial baselines. On 247 held-out records, GPT-4o, DeepSeek V4 Pro, and Claude Sonnet 4.5 reached balanced accuracies of 0.87, 0.89, and 0.81, respectively, while a specification-length baseline reached 0.88. Exact paired tests did not separate any judge from that baseline. The judges nevertheless provided information unavailable to length: the labelled defect category appeared among findings in 84%-92% of violated records, and quoted evidence overlapped the labelled span in 92%-93%. The study therefore reframes specification evaluation from binary issue detection toward evidence-linked, typed findings and benchmark-validity controls.

I. INTRODUCTION

Specification-driven migration makes the intermediate specification an operational artifact, yet existing methods largely assess generated artifacts rather than evaluators against labelled defects. LRFB addresses this measurement gap with a leakage-resistant benchmark that tests evaluator findings beyond binary verdicts.

I. INTRODUCTION

  • In database migration, fluent specifications may omit critical source behavior or invent operations, making the resulting target code behaviorally wrong.Examples include omitted OUT parameters, transaction side effects, NULL-sensitive conditions, and trigger timing constraints.Source: §I · p. 1
  • LRFB tests evaluators without label leakage, compares them with trivial shortcuts, and measures more than binary issue detection.The paper frames its contribution as methodological, noting that record identifiers encoded labels and specification length predicted many benchmark labels.Source: §I · p. 1
  • The paper defines 19 metrics, supplies a 494-pair benchmark, compares three LLM judges with two trivial baselines, and emphasizes category, affected-item, and evidence agreement beyond issue detection.These structured measures are more informative when a surface heuristic can match the judges.Source: §I · pp. 1–2
  • The evaluator compares recovered specifications with source artifacts and produces evidence-linked findings before target-code generation, which is outside the experiment.The findings support defect identification and subsequent correction.Source: §I · p. 2, Figure 1 · p. 2

II. RESEARCH TRAJECTORY AND RELATED WORK

Prior work motivates specification quality assurance, while related evaluation approaches leave gaps in omission detection, evaluator validation, and localized findings. This section positions LRFB against those strands.

A. From Specification-Mediated Migration to Specification

  • Migration and portability studies [1], [2], [3] motivate treating specifications as first-class artifacts that need quality assurance before reuse.Specification-driven DevOps also shows that executable artifacts can omit deployment intent absent from the repository [4].Source: A. From Specification-Mediated… · p. 2

B. Reference-Free Faithfulness and Completeness

  • Reference-free factuality methods are precision-oriented, while reversing QAFactEval’s question direction makes it recall-like for coverage but adds question-generation burden.A short specification can be faithful while omitting much of the source behavior.Source: B. Reference-Free Faithfulness and… · p. 2

C. LLMs as Judges

  • LLM judges can produce typed, localized findings, but their verdicts, categories, locations, and evidence still require validation.Prior work also reports concerns including prompt sensitivity, position and verbosity bias, calibration, and run-to-run variability [14],.Source: C. LLMs as Judges · p. 3

D. Code–Text Alignment and Program Representations

  • Embedding metrics offer deterministic alignment alternatives,, but global scalar outputs do not provide the localized migration findings review needs.Examples of needed findings include a missing ELSE branch or an invented COMMIT.Source: D. Code–Text Alignment and Program… · p. 3

E. Evaluation in Code Generation and Text-to-SQL

Code-generation benchmarks emphasize task-specific oracles, while long-context and shortcut effects complicate evaluation. LRFB addresses gaps in intermediate-specification assessment and benchmark validity.

E. Evaluation in Code Generation and Text-to-SQL

  • Code-generation and text-to-SQL benchmarks use executable or semantic oracles, whereas LRFB evaluates recovered natural-language specifications rather than target code [25],,,,.The shared methodological principle is tying evaluation to a task-specific oracle rather than surface plausibility.Source: E. Evaluation in Code Generation and… · p. 3

F. Long Context, Shortcut Signals, and Benchmark Validity

  • Prompt compression can reduce cost, but preserving task-relevant information is not guaranteed; LLM judges also exhibit characteristic biases,,,,.Long-context models may use relevant information unevenly depending on its position.Source: F. Long Context, Shortcut Signals, and… · p. 3

G. Research Gap

  • LRFB targets four gaps: evaluating intermediate specifications, testing evaluators on known typed defects, measuring omissions, and checking leakage and shortcut baselines.Its focus is measuring the evaluator rather than proposing another generation metric.Source: G. Research Gap · p. 3

A. Source Facts and Specification Claims

The framework separates source facts from specification claims to assess both omitted behavior and unsupported assertions. Its metric system organizes specification-quality judgments into dimensions and evaluates metric-specific outputs.

A. Source Facts and Specification Claims

  • Coverage captures represented relevant source facts, while faithfulness captures whether atomic specification claims are supported; neither quantity alone suffices.A specification can be faithful yet incomplete, or cover much source behavior while adding unsupported claims.Source: A. Source Facts and Specification Claims · p. 3

A. Metric System

  • The benchmark groups 19 metrics into seven specification-quality dimensions, with ten issue categories per metric and a fixed category vocabulary supplied to the evaluator.The dimensions include source coverage, faithfulness, completeness, neutrality, consistency, clarity, and migration-related risks.Source: A. Metric System · p. 4, Table II · p. 4
  • Experiments evaluate individual metric-specific judgments, not downstream aggregation strategies or explainability mechanisms.Fig. 2 depicts seven evaluator groups whose findings may later be aggregated.Source: Figure 2 · p. 4, A. Metric System · p. 5

B. Dataset Construction

The benchmark comprises 494 synthetic source–specification pairs across 19 metrics, split into development and held-out subsets. Its controlled design is not intended to represent industrial defect prevalence.

B. Dataset Construction

  • Each of 19 metrics has 26 pairs, yielding 494 records; six are clean and 20 contain one deliberately introduced defect from ten issue categories.Each source is a 4–29-line Oracle artifact, and each specification contains one to three sentences.Source: B. Dataset Construction · p. 5
  • Each metric’s 13-record development and held-out halves contain one example per issue category and three clean records, with shared-source records kept in the same half.The benchmark is a controlled silver dataset, not a claim about natural industrial defect prevalence.Source: B. Dataset Construction · p. 5

C. Ground-Truth Boundary

The framework isolates ground truth from inference, validates structured findings, and evaluates both detection and evidence-linked defect identification. Its measures address repeated judgments and the benchmark’s class imbalance.

C. Ground-Truth Boundary

  • Inference excludes labels and label-encoding record identifiers; canary and label-permutation tests check that ground truth stays out of requests and outputs.Labels enter only during scoring, as Fig. 3 illustrates.Source: C. Ground-Truth Boundary · p. 5, Figure 3 · p. 5

D. LLM Judge and Structured Findings

  • Judges return structured findings, while a deterministic layer flags issues only when the validated finding list is non-empty and checks quotations against exact input text.Out-of-schema responses or categories are invalid; deterministic and hybrid prototypes are excluded from the reported comparison.Source: D. LLM Judge and Structured Findings · p. 5

E. Measures

  • Three judgments per record are combined by majority vote; unresolved cases and records without valid responses count as incorrect.TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives.Source: E. Measures · p. 6
  • Because 20 of 26 records per metric are violated, an always-issue evaluator has 0.77 raw accuracy but 0.50 balanced accuracy.Balanced accuracy is preferred to ordinary accuracy.Source: E. Measures · p. 6
  • Beyond detection, LRFB measures category presence, first-finding status, affected-item identification, and overlap between quoted evidence and the labelled span.Affected-item identification requires at least 0.5 content-token recall; span overlap uses case and whitespace normalization.Source: E. Measures · p. 6

F. Trivial Baselines

The study uses always-issue and specification-length baselines as deliberately weak comparisons. The length baseline also probes whether detection accuracy demonstrates source-grounded reasoning.

F. Trivial Baselines

  • Two baselines are evaluated on the same records: always-issue labels every record violated, while length labels specifications below a per-metric threshold as violated.The length threshold is fitted on development data and frozen for held-out evaluation.Source: F. Trivial Baselines · p. 6
  • The length baseline is a benchmark-validity probe: if it performs like an LLM reading source and specification, accuracy alone cannot demonstrate source-grounded reasoning.It is not intended as a serious evaluator.Source: F. Trivial Baselines · p. 6

G. Experimental Protocol

On 247 held-out records, the specification-length baseline matched the judges’ balanced accuracy, and exact paired tests found no significant judge–baseline difference. Results also report the experimental setup and an exploratory complete-dataset comparison.

G. Experimental Protocol

  • Three judges were evaluated at temperature 0 with schema-constrained outputs, identical prompts, and three repetitions per record; the primary held-out analysis comprised 247 records.The judges were GPT-4o, Claude Sonnet 4.5, and DeepSeek V4 Pro; majority voting determined final verdicts.Source: G. Experimental Protocol · p. 6

A. Issue Detection

  • Sensitivity was 0.99–1.00, but specificity varied from 0.61–0.79 across judges, with the authors cautioning against generalizing from the synthetic benchmark’s surface correlations.Fig. 4 presents the held-out balanced-accuracy results with 95% intervals.Source: A. Issue Detection · p. 6, Figure 4 · p. 7

B. Paired Comparison with the Length Baseline

  • Exact paired tests found no held-out judge–length difference; DeepSeek V4 Pro’s complete-set advantage over Claude Sonnet 4.5 (p = 0.017) did not survive Bonferroni correction.The adjusted significance threshold was 0.0083 for six pairwise comparisons, so the complete-set difference is exploratory rather than conclusive.Source: B. Paired Comparison with the Length… · p. 7, Table IV · p. 7

C. Metric-Level False Alarms

Binary detection can reward a length shortcut, while structured findings provide category, affected-item, and evidence agreement; metric-specific false alarms and a label-vocabulary caveat qualify these results.

C. Metric-Level False Alarms

  • In omission-oriented metrics, removing information made violated specifications shorter and gave length a surface cue; additive or substitutive defects made length less predictive.The shortcut arises from benchmark construction rather than a demonstrated general property of specifications.Source: C. Metric-Level False Alarms · p. 7
  • False-alarm patterns differed by metric and judge, while the length baseline performed strongly on omission-oriented metrics, supporting metric-specific calibration over one aggregate score.GPT-4o was weakest on Unresolved References and Source Question Answerability; DeepSeek V4 Pro on Risk Explanation Completeness, Data Transformation Completeness, Semantic Migration Risks, and Source Question Answerability; Claude Sonnet 4.5 on Unresolved References, Ambiguous Statements, Source Fact Coverage, and Terminology Drift.Source: C. Metric-Level False Alarms · p. 8, Figure 5 · p. 8

D. Beyond Binary Detection

  • 84%–92% of held-out violated-record findings included the labelled category, 70%–76% identified the affected item when applicable, and 92%–93% overlapped the labelled evidence span.These structured measures require pointing to a category, object, or text span, unlike length-based detection.Source: D. Beyond Binary Detection · p. 8
  • Claude Sonnet 4.5 produced the most findings and most often included the labelled category, but this exhaustiveness coincided with the highest false-alarm rate.The benchmark exposes a precision–thoroughness trade-off that balanced accuracy alone hides.Source: D. Beyond Binary Detection · pp. 8–9
  • Terminology Drift category-agreement results are optimistic because the allowed category names reveal the drifting terms.The vocabulary therefore supplies candidate answers rather than independent evidence of diagnosis.Source: D. Beyond Binary Detection · p. 9

E. Run Stability

Repeated runs still disagreed on some held-out records. The discussion argues for moving beyond verdict accuracy toward actionable findings and leakage-resistant assurance.

E. Run Stability

  • 6/247 GPT-4o, 3/247 DeepSeek V4 Pro, and 2/247 Claude Sonnet 4.5 held-out records differed across runs.Across all 494 records, disagreement counts were 10, 7, and 5, respectively; majority voting changed only a small fraction of verdicts.Source: E. Run Stability · p. 9

A. What the Experiment Does and Does Not Demonstrate

B. Updated Scientific Trajectory

  • The paper adds benchmarked specification assurance as a fourth stage in the research trajectory, before specifications become trusted generation contracts.Source: B. Updated Scientific Trajectory · p. 9
  • Migration pipelines should separate source-fact extraction, neutral-specification generation, evidence-linked evaluation, and target-dialect synthesis.The evaluator should be replaceable so deterministic extraction, question answering, embeddings, and LLM judges can share a benchmark interface.Source: B. Updated Scientific Trajectory · p. 9

C. Implications for Benchmark Design

  • Benchmark design should match clean and defective records on surface statistics, seal labels from every inference channel, and add expert-validated multi-defect annotations with independent evidence spans.Source: C. Implications for Benchmark Design · p. 9

D. Implications for Migration Engineering

On held-out records, LLM judges were not separated from a specification-length baseline, though their typed findings included defect categories and evidence spans. The authors emphasize evaluator limits and the need for stronger benchmark validation.

D. Implications for Migration Engineering

  • With sensitivity at 0.99–1.00, the practical optimization target becomes specificity and finding precision, balancing missed defects against review costs and false alarms.Source: D. Implications for Migration… · p. 9
  • Evidence attached to findings can be inspected, corrected, and linked to migration tasks before specifications feed target-dialect generation.Source: D. Implications for Migration… · p. 9

VII. LIMITATIONS

  • The silver-standard benchmark lacks exhaustive expert verification, and its small self-contained Oracle examples omit dependencies common in industrial objects.Unlabelled defects and debatable injected labels are possible; industrial objects may involve schemas, packages, session state, external routines, privileges, and cross-file data flow.Source: §VII · p. 9
  • Only 13 held-out records per metric support descriptive metric-level results; pooled inference assumes metric comparability and does not model source-artifact clustering.Only 57 held-out clean records also make specificity more uncertain than sensitivity.Source: §VII · p. 9
  • The retained length confound prevents interpreting binary detection as evidence of semantic reasoning.The authors recommend length-matched adversarial pairs and negative controls for other superficial signals.Source: §VII · p. 9
  • The comparison covered only LLM judges that read source and specification; other evaluator families remain untested, and some metric definitions may misfit clean cases.Development-data calibration may reduce false alarms.Source: §VII · p. 10

VIII. CONCLUSION

  • 0.81–0.89 balanced accuracy for LLM judges versus 0.88 for the specification-length baseline; no judge was statistically separated from it.Source: §VIII · p. 10
  • 84%–92% of held-out violated records included the labelled defect category among judge findings, and 92%–93% had quoted evidence overlapping the labelled span.The authors call for expert validation, length-matched records, more clean cases, multi-defect annotations, and comparisons with other evaluator types through the sealed protocol.Source: §VIII · p. 10

Related papers

Loading 2610.12778v1…