PaperlayerView exact revision on arXiv

Source-linked AI summary

SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction

Xue Hu, Zewei Pan, Zeli Su, Zhou Liu, Wentao Zhang

arXiv:2608.24252v1cs.AIcs.SE

TL;DR

LLM-generated reproduction repositories can execute while silently diverging from scientific specifications, and existing benchmarks do not systematically diagnose those deviations. SA-Bench decomposes papers into atomic, verifiable claims and evaluates repositories across four drift dimensions, finding systematic semantic drift across models and scaffolds. The results point toward semantic specification verification as a key direction for narrowing the alignment gap.

  • Problem

    Paper-to-code reproduction requires semantic fidelity beyond executable code, but deviations from distributed paper specifications lack a systematic diagnostic evaluation.

  • Method

    SA-Bench uses human-verified Semantic Alignment Units to score repositories statically against paper claims across numerical, methodological, protocol, and ordering drift.

  • Results

    Semantic drift is systematic across tested configurations, while executability-focused scaffolds provide limited leverage for scientific reproduction.

  • Takeaways & Limitations

    Semantic specification verification is a key direction for future paper-to-code agents because executability alone is insufficient for scientific reproduction.

  • Takeaways & Limitations

    SA-Bench covers 30 ICLR, ICML, and NeurIPS 2025 papers across five ML domains, so findings may not directly generalize to other fields.

Abstract

from arXiv · show

LLM agents can generate paper reproduction code, yet often produce scientifically unfaithful implementations. We define this failure mode as semantic drift, where generated code silently diverges from the paper's specifications. We introduce SemanticAlign-Bench(SA-Bench), a diagnostic benchmark covering 30 papers from ICLR, ICML and NeurIPS 2025. For each paper, we decompose its specifications into atomic and verifiable implementation claims, which we call Semantic Alignment Units (SAUs) and evaluate repositories along four diagnostic dimensions spanning numerical, methodological, protocol and ordering drift. In total, we construct 1,491 SAUs across five ML domains and evaluate 12 generator configurations (4 models $\times$ 3 scaffolds). Even the strongest configuration (Claude+PaperCoder) achieves a mean SAU score of only 0.301 out of 1.0, with an overall mean of 0.221 across 360 evaluations. A failure taxonomy reveals that agents attempt most requirements but implement them incorrectly, with implementation mismatch and stubs accounting for the majority of zero-scored claims. Our analysis further indicates that scaffolds optimized for executability provide limited leverage for scientific reproduction; narrowing the gap requires scaffolds that prioritize semantic specification verification. The benchmark, annotations and evaluation pipeline are publicly available.

1 Introduction

Paper-to-code reproduction requires semantic fidelity beyond executable code, but existing evaluations do not systematically diagnose deviations from paper specifications. SA-Bench addresses this gap with claim-level diagnostic evaluation across a broad benchmark of papers and generator configurations.

  • Motivation: Paper-to-code reproduction must faithfully implement algorithmic logic, numerical details, experimental protocols, and execution order, not merely produce executable repositories.The paper defines this requirement as semantic fidelity: whether generated repositories implement what the paper actually claims.
  • Motivation: A single misread formula or omitted step can fundamentally alter a method even when the resulting code executes successfully.Critical specifications are often distributed across method sections, appendices, and tables.
  • Research Gap: Existing benchmarks assess reproduction quality but do not systematically classify deviations between generated code and paper specifications.Consequently, semantic drift can remain undiagnosed when reproduction falls short.
  • SA-Bench: SA-Bench evaluates semantic alignment through individual Semantic Alignment Units and a four-category taxonomy covering numerical, methodological, protocol, and ordering drift.Repositories are scored against verifiable claims without relying on a single scalar alone.
  • Benchmark Scale: 1,491 human-verified SAU claims from 30 papers across five ML domains support 360 evaluations of 12 generator configurations.The configurations combine four models with three scaffolds.

2 Related Work

Prior paper-to-code work spans specialized generation pipelines, reflective correction, and execution-based benchmarks. SA-Bench differs by diagnosing deviations at the level of atomic implementation claims using a structured drift taxonomy.

  • Paper-to-Code Generation: Paper-to-code systems have used staged planning, analysis, and coding; reflective self-correction; and paper-understanding pipelines feeding code generation.These approaches represent different ways to structure reproduction agents.
  • Existing Benchmarks: Existing benchmarks evaluate reproduction through rubric trees, executable unit tests, or complex scientific algorithm implementations.PaperBench uses author-developed rubric trees, while LMR-BENCH, SciReplicate-Bench, and SciCode use execution-based evaluation.
  • SA-Bench’s Distinction: SA-Bench scores individual Semantic Alignment Units on a five-level rubric and maps deviations to four categories of semantic drift.This connects claim-level scoring with diagnostic classification rather than only measuring reproduction outcomes.

3 SemanticAlign-Bench

SA-Bench formalizes static, claim-level semantic alignment evaluation through atomic SAUs, four drift dimensions, structured extraction, human review, and diagnostic scoring. Its judges inspect paper and repository text without executing code.

  • Task Formulation: The task takes a research paper and purported reproduction repository, then reports paper-level alignment with per-claim diagnostic judgments.Static inspection isolates semantic fidelity from environmental factors affecting execution.
  • SAU Design: Each SAU is an atomic implementation claim that is localizable to a paper span, verifiable by static code inspection, and independently assessable.These properties make paper specifications operational for claim-level evaluation.
  • Drift Taxonomy: D1–D4 classify numerical precision, method or formula, experimental protocol, and step-ordering deviations.The taxonomy ranges from value-level specifications to explicitly stated procedural ordering constraints.
  • Construction Pipeline: The extraction pipeline filters nonimplementable content, runs three dimension-specific specialist agents in parallel with per-section spawning, merges ordering claims, and applies human review.Reviewers can accept, revise, reject, split, merge, or add claims; approximately 17% of extracted candidates are rejected.
  • Scoring: 1,491 SAUs are scored with a five-level rubric, producing paper-level and per-dimension averages plus structured evidence and explanations.The judge searches repositories for claim-matching evidence and records observed alignment or deviation.
  • Scoring Reliability: The judge matched human assessment in approximately 87% of 200 manually verified claims, with disagreements concentrated in ambiguous D2 score boundaries.The ambiguity concerned incomplete versus missing core mechanisms.

4 Evaluation Setup

The evaluation crosses four model families and three agent scaffolds over all benchmark papers. It measures paper-level semantic alignment, per-dimension scores, and zero-score causes using an independent judging pipeline.

  • Evaluation Matrix: 360 paper-level evaluations combine 12 generator configurations formed from four models and three scaffolds across 30 papers.This design enables joint analysis of model capability and scaffold strategy.
  • Models: The models are Claude-Sonnet-4.6, DeepSeek-V4-Pro, Gemini-2.5-Flash, and GPT-4o, spanning multiple capability tiers.All models receive identical paper inputs and share the same tool interface.
  • Scaffolds: The scaffolds represent a minimal ReAct baseline, a dedicated paper-to-code pipeline, and a software-engineering execution scaffold.They are implemented as BasicAgent, PaperCoder, and OpenHands, respectively.
  • Protocol: The primary metric is paper-level SAS, complemented by SASD1–SASD4 diagnostic sub-scores and a zero-score decomposition.Each paper–repository pair is scored by GPT-5.5 to reduce self-evaluation bias.

5 Results and Analysis

Across 360 evaluations, semantic alignment is uniformly low, with model choice generally mattering more than scaffold choice. Performance varies by drift dimension and domain, while implementation mismatches and stubs account for most zero-scored claims.

  • 5.1 Overall Performance: 0.221 overall mean SAS across 360 evaluations, with the best configuration reaching 0.301.The best configuration is Claude-Sonnet-4.6 + PaperCoder.
  • 5.1 Overall Performance: Model choice outweighs scaffold choice on average, although scaffold benefits vary with base-model capability.PaperCoder helps weaker models most, while OpenHands shows a notable gain only on Gemini (+0.093).
  • 5.2 Dimension Analysis: D1 > D2 > D4 > D3 holds across all 12 configurations and across models and scaffolds.The ordering reflects increasing demands from copying numerical values to implementing full experimental protocols.
  • 5.3 Domain and Paradigm Analysis: 0.030 cross-domain spread in overall SAS versus a 0.219 gap between best and worst configurations, indicating larger generator than domain effects.Domain differences remain dimension-specific: NLP/LLM leads D2, probabilistic inference leads D1, and computer vision trails D3 and D4.
  • 5.4 Where Zero Scores Come From: Implementation mismatch accounts for 40.8% and stubs or placeholders for 16.2% of zero-scored claims.The taxonomy classifies 7,034 zero-scored claims, with external knowledge gaps contributing 8.0%.
  • 5.5 Discussion: Why Scaffolds Underperform and What to Build Next: Executability-focused scaffolds provide limited leverage because code can execute cleanly while implementing the wrong algorithm.PaperCoder grounds code in paper specifications but lacks a semantic verification loop.

6 Conclusion

SA-Bench diagnoses whether generated repositories faithfully implement paper specifications through claim-level evaluation across papers, models, and scaffolds. Results show systematic semantic drift and point toward semantic verification rather than executability as the key direction.

  • 6 Conclusion: SA-Bench evaluates paper-to-code faithfulness with 1,491 human-verified SAUs across 30 papers and five ML domains.The benchmark spans 360 evaluations across four models and three scaffolds.
  • 6 Conclusion: Mean SAS is 0.221 overall, while the best configuration reaches only 0.301.Implementation mismatches and stubs account for the majority of zero-scored claims.
  • 6 Conclusion: Scaffolds designed for general software engineering provide limited leverage for scientific reproduction, where executability alone is insufficient.The paper identifies semantic specification verification as necessary for closing the alignment gap.

7 Limitations

SA-Bench’s scalability and generalizability are constrained by expert annotation requirements and its focus on 30 papers from three venues and five ML domains.

  • 7 Limitations: Expert review is required to ensure SAU claim fidelity, limiting annotation scalability.Automatic or semi-automatic verification could enable broader benchmark coverage.
  • 7 Limitations: Findings may not directly generalize beyond ICLR, ICML and NeurIPS 2025 or beyond the five covered ML domains.Extending to fields such as biomedicine or theory requires new curation and domain-specific pipeline adaptation.

A Benchmark Paper List

The benchmark contains 1,491 SAU claims across 30 papers, with claim counts and drift-type composition varying substantially by paper.

  • A Benchmark Paper List: The paper list records each paper’s venue, total SAU count, and per-type breakdown.The benchmark paper list is summarized in Table 6.
  • A Benchmark Paper List: 1,491 SAU claims across 30 papers average 49.7 per paper, ranging from 22 to 94.The smallest paper has 22 claims, while the largest has 94.
  • A Benchmark Paper List: D1 has 523 claims and D2 has 503, followed by D3 with 300 and D4 with 165.Numerical precision and method/formula claims are the largest categories.

B SAU Data Format and Examples

The benchmark stores atomic, source-grounded implementation claims by drift dimension and illustrates the format with a complete NFIG annotation example. The example spans dataset, training, residual extraction, frequency analysis, and pipeline-ordering claims.

  • B SAU Data Format and Examples: 1,491 SAUs are organized across dimensions D1–D4 for the 30-paper benchmark.
  • B SAU Data Format and Examples: Each SAU is an atomic implementation claim that is localizable, statically verifiable, and independently assessable.
  • B.1 Example: nfig SAU Annotation: NFIG is used as the representative example for demonstrating per-paper SAU annotation.
  • B.1 Example: nfig SAU Annotation: The NFIG annotation records paper identity, D1 claim groups, claim identifiers, claim text, and source sections.
  • B.1 Example: nfig SAU Annotation: NFIG D2 claims encode residual token extraction through frequency-band quantization with cumulative residual tracking.
  • B.1 Example: nfig SAU Annotation: The NFIG D3 example compares NFIG with VAR-16 on ImageNet using PSD and FKS across low, middle, and high frequency bands.PSD is lower-is-better, while FKS is higher-is-better and is reported as Low 0.57, Mid 0.28, and High 0.15.
  • B.1 Example: nfig SAU Annotation: The NFIG D4 pipeline specifies training an FR-VAE tokenizer, applying residual quantization and VQGAN losses, generating tokens from low to high frequency, and decoding them.The example specifies 10 autoregressive steps before decoding with the FR-VAE decoder.

C Pilot Study Detailed Results

The pilot appendix details the evaluation setup, inductively derives four semantic-drift dimensions from failed rubric nodes, and describes the extraction architecture and review process. The results show that extraction frequently requires rejection or modification, especially when agents confuse cited prior methods with current-paper contributions.

  • C Pilot Study Detailed Results: All pilot runs used Claude Sonnet 4.6 with BasicAgent and were scored by the official PaperBench-dev pipeline with a GPT-4o judge.The agent used ReAct with max_steps=80 and a 900-second time limit.
  • C Pilot Study Detailed Results: 312 zero-scored leaf nodes from five papers were grouped inductively by the missing or incorrect paper specification they described.The four resulting patterns were value mismatches, algorithmic substitutions or omissions, protocol gaps, and ordering errors.
  • C Pilot Study Detailed Results: The extraction pipeline first applies a code-first filter, then runs three specialist agents over section-level sub-agents, and finally merges outputs while deriving D4 claims.D4 ordering claims come from annotations produced by the D2 and D3 agents rather than a separate extraction pass.
  • C Pilot Study Detailed Results: The Numerical Matcher targets implementation-relevant values by scanning numerically dense regions, pre-scanning numeric patterns, classifying hits, and grouping configurations.Its outputs include numerical value, parameter name, configuration group, and source section.
  • C Pilot Study Detailed Results: The Method Parser extracts code-implementable components and records explicit before-and-after ordering annotations for downstream D4 derivation.It reads method sections, pseudocode, formulas, and implementation details while limiting each method component to one or two claims.
  • C Pilot Study Detailed Results: The Protocol Enumerator binds each protocol to its comparison, data, baselines, and metrics, then validates merged outputs defensively.It also records phase-ordering annotations for later D4 processing.

E.2 Result Value Extraction

The review identifies result-value extraction as a fundamental semantic error and documents systematic confusion between implementation claims, drift dimensions, and claim granularity. It also shows that agents miss appendix-buried and operational specifications, requiring human correction.

  • Result Value Extraction: Agents extract benchmark outcomes as implementation specifications, confusing descriptive results with prescriptive requirements.This mistake led reviewers to delete clusters of pure result-value claims.
  • D1–D4 Type Misclassification: Architecture and workflow decisions are often mislabeled as numerical or methodological claims, especially when claims contain numbers or named components.A visual encoder’s attention and positional encoding design was retyped from D1 to D4, while a training recipe was retyped from D1 to D2.
  • Over-Splitting and Granularity Errors: Agents frequently fragment coupled implementation components into multiple claims that have no independent code existence.Reviewers merged three modality-specific MoE subclaims into one architecture claim and merged an HPD formula, pipeline, and λ search into one claim.
  • Omissions: What the Agent Misses: Agents systematically miss specifications buried in appendices, expressed in prose, or operationally implied by theoretical results.The review added an iteration schedule because the agent extracted the sampler but omitted the rule for choosing K.
  • Scoring Pipeline: The scoring pipeline uses dimension-aware prompts and structured evidence fields so judgments can distinguish implementation presence, absence, and incorrectness.The judge receives the claim, dimension type, source spans, and retrieved repository evidence, while reasoning records why the score was assigned.

J Case Analysis: Representative Failures

Representative cases show that agents substitute familiar templates for paper-specific designs across architecture, mechanisms, evaluation protocols, and ordered pipelines. The taxonomy exposes these failures, while aggregate analyses link difficulty more to specification structure than claim count.

  • Failure Taxonomy: 40.8% of zero-scored claims are implementation mismatches, while 16.2% are stubs or placeholders.Other categories include external knowledge gaps at 8.0% and token mentions at 3.7%; roughly 30% resist a single category.
  • D1: Numerical and Architectural Drift: A standard ViT-B/16 replaced SAM 2’s MAE-pre-trained hierarchical Hiera-B+ encoder, omitting its hierarchical structure and pretraining initialization.The case attributes the failure to defaulting to a familiar ViT implementation instead of implementing the specified Hiera architecture.
  • D2: Method and Formula Drift: Temporal concatenation and global self-attention replaced the paper’s pointwise spatial memory cross-attention mechanism.The substituted implementation flattened past-frame features and omitted the specified spatial memory attention.
  • D3: Experimental Protocol Drift: Evaluation scripts covered only 2 of 9 VOS datasets, leaving seven as placeholders and omitting zero-shot and interactive segmentation benchmarks.The case identifies dataset-specific downloading, preprocessing, and protocol implementation as an external knowledge gap.
  • D4: Step Ordering Drift: The three-phase SA-V data engine was collapsed into a flat dataset loader, erasing the required human annotation, propagation-and-correction, and automatic-generation sequence.The case labels this collapse as a canonical ordering failure when temporal relationships remain implicit in prose.
  • Paper Difficulty: Specification structure matters more than claim volume: claim count correlates negatively with SAS (r = −0.43, p < 0.05), but the relationship is noisy.The paper suggests that distributed prose specifications are harder than requirements concentrated in an algorithm box and hyperparameter table, while noting this locality explanation remains a hypothesis.
Loading 2608.24252v1…