PaperlayerView exact revision on arXiv

Source-linked AI summary

PaperCompiler: Faithful Paper-to-Code Generation via Repository-Level Specification Compilation

Yunhao Liu, Hong Phuc Pham, Jaehong Yoon

arXiv:2609.02272v1cs.CLcs.AIcs.SE

TL;DR

Paper-to-code systems must preserve implicit paper-specific requirements across coherent repositories, but free-form intermediates can weaken or reinterpret those requirements. PaperCompiler compiles grounded evidence into explicit specifications with provenance, ownership, dependencies, and constraints, then generates repositories under them; it improves reference-based fidelity by 13.8% and reduces high-severity evaluator critiques from 13.2% to 6.1%.

  • Problem

    Paper-to-code generation must recover implicit implementation details while preserving method logic, evaluation protocols, and cross-file consistency.

  • Method

    PaperCompiler grounds paper evidence and compiles it into repository-level specifications encoding provenance, non-degradation requirements, ownership, dependencies, and file-level constraints.

  • Results

    13.8% relative improvement in reference-based fidelity, from 3.647 to 4.152, accompanied by high-severity evaluator critiques falling from 13.2% to 6.1%.

  • Takeaways & Limitations

    Reliable paper-to-code generation requires preserving paper-derived requirements, assigning file-level responsibilities, and maintaining cross-file consistency.

  • Takeaways & Limitations

    Fine-grained API and schema consistency remains an interface-alignment challenge for future work.

Abstract

from arXiv · show

Faithfully translating research papers into repository-level implementations remains challenging because papers often describe methods at a high level, leave implementation assumptions implicit, and require generated repositories to preserve method logic, evaluation protocols, and cross-file consistency. Despite recent advances in paper-to-code agents, their intermediate outputs are often presented as free-form plans or summaries that downstream coding agents may ignore, reinterpret, or compress, leading to algorithmic simplification and inconsistent repository structure. To address these challenges, we introduce PaperCompiler, a paper-to-code generation framework that compiles paper-grounded evidence into explicit repository-level implementation specifications. PaperCompiler grounds implementation-relevant evidence while preserving source provenance and distinguishing paper-supported, inferred, externally delegated, and unresolved information. The resulting specifications encode non-degradation requirements, ownership assignments, cross-file dependencies, and file-level constraints. Repository generation proceeds under these compiled specifications while retaining flexibility over local engineering choices not fixed by the paper. PaperCompiler outperforms strong baselines on Paper2CodeBench, achieving a 13.8% relative improvement in reference-based fidelity (from 3.64 to 4.15) and reducing high-severity evaluator critiques (from 13.2% to 6.1%).

1 INTRODUCTION

Paper-to-code generation must recover implicit implementation details while preserving methodological fidelity and repository consistency. PaperCompiler addresses this by compiling grounded evidence into persistent specifications, improving benchmark fidelity and reducing severe failures.

  • Papers leave implementation-critical details implicit, requiring systems to recover preprocessing, initialization, and evaluation decisions while maintaining repository-wide fidelity.
  • Free-form plans and summaries can lose or reinterpret requirements because they do not bind implementation obligations to responsible repository components.
  • PaperCompiler links each paper-grounded requirement to its implementation location while keeping uncertain details explicit and coordinating cross-file dependencies.
  • Its three phases ground evidence, compile explicit repository-level requirements and ownership, and generate files under cross-file consistency constraints.
  • 13.8% relative improvement in reference-based fidelity, from 3.647 to 4.152, accompanies a reduction in high-severity failures from 13.2% to 6.1%.

2 RELATED WORK

Prior work combines paper analysis, multimodal extraction, retrieval, planning, memory, debugging, and repository-aware generation, but paper-to-code remains distinct from general software synthesis. Paper2CodeBench specifically evaluates alignment between generated repositories and source papers.

  • PaperCoder, AutoP2C, AutoReproduce, and DeepCode use staged planning, multimodal extraction, lineage, execution feedback, retrieval, memory, and iterative correction.
  • Repository-level generation research addresses distributed context through cross-file retrieval, repository-aware training, dependency-aware planning, and component validation.
  • ML-Bench, MLE-bench, RE-Bench, PaperBench, and Paper2CodeBench target different combinations of repository coding, long-horizon engineering, research tasks, replication, and paper-code alignment.
  • The Universal Neural Functionals case study shows PaperCoder retaining one partition candidate while PaperCompiler enumerates valid partitions and assembles the corresponding basis blocks.

3 PAPER-TO-REPOSITORY GENERATION VIA SPECIFICATION COMPILATION

PaperCompiler treats repository generation as controlled transformation from paper evidence to coordinated file-level specifications. Grounding preserves provenance and references; compilation assigns obligations and artifact ownership; generation follows dependency-compatible constraints.

  • 3 PAPER-TO-REPOSITORY GENERATION VIA SPECIFICATION COMPILATION: Repository synthesis must preserve method, data processing, execution, and evaluation information while distinguishing grounded facts from inferred decisions.
  • 3 PAPER-TO-REPOSITORY GENERATION VIA SPECIFICATION COMPILATION: PaperCompiler separates what the paper supports, what must remain preserved or unresolved, and where each requirement belongs in the repository.
  • 3.1 PAPER GROUNDING: BLUEPRINT CONSTRUCTION AND REFERENCE EXTRACTION: Paper Grounding builds an implementation blueprint and reference registry, recording source locations, evidence status, and implementation roles for atomic items.
  • 3.2 SPECIFICATION COMPILATION: FROM RECONCILED REQUIREMENTS TO FILE-LEVEL CONTRACTS: Specification Compilation transforms grounded evidence into reconciled requirements, an ownership graph, and localized file-level specifications.
  • 3.2 SPECIFICATION COMPILATION: FROM RECONCILED REQUIREMENTS TO FILE-LEVEL CONTRACTS: Architecture synthesis assigns requirement ownership and artifact producers and consumers, producing an acyclic generation graph while allowing runtime cycles.
  • 3.2 SPECIFICATION COMPILATION: FROM RECONCILED REQUIREMENTS TO FILE-LEVEL CONTRACTS: File-level contracts retain interfaces, implementation recipes, artifact relations, handoffs, dependencies, references, and explicit unresolved or non-degradation constraints.
  • 3.3 CONSTRAINT-GUIDED REPOSITORY GENERATION: Constraint-Guided Repository Generation creates files in dependency-compatible order using each file specification, committed upstream code, and downstream compatibility constraints.

4 EXPERIMENTS

PaperCompiler achieves the strongest reported Paper2CodeBench performance while improving per-paper win rates and reducing implementation failures. Its gains are associated with explicit specifications, ownership, cross-file dependencies, and constraints against method-degrading simplifications, though API/schema mismatches remain a challenge.

  • Main Results: PaperCompiler achieves the highest scores across conference subsets and protocols, improving reference-free evaluation from 4.562 to 4.777.It also improves P2C-Ex from 4.535 to 4.728 and reference-based fidelity from 3.647 to 4.152.
  • Main Results: PaperCompiler attains higher per-paper win ratios than PaperCoder across all three conference subsets and evaluation protocols.The largest margin occurs under reference-based evaluation, and the gains are broadly distributed across papers.
  • Comparison with End-to-End Systems: PaperCompiler records the highest average score and per-paper win rate against AutoP2C, AutoReproduce, and PaperCoder on the ten-paper subset.The margins are particularly large under P2C-Ex and reference-based evaluation.
  • Performance-Efficiency: PaperCompiler uses 1.71M tokens per repository on average, compared with 0.98M for PaperCoder, while AutoP2C uses a comparable 1.68M tokens but scores substantially lower.Average usage corresponds to approximately $1.88–$7.51 per generated repository at current o3-mini API rates.
  • Failure Analysis: PaperCompiler reduces algorithmic degradation from 28.0% to 24.6% and high-severity critiques from 13.2% to 6.1%.Missing core components fall from 12.3% to 6.8%, and evaluation mismatches from 13.4% to 8.4%.
  • Ablation Study: Removing Reconciliation reduces reference-based performance from 4.38 to 3.86, while removing Contracting reduces it to 3.92.These ablations indicate the importance of carrying method-level obligations into file-level specifications.

5 CONCLUSION

PaperCompiler frames faithful paper-to-code synthesis as a repository-level specification-compilation problem. Its conclusion emphasizes explicit control over paper-derived requirements, their propagation, and their consumption during generation.

  • The framework preserves paper-derived requirements, assigns file-level responsibilities, and maintains cross-file consistency.
  • PaperCompiler shows consistent gains over a strong staged baseline across reference-free, P2C-Ex, and reference-based evaluation.
  • Ablations identify Reconciliation and Contracting as the most important components, while diagnostics show fewer severe semantic and method-completeness failures.
  • Reliable paper-to-code generation requires explicit control over how paper-derived information is represented, propagated, and consumed during repository construction.

A LIMITATIONS

PaperCompiler’s current scope is limited by its reliance on text-parsed paper content and by evaluation protocols that do not establish complete reproduction or universal executability. The paper presents multimodal understanding, retrieval, tool interaction, and automated validation as future directions.

  • PaperCompiler may miss implementation-critical information conveyed through architecture diagrams, complex figures, visual examples, or other non-textual forms.
  • When parsed text omits details, the system may rely on the underlying LLM’s parametric knowledge to resolve implicit implementation gaps.
  • Specialized external tools, proprietary APIs, simulators, undocumented benchmark conventions, and unavailable auxiliary resources remain challenging settings.
  • The evaluation does not establish full reproduction of reported experimental results or executability across all repositories, seeds, and model families.
  • Requirements and file-level specifications provide generation-time guidance rather than formal correctness guarantees.

B EXTENDED EXPERIMENTAL DETAILS

PaperCompiler’s repository synthesis follows three conceptual phases: grounding, specification compilation, and constraint-guided generation. Algorithm 1 instantiates these phases through evidence preservation, ownership-aware file specifications, and dependency-respecting generation.

  • Algorithm 1 summarizes PaperCompiler’s end-to-end repository synthesis procedure.
  • Paper Grounding: Paper Grounding constructs an implementation blueprint and preserves requested source materials in a reference registry.
  • Specification Compilation: Specification Compilation reconciles evidence, builds a repository ownership graph, and produces localized file-level specifications.
  • Specification Compilation: File-specific context is extracted by slicing the compiled requirements, repository graph, and reference registry for each file.
  • Constraint-Guided Repository Generation: Files are generated in the topological order induced by the ownership graph while respecting committed interfaces and cross-file compatibility constraints.
  • The qualitative diagnostics illustrate how systems can instantiate, weaken, or miss paper-specific implementation structures, but are not statistical evidence.

D.1 Universal Neural Functionals: ALGORITHMIC BASIS-CONSTRUCTION COLLAPSE.

The Universal Neural Functionals case contrasts a structurally connected implementation that collapses partition-based basis construction with PaperCompiler’s separated modules preserving the required algorithmic path.

  • The target repository structure includes specification parsing, UNF layer generation, layer assembly, channel extension, and training or experiment orchestration.
  • Valid partitions must be enumerated and converted into basis elements for the equivariant layer.
  • The P2C rerun returns only the trivial partition or first available candidate, collapsing the basis family despite a connected flat repository path.
  • PaperCompiler separates specification parsing, basis generation, layer assembly, and multi-channel extension into distinct modules.
  • Its generated basis-generator module preserves partition enumeration and constructs corresponding basis functions consumed by layer assembly.
  • The file-level specification forbids hard-coded or heuristic defaults and preserves the valid-partition mapping needed by the basis generator.

D.2 iTransformer: REPOSITORY-STRUCTURE DIAGNOSTIC.

The iTransformer diagnostic shows that repository organization and experimental coverage vary across systems even when the central variate-as-token method is preserved. PaperCompiler improves explicit modular routing but still compresses parts of the broader protocol.

  • AutoReproduce preserves the variate-as-token transformation and connected training–evaluation path while simplifying the surrounding data pipeline and experimental framework.
  • P2C preserves input transposition and the Transformer backbone but compresses the repository into six flat files and simplifies the data pipeline.
  • PaperCompiler separates configuration, data loading, variate embedding, Transformer blocks, projection, training, evaluation, and orchestration into a connected src/ pipeline.
  • The case demonstrates a trade-off among model-path preservation, repository organization, and experimental coverage rather than superiority of PaperCompiler’s directory structure.

D.3 INTL: TRAINING-PIPELINE PRESERVATION

The INTL case shows PaperCompiler preserving a connected training topology by routing IterNorm, losses, and orchestration across dedicated files. It improves cross-file dependency fidelity but does not reproduce every extension or detail.

  • The INTL target requires connected augmentation, backbone, projection, IterNorm whitening, composite losses, training, and k-NN evaluation.
  • AutoP2C includes an iter_norm component, but it is not properly connected to the projection head or training loop.
  • P2C implements IterNorm but whitens across the sample dimension, producing covariance over the batch axis.
  • AutoReproduce connects augmentation, representation learning, iterative normalization, the training objective, and evaluation, while simplifying mathematical and evaluation details.
  • PaperCompiler routes IterNorm, composite losses, and training orchestration into dedicated files and connects whitening with the loss and evaluation pipeline.
  • PaperCompiler leaves EMA and multi-crop extensions incompletely integrated because they are treated as optional extensions.

D.4 SEABO: A FAILURE-ORIENTED DIAGNOSTIC CASE

SEABO exposes a gap between preserving a method’s intended modular topology and realizing its executable external protocol. PaperCompiler routes the main offline-RL workflow explicitly, but unsupported runtime paths remain high-severity failures.

  • D.4 SEABO: A FAILURE-ORIENTED DIAGNOSTIC CASE: The SEABO target requires D4RL loading, expert extraction, nearest-neighbor reward annotation, offline-RL training, and normalized-return evaluation.
  • D.4 SEABO: A FAILURE-ORIENTED DIAGNOSTIC CASE: P2C connects expert extraction, KD-tree construction, reward computation, offline-RL training, and evaluation, preserving state-only support and achieving a perfect reference-based score.
  • D.4 SEABO: A FAILURE-ORIENTED DIAGNOSTIC CASE: AutoReproduce connects the main SEABO path in one file but narrows the method to one configuration and simplifies reward, training, and evaluation protocols.
  • D.4 SEABO: A FAILURE-ORIENTED DIAGNOSTIC CASE: PaperCompiler separates data loading, KD-tree construction, reward annotation, offline-RL training, evaluation, and orchestration into dedicated modules.
  • D.4 SEABO: A FAILURE-ORIENTED DIAGNOSTIC CASE: The runner falls back to a dummy dataset without a D4RL path, while the state_only=True trainer raises NotImplementedError.
  • D.4 SEABO: A FAILURE-ORIENTED DIAGNOSTIC CASE: The failure indicates that file-level specifications can preserve intended topology without completing external-protocol and runner integration.

F.2 SELECTED BLUEPRINT CONSTRUCTION AND REFERENCE EXTRACTION OUTPUTS

The selected outputs trace iTransformer from blueprint construction and reference extraction through routed implementation contracts. They preserve axis inversion, token-wise processing, and dimension-sensitive forecast generation as explicit constraints.

  • Blueprint Construction: The blueprint identifies multivariate time-series forecasting, records inverted tokenization, and separates the main method from optional ablations and analysis.
  • Reference Extraction: The pipeline loads split-protocol data, transposes X from T×N to N×T, and produces X_transposed for embedding.
  • Reference Extraction: The embedding MLP maps each variate row of length T to a token vector of dimension D, producing H^0 ∈ R^(N×D).
  • Reference Extraction: Each Transformer block computes attention across variate tokens, applies token-wise FFNs, and preserves an N×D token matrix.
  • Reference Extraction: After L blocks, a projection MLP maps each token from R^D to R^S and transposes outputs into forecasts of shape R^(S×N).
  • Reference Extraction: Training computes MSE loss against future series, backpropagates, and updates parameters with ADAM; evaluation reports MSE and MAE over forecast lengths.
  • Reconciled Contracts: The contracts forbid temporal attention, cross-token FFNs, and cross-time normalization, requiring attention and LayerNorm to operate per variate token.

F.4 SELECTED ARCHITECTURE OUTPUT

PaperCompiler turns global method requirements into concrete, file-owned contracts and generated code for the Transformer Block. The resulting artifacts preserve axis inversion, token-wise processing, interfaces, and cross-file handoffs.

  • Architecture: The Architecture stage assigns contracts CR2–CR4 to src/transformer_block.py and routes attention, FFN, and LayerNorm through public APIs.The file serves as the implementation boundary for these subcomponents.
  • Artifact handoff: The transformer output is handed from src/transformer_block.py to src/projection_module.py through the orchestrator while preserving token structure.The handoff forbids incorrect dimension handling and references contracts CR2, CR3, and CR4.
  • Implementation recipes: LayerNorm operates per token, maintaining token identity and the (N×D) output shape without side effects or state mutation.The contract associates this behavior with CR4 and validates it during generation.
  • Implementation recipes: The FFN applies an MLP independently to each row of an (N×D) token matrix and returns updated tokens with the same shape.Acceptance checks explicitly require per-token processing and shape preservation.
  • Compact handoff to coding: The compiled handoff requires the Transformer Block to consume embedded tokens, produce Transformer Block Output (H^L), and fail fast on shape or contract violations.Its required functions are compute_self_attention, apply_ffn, and apply_layer_norm.
  • Generated code: The generated Transformer Block uses separate LayerNorm modules and a residual connection around the FFN before returning the normalized output.The code excerpt shows ffn_output, residual addition, and self.norm2.
  • Implementation recipes: Attention must compute softmax((QKT)/√dk) with an (N×N) map, preserving the paper’s inverted attention over variate tokens.The contract validates input shape (N×D) and requires the attention map to retain shape (N×N).
Loading 2609.02272v1…