Source-linked AI summary
SciCoQA: Quality Assurance for Scientific Paper--Code Alignment
Tim Baumgärtner, Iryna Gurevych
TL;DR
Paper-code discrepancies threaten reproducibility, yet their detection has not been systematically measured as scientific output scales. SCICOQA builds a cross-modal benchmark from real and synthetic discrepancies and evaluates LLMs, finding that the best models detect only 46.7% of real-world cases. The results position current LLMs as promising assistants but not autonomous verifiers of paper-code faithfulness.
Problem
Code availability does not ensure that implementations faithfully match scientific descriptions, creating a need to measure automated detection of paper-code discrepancies.
Method
SCICOQA combines GitHub issues and reproducibility papers with synthetic discrepancies across computational science domains, then benchmarks 22 LLMs on cross-modal verification.
Results
46.7% of real-world discrepancies are detected by the best-performing models, Gemini 3.1 Pro and GPT-5 Mini.
Takeaways & Limitations
Current LLMs show promise as assistants but cannot yet serve as autonomous verifiers of paper-code faithfulness.
Takeaways & Limitations
Real-world data is predominantly skewed toward Computer Science and Artificial Intelligence because paper-code discrepancy infrastructure is largely absent outside CS/AI.
Abstract
from arXiv · showhide
Discrepancies between scientific papers and their code undermine reproducibility, a concern that grows as automated research agents scale scientific output beyond human review capacity. Whether LLMs can reliably detect such discrepancies has not been systematically measured. To this end, we present SciCoQA, a dataset of 635 paper-code discrepancies (92 real, 543 synthetic) for this cross-modal verification task. Across 22 evaluated models, even the best-performing LLMs, Gemini 3.1 Pro and GPT-5 Mini, detect only 46.7% of real-world discrepancies, revealing a critical gap in automated scientific quality assurance. We construct SciCoQA from GitHub issues and reproducibility papers, and propose a synthetic generation pipeline to scale beyond AI to Physics, Quantitative Biology, and other computational sciences. We further introduce a taxonomy of discrepancy types and categories to characterize the occurring mismatches. Our analysis shows that models particularly struggle with omitted paper details, long-context inputs, and papers outside their pre-training corpus.
1 Introduction
Paper-code inconsistencies threaten reproducibility because code availability does not ensure faithful implementation, while manual verification is increasingly impractical as scientific output scales. SCICOQA addresses this need by benchmarking LLMs on cross-modal discrepancy detection, but the best models detect only 46.7% of real-world cases.
- Code availability does not guarantee reproducibility or consistency with a paper’s scientific description.Implementation divergences can introduce unreported performance variations and invalidate comparisons.
- Manual code review is time-consuming and increasingly impractical under growing submission volumes and agentic scientific production.Humans cannot verify the rapidly expanding volume of generated scientific output unaided.
- 46.7% of real-world discrepancies are detected by the best-performing models evaluated in SCICOQA.The paper reports insufficient recall despite high precision for the strongest systems.
- SCICOQA benchmarks LLMs on detecting semantic discrepancies between scientific papers and their code.The benchmark combines realistic sources with synthetic discrepancies spanning additional computational science domains.
2 Related Work
Prior work detects scientific errors mainly within papers or at local code-comment scales. SCICOQA extends this focus to global alignment between dense scientific papers and long, multi-file codebases, including constructed and real-world discrepancies.
- Earlier scientific-error studies primarily analyze papers alone and report difficulty detecting introduced errors.Some approaches use manually crafted or generated erroneous papers, while others use retraction notes and post-publication reports.
- SCICOQA treats paper-code checking as cross-modal claim verification with code serving as the evidence source.It targets deviations from paper descriptions and omissions affecting understanding or reproducibility.
- Code-comment inconsistency methods usually process a single function with an inline comment or docstring.SCICOQA instead requires reasoning over a dense paper and a long, multi-file repository.
- Execution success and high performance can still mask a method that fundamentally differs from the paper.SCICOQA supplies ground-truth paper-code discrepancies for evaluating faithfulness verification.
3 SCICOQA
SCICOQA defines meaningful paper-code discrepancies as semantic conflicts that prevent code from faithfully reproducing a reported method. It combines manually filtered real cases with synthetic discrepancies across computational domains and analyzes their types, categories, and long-context demands.
- Definition: A paper-code discrepancy is a semantic conflict that fundamentally alters the scientific logic, experimental protocol, or mathematical formulation.The definition excludes independent bugs, configurable default hyperparameters, and trivial engineering details.
- Data Collection: Real discrepancies come from GitHub issues and reproducibility papers, with candidates filtered manually and validated by Gemini 3.1 Pro and GPT-5.The real set contains 42 GitHub-issue discrepancies and 50 reproducibility-paper discrepancies.
- Data Analysis: 55% of real discrepancies are Differences, while Algorithm and Loss dominate real category assignments at 25% and 24%.In synthetic data, Differences account for 78%, and Algorithm is the largest category at 26%.
- Data Analysis: 56,903 tokens is the median combined length of papers and codebases, with 73 of 276 papers exceeding 100k combined tokens.The median lengths are 14,200 tokens for papers and 39,272 for codebases.
4 Experiments
SCICOQA prompts models to generate paper-code discrepancy lists and evaluates matches against reference discrepancies using an LLM judge validated against human annotations. The experiment measures recall across diverse commercial and open-weight model families.
- Prediction: Models receive a paper and its code and generate a list of discrepancies between them.A code-only ablation estimates which discrepancies require cross-modal reasoning rather than code or parametric knowledge alone.
- Evaluation: GPT-OSS 20B serves as the reproducible, open-weight LLM judge for matching predictions to reference discrepancies.The setup was retained because Qwen3 32B aligned less strongly with human evaluation.
- Evaluation: Recall is the reported performance measure because evaluation checks whether annotated discrepancies are detected.Predictions may also contain valid but unannotated discrepancies.
- Models: 22 model families include commercial and open-weight reasoning, instruction-tuned, and code-specific variants.Evaluated families include GPT-5, Gemini, GPT-OSS, Qwen3, DeepSeek R1, Nemotron, and Mistral.
5 Results
SCICOQA remains challenging for state-of-the-art models: recall is limited overall, with performance declining for paper omissions, longer inputs, recent papers, and some open-weight models. Additional analyses show high precision for some models but confirm recall as the main bottleneck.
- 46.7% recall is achieved by Gemini 3.1 Pro and GPT-5 Mini, while GPT-5 detects 70.0% of synthetic discrepancies.Recall on real and synthetic data is strongly correlated (r = 0.94), supporting synthetic data as a proxy for ranking models.
- Origin and Type: Code Omissions are easiest to detect, followed by Differences, while Paper Omissions are most challenging.GitHub discrepancies are dominated by Differences (71.4%) and Code Omissions (19.0%), whereas reproducibility-paper discrepancies often contain Paper Omissions (50%).
- Publication Year: Most models perform worst on 2025 data, suggesting that pre-training coverage of specific papers and codebases benefits discrepancy detection.The synthetic pipeline supplies continuously updateable data outside future models’ pre-training sets.
- Open-Weight Models: Nemotron 49B and Qwen3 30B Coder reach only 23.9% and 23.5% average recall on combined data, respectively.These are the best-performing models among the discussed open-weight systems.
- Validation of Unlabeled Discrepancies: Gemini 2.5 Pro achieves 94.1% precision, GPT-5 85.7%, and GPT-OSS 20B 67.0% in the precision analysis.On the pooled ground truth, GPT-5 achieves the best precision-recall tradeoff with an F1 score of 64.7%, while recall remains the primary bottleneck.
6 Conclusion
SCICOQA introduces a 635-discrepancy benchmark for evaluating paper-code alignment and shows that current LLMs remain insufficient for autonomous verification.
- 635 discrepancies form SCICOQA, a dataset for evaluating alignment between scientific papers and their code.
- 46.7% of real-world discrepancies are detected by the best-performing models, GPT-5 and Gemini variants.The benchmark identifies low recall as consequential because missed discrepancies can provide false assurance.
- Models struggle with Paper Omissions and recent uncontaminated publications.
- Current LLMs can assist quality assurance but cannot yet serve as autonomous verifiers of paper-code faithfulness.
- SCICOQA supports long-context reasoning, cross-modal claim verification, and scientific code-comprehension evaluation.The authors release the dataset, evaluation pipeline, and synthetic generation code for these applications.
7 Limitations
The dataset’s coverage and evaluation require caution because non-CS domains rely heavily on synthetic data, the discrepancy scope is intentionally narrow, and synthetic performance may advantage GPT-5.
- Domains: Real-world data is predominantly from Computer Science and Artificial Intelligence, while non-CS performance relies on synthetic approximations.Error distributions outside CS/AI may differ from those approximations.
- Discrepancy Definition: SCICOQA excludes simple bugs, hyperparameter mismatches, and documentation nits by focusing on meaningful reproducibility mismatches.This emphasizes scientific validity but does not cover the full spectrum of software engineering defects.
- Dataset Size: 635 discrepancies represent a deliberate quality-over-scale trade-off because real-world discrepancies are sparse and entries were manually verified.The verification process aimed to ensure each entry was meaningful rather than trivial or noisy.
- Synthetic GPT-5 Performance: GPT-5 achieves disproportionately higher recall on synthetic data because it also generated those discrepancies.Absolute performance should therefore be assessed on the real subset, while synthetic data supports relative model comparison.
8 Ethical Considerations
The authors describe data-release considerations and caution that automated discrepancy detection should support, rather than replace, human review.
- Synthetic data uses repositories with permissive licenses, while real-world discrepancies are contextualized from publicly available GitHub issues and reproducibility reports.
- Automated tools should not be the sole arbiter of paper validity because the best models detected only 46.7% of real discrepancies.The authors emphasize human verification of model outputs and reproducibility efforts.
- Model detections require expert verification before conclusions about research quality are drawn.The authors note that false positives were observed in their experiments.
A Implementation Details
The implementation constructs versioned paper-code inputs from GitHub repositories, processes papers and repositories to fit model contexts, and analyzes synthetic and real discrepancy descriptions.
- Repositories published since 2020 are found through GitHub API searches for paper-identifying URLs in descriptions and homepage fields.
- Versioned paper and code links use the discrepancy-reporting date and corresponding repository commit history.This preserves the versions in which the reported discrepancy existed.
- Paper text is converted from PDF to markdown with figures excluded and captions retained, while references are removed.
- The processed prompt includes relevant code files and a repository file list, truncating whole files when paper and code exceed the context window.
- Models are run with FP16 or MXFP4 configurations, using Ollama or vLLM for open-weight models and APIs for commercial models.Reasoning models generally receive high reasoning effort or token budgets.
B.2 Synthetic Code Analysis
The synthetic discrepancies are designed as targeted, minimal code changes, while the dataset spans several programming languages but remains dominated by Python.
- Code Changes: 1.08 ± 0.29 files are affected per generated discrepancy on average, with a maximum of three files.The mean number of distinct generated diffs is 1.24 ± 0.61 snippets per discrepancy, supporting targeted logic deviations rather than broad refactors.
- Code Similarity: 0.74 ± 0.20 is the mean Jaccard similarity for entire generated code blocks, indicating that surrounding context remains largely identical.Similarity over only modified lines is lower, at 0.50 ± 0.29.
- Code Similarity: 0.50 ± 0.29 is the mean Jaccard similarity over modified lines, suggesting changes usually modify existing statements rather than replace them completely.The synthetic discrepancies therefore adhere to small and conceptually meaningful changes.
- Programming Languages: Python is the dominant programming language in SCICOQA, while C/C++, MATLAB, CUDA, Java, Scala, Julia, and R appear in smaller portions.Real data is concentrated in CS/AI and synthetic data broadens the language coverage beyond Python.
- Programming Languages: 46 non-Python synthetic instances are grouped together because the dataset is skewed toward Python and low-resource language samples are small.Most models differ by no more than ±5 percentage points between Python and non-Python recall, with several notable exceptions.
C.3 Truncation Analysis
Recall declines when relevant code appears later in the prompt, indicating a long-context reasoning problem rather than simple truncation.
- Prompt Position: 89.4% of samples place the last relevant code file within the analyzed prompt-position distribution.The analysis relates this position to recall degradation across the dataset.
- Recall Degradation: GPT-5 recall drops from 77.5% to 50.7% as the relevant code file appears later in the prompt.Gemini 3.1 Pro similarly falls from 64.4% to 37.3%.
- Recall Degradation: Gemini 3.1 Pro recall drops from 64.4% to 37.3% when the relevant code file moves later in the prompt.This decline occurs even for models with virtually zero truncation.
- Interpretation: The observed performance pattern identifies long-context reasoning as the dominant factor, rather than truncation.The conclusion is consistent with the reported code-file position distribution and the paper’s other long-context analyses.
C.4 Real vs. Synthetic Recall
Model recall on synthetic discrepancies strongly predicts recall on real discrepancies, supporting synthetic data for relative model comparison but not as a replacement for real-world evaluation.
- Correlation: r = 0.94 is the Pearson correlation between model recall on synthetic and real discrepancies across the evaluated models.The relationship is visualized with synthetic recall on the x-axis and real recall on the y-axis.
- Correlation: r = 0.98 is the Pearson correlation after excluding the GPT-5 family.The GPT-5 family sits slightly off the shared trend line followed by the other models.
- Model Ranking: Relative model rankings remain consistent across synthetic and real discrepancies, apart from the GPT-5-family caveat.Models strong on synthetic injections are reliably better at identifying real SCICOQA discrepancies.
- Evaluation Use: Absolute performance should be assessed on the real subset, while the synthetic subset serves as a proxy for relative model comparison.This division supports scaling the benchmark to scientific domains where real-world examples are scarce.
- Ground-Truth Robustness: Across 22 models, the overall mean difference between GPT- and Gemini-generated ground-truth evaluations is +0.2 percentage points.The analysis reports no consistent family-specific preference overall, despite a slight Gemini self-generation effect.
C.6 Pooled Ground Truth Results
Evaluation against a pooled ground truth shows that GPT-5 leads in F1, but recall remains the central bottleneck even under favorable evaluation conditions.
- Pooled Ground Truth: 129 verified paper-code discrepancies form the pooled ground truth for 20 NLP and CV papers.The pool combines manually verified predictions with existing real-world SCICOQA discrepancies.
- Results: 72.4% F1 is GPT-5’s highest pooled-ground-truth score, balancing 84.1% precision and 63.6% recall.GPT-5 Mini and GPT-OSS 20B follow with F1 scores of 66.5% and 66.1%.
- Results: 85.7% precision is Gemini 2.5 Pro’s highest score, accompanied by 48.8% recall.The results show a precision–recall tradeoff across models.
- Interpretation: 63.6% recall is the maximum detection rate for verified discrepancies, even under favorable pooled-ground-truth conditions.Systems contributing predictions to the pool have a structural evaluation advantage, so their results are upper bounds.
- Open-Weight Models: GPT-OSS 20B and 120B are the strongest open-weight models, while the next-best remaining open-weight model reaches 39.3% F1.The next-best model cited is Nemotron 49B.