AI Coding Agents Can Reproduce Social Science Findings

1 Introduction

• Computational reproducibility is a minimal but necessary benchmark for AI participation in scientific knowledge production, yet workflows can fail despite available code and data.Potential sources include undocumented dependencies, version mismatches, operating-system differences, and stochastic pipeline elements.Source: §1 · pp. 1–2
• Existing evaluations are limited in social-science coverage, and PDF access may encourage confirmatory specification search rather than independent reproduction.CORE-Bench has 28 social-science tasks from CodeOcean, while Repro-Bench draws on nine economics and three political-science journals and omits sociology, psychology, and communication.Source: §1 · p. 2
• 221 tasks across four disciplines and 13 domains form SocSci-Repro-Bench, built from systematically selected papers and materials spanning five repositories and three programming languages.Tasks were included only when results matched across three manual code executions; a small set also tests whether agents identify restrictions caused by missing data.Source: §1 · p. 2
• The study evaluates Claude Code and Codex on reproducing results, inferring research questions from replication materials, and responding to original-paper PDFs.It also tests whether subtle prompt framing encourages agents to prioritize matching reported results over faithful execution of supplied code.Source: §1 · p. 2

2.1 Reproducibility Results

• 93.4% vs. 62.1% task accuracy and 78.0% vs. 35.8% paper accuracy favored Claude Code over Codex; both achieved 100% accuracy on non-reproducible tasks.A paper counted as reproduced only when all its tasks were correct; Claude Code had no failures, while Codex had 17.8% task-level and 27.0% paper-level failure rates.Source: Figure 1 · p. 3, §2.1 · p. 3
• Claude Code’s task accuracy was 100% for Python, 94.4% for Stata, and 91.9% for R, while Codex ranged from 40.0% for Python to 69.1% for R.Language comparisons are descriptive because task counts differed across languages; neither agent showed a meaningful performance difference between pre- and post-cutoff papers.Source: §2.1 · p. 4
• With original PDFs, overall accuracy rose modestly, but non-reproducible-task accuracy fell for both agents.Claude Code’s task accuracy rose from 93.4% to 94.5%, while its accuracy on non-reproducible tasks fell from 100.0% to 63.3%; Codex’s corresponding figures were 62.1% to 65.4% and 100% to 90.0%.Source: §2.1 · p. 4

2.2 Benchmark Performance Is Unlikely to Be Driven by Direct Paper Recall

• Metadata recovery was limited, and performance did not meaningfully differ for papers published before versus after either agent’s training-data cutoff.Claude Code recovered all four metadata fields correctly for 11.1% of papers; Codex returned author, journal, and year as unknown for 92.6% of papers, while achieving 62.1% task accuracy.Source: §2.1 · p. 4, §2.2 · pp. 6–7

2.3 Evidence of Abstract Reasoning

• Claude Code achieved a 73.5% research-question semantic match rate versus 70.0% for Codex, with a wider advantage on paper-level full matches.All Gold-standard questions were matched for 33.3% of papers by Claude Code and 24.1% by Codex; at least 60% were matched for 87.0% and 77.8%, respectively.Source: §2.3 · p. 7

2.4 Evidence of Sycophancy under Confirmatory Prompt Nudging

• Confirmatory prompting improved overall accuracy but reduced accuracy on non-reproducible tasks, where both agents sometimes supplied outputs instead of flagging impossibility.Task-level accuracy on non-reproducible tasks fell from 100.0% to 70.0% for Claude Code and from 90.0% to 60.0% for Codex.Source: §2.4 · p. 9, Figure 5 · p. 9

• With visible metadata, both agents achieved 100.0% task-level and paper-level accuracy across 54 tasks and 28 papers, with 0.0% failure rates.The caption identifies this as the non-anonymized condition.Source: §3 · pp. 9–10, Figure 6 · p. 10
• With anonymized metadata, Claude Code achieved 88.9% task-level and 81.0% paper-level accuracy, versus Codex’s 76.5% and 69.0%.Failure rates rose to 1.2% (task) and 2.4% (paper) for Claude Code, and 16.7% and 20.2% for Codex.Source: §3 · pp. 10–11
• Anonymization reduced accuracy from ceiling performance, showing that non-anonymized evaluation conflates information retrieval with computational reproducibility.CORE-Bench uses containerized CodeOcean capsules, a more standardized and portable setting than general-purpose repositories.Source: §3 · pp. 9–10
• The anonymized results reproduce SocSci-Repro-Bench’s agent ranking, while both agents performed somewhat better on CORE-Bench’s standardized environments.The authors interpret cross-benchmark consistency as convergent validity and suggest the agent gap reflects analytical and debugging capability rather than benchmark idiosyncrasies.Source: §3 · pp. 10–11

• SocSci-Repro-Bench covers four disciplines and 13 domains, with anonymized packages and tasks verified to reproduce consistently under manual execution.Its accuracies measure agent performance conditional on executable materials, not reproducibility across the broader social-science literature.Source: §4 · p. 11
• Claude Code substantially outperformed Codex, with 93.4% task-level and 78.0% paper-level accuracy versus 62.1% and 35.8%.The authors attribute much of the gap to execution robustness, including handling dependencies, hard-coded paths, and incomplete environment specifications.Source: §4 · p. 11
• Providing paper PDFs modestly improved accuracy and reduced failures, particularly for Codex, but introduced bias when missing data made reproduction impossible.Models sometimes extracted expected results from the paper rather than diagnosing missing data, creating a trade-off with independent reproduction checks.Source: §4 · p. 11
• The benchmark covers only a subset of social-science methods, and its structured tasks cannot represent the full diversity of empirical workflows.Because it focuses on results reproducible with available materials, it may overestimate performance in settings with incomplete or poorly documented packages.Source: §4 · p. 11
• The authors describe frontier coding agents as beginning to function as reliable executors of established computational workflows in social science.They call for careful benchmarking and methodological transparency as these systems become more integrated into scientific practice.Source: §4 · p. 12

5.1 Benchmark Construction

• The benchmark samples four disciplines and 13 shared substantive domains, prioritizing leading general-science journals and supplementing with disciplinary journals when needed.The search used Semantic Scholar’s relevance rankings, retaining the top 25 results per domain query before screening for data and code availability.Source: §5.1 · p. 12
• The final sample contains 54 papers whose available code generated at least two main-text figures or tables, using R, Python, or Stata.Papers without substantial data access, analysis code, or qualifying reproducible outputs were excluded.Source: §5.1 · p. 12
• Tasks were limited to findings that either reproduced with available materials or were clearly non-reproducible because of documented data restrictions; retained outputs matched published findings across three manual runs.Researchers formulated two to seven tasks per study, yielding 221 tasks, and annotators identified or inferred each paper’s research questions through discussion and consensus.Source: §5.1 · p. 12, Table 1 · p. 13
• Research assistants anonymized 54 replication folders by removing identifying information, result files, and preregistration documents while preserving eligible survey materials.A further scan identified residual names and identifiers in paths, links, and filenames, which were manually removed and script references updated.Source: §5.1 · p. 13

5.2 Evaluation Metrics

• Task accuracy is the proportion of tasks for which all associated questions are answered correctly.Source: §5.2 · p. 13

5.3 Experimental Setup

• Claude Code and Codex ran in sandboxed working directories with replication materials and task prompts; agents could install dependencies but could not access the web or other system directories.Codex’s initial average task-level accuracy was 47.2%, so its prompt was augmented with guidance for resolving dependency conflicts and path inconsistencies; Claude Code handled compatibility issues autonomously.Source: §5.3 · p. 13

6.1 Reproducibility Crisis

• Computational results can fail to reproduce even when original data and code are available, with failure rates exceeding 50% in some fields.The passage describes this phenomenon as a reproducibility crisis.Source: §6.1 · p. 14

6.2 Reproducibility and Replication Benchmarks

• CORE-Bench offers end-to-end tasks and scalable VM evaluation but draws on a standardized CodeOcean selection, while Repro-Bench evaluates major social-science findings across papers and materials.ReplicatorBench evaluates stages from information extraction through interpretation but has only 19 studies; HAL supplies evaluation infrastructure rather than a benchmark.Source: §6.2 · p. 14

6.3 LLM and Agent Performance on Reproducibility Tasks

• Prior evidence finds that general-purpose LLM agents struggle with computational reproducibility, and none of the reviewed studies systematically evaluates specialized CLI coding agents such as Claude Code and Codex.Reported difficulties include environment setup, debugging, complex workflows, and reasoning about evidence and methods.Source: §6.3 · p. 14

A.1 Examples of Tasks Excluded Due to Non-Deterministic Outputs

• The vaccine-misinformation task returned 2.32, 2.34, and 2.38 across runs and was excluded for inconsistency.It asked for the Crowdsourced Aggregate Score with the lowest observed treatment effect on vaccination intentions.Source: §A.1 · p. 18
• The fake-news correction task returned 3.73, 3.74, and 3.73 across runs and was excluded for inconsistency.It asked for the pairwise F-statistic comparing After with During on discernment.Source: §A.1 · p. 18
• The survey-cheating task returned 0.43, 0.40, and 0.41 across runs and was excluded for inconsistency.It asked for the posterior median log-odds coefficient for rescaled Age in a Bayesian logistic mixed-effects model.Source: §A.1 · p. 18

A.2 Examples of Tasks With Partial Data Access

• The survey-cheating study’s German sample supported an answer of 0.236, while the unavailable U.S. sample yielded “No Data.”The repository contained the German data but not the U.S. sample.Source: §A.2 · p. 18
• The sexism-in-teams materials supported a Joy cross-correlation coefficient of 0.133, but lacked data to compute the mean difference in emotional synchrony between conditions.The repository included cleaned time-series datasets and R scripts for cross-correlation analysis.Source: §A.2 · p. 18

B.1 Reproducibility Prompt

• The Codex protocol instructs agents to inspect materials, adapt and execute replication code, answer from outputs, log fixes, and infer research questions and metadata.Its instructions include offline use of local files and preserving original files while saving outputs separately.Source: §B.1 · pp. 19, 21, 22

B.2 Confirmatory Prompt Nudging

• The uncertainty prompt asks agents to explore analytically defensible alternatives and select results closest to the original paper’s analyses.Source: §B.2 · p. 23

C.1 Claude Code and C.2 Codex

• Claude Code’s project configuration allows common development operations while blocking web access; Codex’s profile confines execution to the workspace and disables web search.The Codex configuration permits network access for package installation.Source: §C.1 · p. 24, §C.2 · pp. 24–25

D.1 Execution Failures in Codex Reproduction Runs

• Table S1 consolidates failure categories in replication materials.Source: Table S1 · p. 26

E List of the 54 Papers

• Other listed studies address polarization and social ties, including “Shared partisanship dramatically increases social tie formation in a Twitter field experiment.”Source: E List of the 54 Papers · pp. 28, 30
• The roster includes studies on misinformation and accuracy prompts, including “Shifting attention to accuracy can reduce misinformation online.”Source: E List of the 54 Papers · p. 28
• The roster includes “Sexism in Teams: Exposure to Sexist Comments Increases Emotional Synchrony but Eliminates Its Benefits” and “Who’s cheating on your survey? A detection approach with digital trace data.”Source: E List of the 54 Papers · pp. 28–29

F Extended Results

• Overall task accuracy rose from 93.4% to 94.5% for Claude Code and from 62.1% to 65.4% for Codex with PDF access.Means across three independent runs.Source: Figure S1 · p. 31
• On non-reproducible tasks, accuracy fell from 100.0% to 63.3% for Claude Code and from 100% to 90.0% for Codex with PDF access.These tasks had gold-standard answers indicating missing or inaccessible data.Source: Figure S1 · p. 31
• Codex’s task failure rate fell from 17.8% to 12.2% with PDF access, while Claude Code maintained zero failures in both conditions.At the paper level, Codex’s failure rate fell from 27.0% to 5.6%.Source: Figure S1 · p. 31