PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies

Shaolong Chen, Yanlin Fei, Nazhou Liu, Xinmiao Yu, Lei Li, Rahul Thapa, Madalina Ciobanu, Qingqing Mao, Ritankar Das

arXiv:2608.16645v1cs.AIcs.CLcs.MA

TL;DR

Can language models recover a published paper’s research idea from only its pre-publication bibliography? Reconstruction benchmarks this under strict anti-leakage conditions and finds that reference-only multi-agent review and Swiss selection raises Match rates to approximately 23–42% across six domains, versus approximately 3–15% for single models.

  • Problem

    It remains unclear whether language models can recover a published paper’s actual research idea from references available before publication.

  • Method

    Reconstruction uses blind, time-cut bibliographies with anonymous references and evaluates hypotheses against held-out seed ideas using an independent LLM judge.

  • Results

    Multi-agent review plus Swiss selection reaches Match rates of approximately 23–42% across six domains, versus approximately 3–15% for single models.

  • Takeaways & Limitations

    Reconstruction provides a controlled stress test of literature understanding and multi-agent coordination under a strict information cutoff.

  • Takeaways & Limitations

    The observed approximately 2.4× gain may partly reflect selecting five hypotheses from 20 candidates rather than collaboration mechanics alone.

Abstract

from arXiv · show

Can a language model recover the true research idea of a published paper when given only that paper's pre-publication bibliography? We introduce Reconstruction, a blind idea-recovery benchmark that withholds the seed paper and all contemporaneous or future literature, and asks models to propose hypotheses that an independent large language model judge matches against the held-out ground-truth idea. A strict anti-leakage protocol-temporal citation cutoff, anonymous reference IDs, and frozen per-paper bibliographies, which prevents prompt-time leakage of the seed idea. Across six scientific domains and 643 evaluated papers, seven frontier models achieve only modest Match rates (approx. 3-15%). We then evaluate a reference-only multi-agent (top 4) pipeline that combines cross-model review with a Swiss tournament over aligned hypothesis slots, without external web search. Cross-model review plus tournament selection raises Match rates to approx. 23-42% across all six domains, which is an observed approx. 2.4x lift over the best single-model baseline. This draft reports the protocol, anti-leakage design, and current results as an arXiv timestamp.

1 Introduction

Reconstruction tests whether models can recover a published paper’s research idea from a blind bibliography containing only pre-publication literature. The benchmark reports modest single-model performance and substantially higher Match rates from a reference-only multi-agent selection pipeline.

  • Protocol: Models receive only literature published strictly before the seed paper, without its title or abstract, and an LLM judge compares hypotheses with the held-out idea.Match rate is the fraction of hypotheses judged to match the ground truth.
  • Benchmark: Reconstruction measures idea recovery from pre-publication bibliographies across ML and five Nature-family domains.The benchmark uses a time-cut, anti-leakage protocol.
  • Single-model baselines: 13.3% ± 2.3% is the best average Match rate among seven frontier models, achieved by Claude-Opus-4.8 across 643 papers.Domain scores typically fall in the ∼3–15% band.
  • Multi-agent pipeline: ∼2.4× is the observed lift over the best single-model baseline from the full selection pipeline.The paper credits the full selection pipeline rather than collaboration alone.

2 Related Work

Prior work studies LLM-based scientific ideation, novelty evaluation, and temporal forecasting, whereas Reconstruction recovers a held-out seed idea from its pre-publication bibliography. Its reference-only multi-agent method adapts cross-model review and Swiss-system selection, with separate independent-judge scoring.

  • Automated scientific discovery and ideation: LLM scientific-discovery systems generate research ideas from literature, while IdeaBench evaluates idea quality and RINoBench targets automated novelty judgment.Human studies compare LLM idea novelty with expert proposals and examine the gap between ideation and execution outcomes.
  • Temporal evaluation: Reconstruction recovers a seed paper’s idea from its pre-publication bibliography, unlike HindSight, which scores pre-cutoff ideas against future publications.Match rate uses multi-model LLM judges with leave-one-out or origin recusal to reduce self-evaluation bias.
  • Multi-agent debate and selection: The multi-agent pipeline adapts cross-model review and Swiss-system selection, using only the same frozen blind bibliography without runtime web search.An independent LLM judge separately compares each hypothesis with the held-out seed title and abstract for final Match scoring.

3 The Reconstruction Benchmark

The Reconstruction benchmark tests idea recovery from a seed paper’s pre-publication bibliography under strict information isolation, anonymous references, and frozen reading lists. It scores five hypotheses per case with independent judge matching and defines a recused, reference-only multi-agent comparison.

  • Anti-leakage protocol: The blind corpus exposes only pre-publication bibliography title/abstract text through anonymous reference IDs, withholding the seed and same-day or future literature.The temporal cutoff excludes undated references, and proposers never see the seed paper or contemporaneous literature.
  • Case construction: Every scored case produces exactly five distinct hypotheses, each with anonymous supporting-reference IDs, while cases failing this requirement are excluded.Evidence binding requires each hypothesis to cite anonymous supporting references.
  • Scoring: A binary judge match label determines the paper-level Match rate, with the judge seeing the seed title/abstract and hypothesis title/summary.Default cases use a judge different from the proposer; multi-agent judges recuse on hypotheses they originated.
  • Multi-agent pipeline: The multi-agent pipeline generates five hypotheses per model, aligns candidates by slot, performs reference-only cross-model review and Swiss selection, then independently scores five champions.There is no web search, and origin-matched judgments are treated as missing rather than No-match.
  • Multi-agent pipeline: The reported multi-agent evaluation uses a post-selected fixed roster of Claude-Opus-4.8, GPT-5.6-Sol-Pro, Kimi-K3, and GLM-5.2, not a held-out selection rule.The four models were selected by six-domain average Match rate on the same reported seed-paper set used for scoring.

4 Experiments

Experiments evaluate reconstruction on six scientific domains using eligibility-filtered seed papers and pre-publication bibliographies. Single models achieve modest Match rates, while reference-only cross-model review plus Swiss selection substantially improves performance across domains.

  • Seed collection and eligibility: 879 titles were collected across six sources, and seeds required a usable publication date plus at least three resolved references published strictly before T0.The collection included 168 ICML 2026 Oral titles and 91–221 titles from five Nature-family journals.
  • Evaluated set: 643 aligned papers support the main multi-agent comparison, after incomplete Default runs and multi-agent exclusions were removed.Multi-agent runs froze each paper’s pre-T0 bibliography; Medicine was capped at 80 papers for API-cost control.
  • Single-model results: 3.4%–15.0% observed cell means span single-model Match rates, with Claude-Opus-4.8 achieving the best six-domain average of 13.3% ± 2.3%.Using other-top-3 judges for Default top-4 proposers changes point estimates only slightly and does not reach the multi-agent band.
  • Uncertainty analysis: 2.4× is the overall paper-level bootstrap ratio, with a 95% CI of [2.3, 2.6], while success@5 rises from 55% to 57%.The bootstrap resamples papers while retaining each paper’s full set of judged hypotheses.
  • Candidate-count controls: C < MA < D: multi-agent exceeds the Default 4×5 slot oracle C in every domain but remains below the unconstrained Default top-5 oracle D, 35.6% versus 44.5% Overall.This comparison indicates gains beyond slot-wise Default selection while preserving headroom relative to infeasible cherry-picking.

5 Limitations •

The benchmark has unresolved validity and generalization limitations, including possible parametric contamination, judge dependence, non-like-for-like multi-agent comparisons, and restricted domain coverage. The reported multi-agent advantage may also reflect candidate count, compute, hypothesis length, or post-selection rather than collaboration alone.

  • Validity: Seed papers may have appeared in model pretraining, so recovered ideas could partly reflect memorization despite the reference-only protocol and citation cutoff.The protocol reduces prompt-time leakage but cannot rule out parametric contamination.
  • Validity: Match rates depend on LLM judges, and human agreement with the Match rubric has not yet been reported.Leave-one-out or origin recusal mitigates self-evaluation bias, while judge variance is reported.
  • Comparison design: The ∼2.4× Default-versus-multi-agent association is not isolated: multi-agent selects 5 of 20 hypotheses, whereas Default keeps 5 of 5.The observed lift may therefore reflect inference-time scaling rather than cross-model review or tournament mechanics alone.
  • Comparison design: The Default and multi-agent settings use different judge panels, so their comparisons remain not strictly like-for-like.Table 2’s dagger rows partially align the Default top 4 panel, but do not fully resolve the mismatch.
  • Comparison design: The multi-agent roster is post-selected on same-paper Default averages, uses more total compute, and has longer hypotheses than Default ones.Overall mean lengths are 114, 56, and 191 words for multi-agent, Default, and seed title+abstract, respectively; equal-budget and length-matched controls are not reported.
  • Scope and measurement: The benchmark covers only six domains, while match remains a coarse binary outcome that could be replaced or supplemented by graded scoring.Broader scientific coverage and finer-grained evaluation remain open.

6 Future Work

Reconstruction is presented as a first step toward generating novel, feasible research ideas rather than merely recovering known ones. Its harness is intended to transfer from reconstruction to forward-looking Generation mode while preserving strict information-cutoff principles.

  • Reconstruction as a first step: ∼23–42% Match rates result from reference-only cross-model review plus Swiss selection, compared with ∼3–15% under single-model Default.The full pipeline shows an observed mean 2.4× ratio versus the best dagger single-model baseline.
  • Generation mode: The intended end goal is Generation mode, in which models propose novel, feasible, forward-looking research directions.Reconstruction functions as a controlled stress test of literature understanding and multi-agent coordination under a strict information cutoff.

7 Conclusion

Reconstruction is a blind bibliography-to-idea recovery benchmark with strict anti-leakage controls. Reference-only cross-model review plus Swiss selection reaches ∼23–42% Match rates across six scientific domains, versus ∼3–15% for the single-model Default regime.

  • Conclusion: Reconstruction is a blind bibliography-to-idea recovery benchmark with a strict anti-leakage protocol.The draft is released to timestamp the task and finding.
  • Conclusion: ∼23–42% Match rates are achieved across six scientific domains by reference-only cross-model review plus Swiss selection over the top 4 models.The selection pipeline combines cross-model review with Swiss selection over the top 4 models.
  • Conclusion: ∼3–15% Match rates are achieved by the single-model Default regime.Further analyses and Generation-mode evaluation will follow.

A Final Match judge prompt and rubric

The final Match judge independently compares each hypothesis with the held-out seed title and abstract under leave-one-out origin recusal. A binary match requires the same core research question and central claim, not merely shared topic or vocabulary.

  • Evaluation procedure: Each hypothesis is judged independently against the held-out seed title and abstract, with the proposer’s origin judge recused.For a champion, the origin label is excluded and the score averages the three eligible non-origin labels.
  • Evaluation procedure: The judge uses only the supplied seed title and abstract and the hypothesis title and summary, without external seed-paper knowledge.The output is a binary JSON decision with a matched label and rationale.
  • Binary Match rubric: Match requires the same research question and essentially the same proposed mechanism or central claim, even when wording differs.The rubric accepts hypotheses describing the same core idea rather than requiring textual identity.
  • Binary Match rubric: No match includes shared fields or keywords with a different question, generic bibliography restatements, and related approaches lacking the seed’s distinctive claim.These cases are rejected even when the hypothesis is broadly related to the seed paper.
  • Borderline examples: Table 6 illustrates the rubric with unanimous Match and No-match examples across different domains and top 4 proposers.Default and multi-agent hypotheses share each seed paper and origin proposer, and come from a high-agreement pool.

B Swiss tournament mechanics (reconstruction multi-agent)

The reconstruction multi-agent tournament pools four model proposals per hypothesis slot and selects a champion through Swiss rounds using reference-only ballots. Two non-proposing models judge each match in both presentation orders, with scores based on four votes and ties resolved by aggregate importance then session ID.

  • Candidate pool: Each hypothesis slot forms a candidate pool of n=4, with all candidates entering Swiss selection without a novelty gate or external web search.Ballots see only the anonymous reference corpus.
  • Rounds and pairing: The tournament uses max(1, n−1) rounds—three here—with initial shuffling, consecutive pairing, score-based later pairings, and byes for leftovers.Later rounds pair candidates with similar scores while avoiding rematches when possible.
  • Judging rules: For each match, the two candidate proposers recuse themselves, while the remaining two models judge both A/B and B/A presentation orders.Swapping only presentation order reduces position bias while preserving candidate identities.
  • Scoring: Each match yields four ballots, awarding match points equal to vote count/2; the highest cumulative Swiss score wins the slot.The two candidates’ points always sum to 2.0, and disagreements across presentation orders are logged for position-bias auditing.
  • Reference-only evaluation: Judges compare hypotheses only against the anonymous reference corpus, prioritizing grounding, problem-gap capture, and specific coherent claims over novelty.The final Match judge is applied only after five slot champions are assembled and remains independent of prior review and selection calls.

C Paper-level bootstrap confidence intervals

Table 3 quantifies paper-set uncertainty with paper-level nonparametric bootstrap percentile intervals, complementing Table 2’s judge-dispersion summaries. Resampling preserves within-paper dependence, uses 2,000 replications, and forms lift intervals directly from bootstrap ratios.

  • Motivation: Table 3 reports paper-level nonparametric bootstrap percentile intervals because Table 2’s ± values capture judge dispersion, not uncertainty in paper inclusion.The intervals address uncertainty from which papers enter the seed-paper set.
  • Estimands: Domain Match rates are means of paper-level scores, while domain lift is the ratio of corresponding domain means; the Overall row pools all 643 papers.The multi-agent and fixed best top 4 proposer scores are defined per aligned paper before aggregation.
  • Resampling: Bootstrap resampling draws papers with replacement within each domain and reuses each paper’s complete judged hypothesis set, preserving dependence among its 5 slots.The Overall row instead resamples all 643 papers with replacement and forms the ratio of pooled means.
  • Implementation: B=2000 independent bootstrap replications are used with a fixed random-number-generator seed for reproducibility.The choice follows guidance that larger replication counts reduce Monte Carlo error in percentile-interval endpoints.
  • Percentile 95% CI: Lift intervals use the empirical 2.5% and 97.5% quantiles of bootstrap ratios directly, rather than dividing separately computed Match-rate endpoints.Table 3 point estimates use the original aligned seed-paper set; Success@5 columns are point estimates only.

D Paper-wise paired comparison and sign test

The paper compares the multi-agent system with a stronger per-paper foil that selects the best of four single models independently for each paper. Across 643 papers, multi-agent wins significantly more often than it loses, including in every domain.

  • Per-paper foil: The per-paper single-model foil takes the maximum paper-level Match rate across four dagger models, allowing the baseline to switch models across papers.This is stronger than comparison with any one fixed model.
  • Paired outcomes: 343 wins, 160 losses, and 140 ties occur across n=643 aligned papers.The counts sum to 643; ties are not directional evidence.
  • Sign test: p ≈2.3 × 10−16 under the exact two-sided sign test, using W = 343 wins among neff = 503 decisive papers.Ties are discarded, and the test evaluates directional consistency rather than the size of the performance gap.
  • Per-domain breakdown: All six domains show a significant directional advantage at the 0.05 level: ML p ≈0.045, Astronomy p ≈0.0026, Chemistry p ≈0.0018, Materials p ≈2.1×10−5, Medicine p ≈1.7×10−4, and Physics p ≈2.0 × 10−5.ML is borderline despite meeting the 0.05 threshold.

E Candidate-count bounds (A/B/C/D) and E[A] = B

The section defines candidate-count baselines and oracles over four top-4 proposers with five aligned hypothesis slots, showing that the expected random selector A equals pool mean B. Slot-wise oracle C and unconstrained oracle D are non-deployable upper bounds, with D ≥ C per paper.

  • Setup: Each aligned paper contributes 20 hypotheses from four top-4 proposers and five slot-aligned hypotheses, scored by the fraction of the other three models judging each as matched.Scores r_k,g lie in [0, 1].
  • Expectation identity: E[A] equals B, the deterministic mean over all 20 hypotheses, after conditioning on each paper and also after averaging across papers.Realized domain and Overall means for A may differ from B because A samples model choices.
  • Random selector A: A randomly selects one model independently in each slot, producing a paper-level Match rate from five retained hypotheses.The random choices are uniform over the four models.
  • Oracle bounds: C takes the maximum score within each slot and averages across slots, making it an infeasible upper bound under slot alignment and peer judges.C uses match-label peeking and is not deployable.
  • Oracle bounds: D averages the five largest scores across all 20 hypotheses while ignoring slots, and necessarily satisfies D ≥ C for each paper.D also uses match-label peeking and is not deployable.
Loading 2608.16645v1…