PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenBioRQ: Unsolved Biomedical Research Questions for Agents

Minbyul Jeong

arXiv:2606.21959v1cs.CL

TL;DR

Existing biomedical benchmarks assume known answers, so they do not test whether agents verify citation support or appropriately handle unsolved questions. OpenBioRQ addresses this with 12,553 retrieval-grounded unsolved questions and shows a hard, non-saturating benchmark that also exposes agentic collapse.

  • Problem

    Existing biomedical benchmarks presuppose known answers, leaving citation support and abstention on unsolved research questions underexplored.

  • Method

    OpenBioRQ constructs 12,553 unsolved questions across 12 domains with retrieval-grounded openness verification, empirical difficulty labels, and frozen per-question checklists.

  • Results

    The benchmark is hard, non-saturating, and discriminating across capability tiers, with held-out same-lineage models solving only ~17% of core questions.

  • Takeaways & Limitations

    OpenBioRQ evaluates literature synthesis and grounding while showing that agents may abandon tools on the hardest questions, so it is not a clinical decision-support tool.

  • Takeaways & Limitations

    The openness audit cannot detect narrative or guideline resolution, and its recall for those forms of resolution is unquantified.

Abstract

from arXiv · show

A working citation looks like proof -- but the fact that a link resolves does not mean the cited paper supports the claim. I find that current agentic models rarely fabricate citations (over $99\%$ resolve), yet roughly $15.9\%$ link to the wrong paper. Existing benchmarks miss this failure mode: when a question has a fixed answer key, a model can reproduce the expected source from that key rather than independently verifying that the source supports the claim. I introduce \textbf{\openbiorq{}}, a retrieval-grounded agentic benchmark of $12{,}553$ unsolved biomedical research questions across $12$ domains that treats open questions as a faithfulness-and-abstention probe. To my knowledge, this is the first biomedical benchmark to combine an agentic setting -- where the model must issue multiple tool calls -- with unsolved questions that have no answer key. Openness is verified against real follow-up evidence rather than a model's parametric knowledge. Difficulty is empirical: I anchor it on questions that three open-weight reference models fail to answer, rather than on subjective hardness labels. On this hardest subset, held-out models from the same lineage as the difficulty anchors solve only ~17%, while three independent frontier agents (Gemini-3-Pro, Opus-4.7, GPT-5.5) span a wide 29-60% range. The benchmark is thus hard, non-saturating (the best agent still leaves ~33-40\% unsolved), and discriminating across capability tiers. Beyond difficulty, I observe agentic collapse on the hardest questions, where agents stop using their tools. For the most collapse-prone model, blocking tool access entirely barely changes its score -- so tools stop paying off exactly where they are needed most. A frozen per-question checklist raises inter-judge agreement from Spearman 0.35 to 0.82.

1 Introduction

OpenBioRQ targets a neglected biomedical evaluation setting: agentic models must investigate genuinely unsolved questions without answer keys, exposing wrong-paper citations, tool abandonment, and abstention-versus-confabulation failures. It also separates citation existence from whether the cited paper supports the claim, while verifying openness through retrieval-grounded re-audits.

  • Motivation: A resolving PMID can still point to the wrong paper, making a biomedical claim appear supported despite lacking content support.An audited vaccine-efficacy example cited a real ophthalmology article that never mentioned vaccines, even though the reported efficacy was roughly correct.
  • Benchmark gap: OpenBioRQ occupies the underexplored agentic-and-unsolved quadrant, where removing answer keys makes wrong-paper citation, agentic collapse, and abstention-versus-confabulation measurable.Existing closed-form, long-form, and agentic-with-answers benchmarks assume known answers and therefore cannot expose these failures structurally.
  • Citation faithfulness: Citation auditing distinguishes identifier existence from content support, because biomedical agents rarely fabricate identifiers even when the linked papers fail to support their claims.The audit evaluates both whether an identifier resolves to a real paper and whether that paper supports the stated claim.
  • Openness and difficulty: Open questions are verified against real follow-up evidence rather than framing alone, and a re-audit of the 657-question core found none resolved.This design addresses the risk that labeling questions open would otherwise introduce confirmation bias.
  • Agentic collapse: On the hardest questions, agents abandon their tools, and blocking tool access altogether barely changes performance for the most collapse-prone model.Tool access therefore stops paying off precisely where it should matter most; OpenBioRQ is intended for research-assistance evaluation, not patient care.

2 Related Work

Prior biomedical QA benchmarks are increasingly saturated, while OpenBioRQ combines rubric-based scoring with open-question evaluation. It also builds on research-level dataset construction, tool-using agent evaluation, and citation-faithfulness research.

  • Biomedical question answering benchmarks: Closed-form biomedical QA benchmarks increasingly cluster frontier systems tightly, limiting their value as discriminators.This includes MedQA and related medical QA benchmarks evaluated by frontier systems from the Med-PaLM line onward.
  • Biomedical question answering benchmarks: OpenBioRQ inherits rubric-based scoring from long-form medical QA benchmarks while removing their presupposition of a curated answer.MedLFQA/OLAPH and HealthBench grade responses against must-have, nice-to-have, or physician-written criteria, whereas OpenBioRQ evaluates open questions.
  • Research-level and open-problem datasets: OpenBioRQ extends ResearchMath-14k’s extract-then-refine pipeline, self-containment audit, near-duplicate removal, and behavioral–factuality separation.Its self-containment rate is 85.4% overall, 95% on the retrieval-verified track, and 75% on expert-consensus.
  • Tool-using and agentic LLM evaluation: OpenBioRQ adopts multi-round tool-use evaluation rooted in ReAct-style reasoning–action interleaving and learned tool invocation.Related agentic benchmarks score whether models select, parameterize, and sequence tools to complete tasks with verifiable targets.
  • Citation faithfulness, attribution, and grounding: Citation-faithfulness research studies fabricated references, source attribution, and inline grounding of generated claims.Prior work also audits whether generative search engines’ inline citations actually support their associated statements.

3 The OpenBioRQ Benchmark

OpenBioRQ is a corpus of unsolved biomedical questions built through a crawl-to-export pipeline, with independently auditable retrieval-grounded openness and empirical difficulty labels. Its questions are refined for standalone use and difficulty is defined by failures of three tool-using roster models against frozen checklists.

  • Corpus construction: OpenBioRQ turns source fragments into standalone questions and attaches retrieval-grounded openness, empirical difficulty, and gradability layers.The PSVD example illustrates how refinement converts an extracted fragment into a question that can be evaluated independently.
  • Corpus construction: 12,553 questions form the base corpus across retrieval-verified and expert-consensus tracks, while priority-setting and expand tracks are additional and deduplicated.Retrieval verification uses real follow-up evidence; expert consensus relies on authoritative bodies declaring questions open.
  • Audit: 85.4% of refined questions are self-contained, compared with 51.6% for an extractor-only baseline, a gain of +33.8 points.The retrieval-verified and expert-consensus rates are 95% and 75%, respectively.
  • Audit: 56.5% of a 200-question sample had their openness status changed by retrieval-grounded rejudgment, which returned unknown for 14%.On the hardest subset, re-audit found 0/657 core questions with detectable resolution.
  • Difficulty: A question is core when GLM-5.1, Qwen3.6, and DeepSeek-V4 all score below 0.5 with full tool access; priority-setting questions are 49% core, 45% discriminating, and 6% easy.Difficulty is assigned empirically using frozen checklists rather than by hand.

4 Evaluation Protocol

OpenBioRQ evaluates unsolved biomedical questions with frozen, auditable checklists and multi-round biomedical tool use rather than fixed answer matching. Its headline solve rate uses deterministic single-attempt scoring, while synthesized gold answers remain non-authoritative because their citations are frequently misattributed.

  • Checklist-based grading: Frozen checklists convert open-ended grading into concrete verdicts covering required facts, uncertainties, evidence grounding, and prohibited behaviors.Criteria include must mention, must acknowledge, must ground, and must avoid requirements.
  • Solve-rate definition: A model solves q iff scoreT_m(q) ≥0.5 under a single deterministic T=0 completion, with mean frontier-agent scores of 0.39/0.44/0.53 supporting cutoff robustness.The 0.5 cutoff is a reporting convention; the underlying capability ordering and cross-tier band drive the claims.
  • Checklist-based grading: Freezing criteria makes judgments reproducible and prevents collapsed or hedging answers from receiving credit merely for fluent prose.Answers must satisfy the same must mention and must ground criteria regardless of agentic collapse.
  • Agentic evaluation: Each question permits up to ten rounds of interleaved reasoning and tool calls across ten biomedical retrieval APIs before the model submits its answer.Traces record the final answer, ordered tool calls, token usage, and wallclock time.
  • Citation evaluation: ≈100% of synthesized gold-answer citations resolve, yet ≈74% are gold-misattributions whose cited papers do not support the claims.Gold answers are used only as rubric-drafting context, never as matching targets or ground-truth citations.

5 Experiments and Analysis

The experiments evaluate citation existence versus content support, agent behavior on open biomedical questions, and the empirical difficulty and robustness of a frozen core. They show that wrong-paper citations remain common despite rare identifier fabrication, while the hardest questions expose agentic collapse, decoding-sensitive difficulty, and discrimination across model capabilities.

  • Citation faithfulness: 15.9% of 4,649 real citations were wrong-paper citations, despite Qwen3.6 and DeepSeek-V4 achieving 99.6% and 99.8% existence, respectively.Wrong-paper rates were 20.2% for Qwen3.6 and 13.1% for DeepSeek-V4; GLM-5.1 was the existence outlier at 84.7%.
  • Citation faithfulness: Cohen’s κ = 0.755 between independent judges supports the robustness of the wrong-paper verdict, after correcting an extraction bug affecting ∼11% of snippets.Both judges ranked GLM-5.1 worst and DeepSeek-V4 best.
  • Agent behavior: On the harder priority track, GLM-5.1 and DeepSeek-V4 abandoned tools on 69% and 62% of questions, whereas Qwen3.6 remained engaged with a 22% zero-tool rate.The corresponding zero-tool rates were 65% for GLM-5.1 and 62% for DeepSeek-V4, while Qwen3.6’s rate was flat.
  • Core difficulty: 17% was the approximate solve rate for same-lineage held-outs, while Gemini-3-Pro, Opus-4.7, and GPT-5.5 solved 28.8%, 37.8%, and 59.6% of the frozen core.The 423-question frozen core was selected from 657 questions; its apparent ≈0% difficulty at T=0.3 changed to GLM-5.1’s 26.6% at T=0.
  • Core difficulty: 30.8% without tools versus 26.6% with tools was observed for GLM-5.1 on the full core, with overlapping Wilson confidence intervals.The data therefore do not support a measurable tool advantage, consistent with the model’s tendency toward agentic collapse.
  • Robustness and limitations: 46.5% of boundary items and 34.7% of near-threshold deep-failures changed membership under re-sampling, so the frozen core is a single-sample snapshot.The authors retract an earlier 85.8% retention estimate and report no single retention figure, while all models remain scored on the identical released 423-item set.

6 Discussion and Limitations

The benchmark’s openness, difficulty, and wrong-paper estimates are time-, roster-, and judge-dependent, while scoring new models requires live tools. The authors release records, fixed task IDs, and replay data to support independent reverification and evaluation.

  • Limitations: Openness is time-dependent: none of 657 re-audited core questions had resolved, but periodic re-auditing remains necessary.Per-item source records enable independent reverification because questions may resolve after their sources are written.
  • Limitations: The reported robustness checks were pre-specified probes, including a deep-failure test that overturned the earlier 85.8% retention figure.The checks included crossfamily judge swaps, per-domain breakdowns, re-decode stability, and roster-composition variation.
  • Limitations: The L2 wrong-paper rate is judge-relative, with only preliminary non-expert human spot-checking; independent judge and full-text re-judge results leave it intact.Domain-expert κ validation is deferred.
  • Limitations: Difficulty is a single T=0, model-relative snapshot: a second decode changes frozen-core membership, while the strongest frontier agent still solves a majority.The range remains essentially unchanged on items surviving both decodes, so no single retention figure is reported.
  • Limitations: Evaluating a new model still requires live tools, so the authors release a fixed 423-item task list and tool-response replay cache instead of a rerunnable procedure.Rerunning live selection produces a substantially different core.

7 Conclusion … A.1 Self-Containment Audit

OpenBioRQ is a retrieval-grounded benchmark of unsolved biomedical questions for tool-using agents, emphasizing citation correctness, openness verification, and abstention without answer keys. Its construction pipeline improves question self-containment, while audits quantify refinement gains and track differences.

  • 7 Conclusion: OpenBioRQ is an agentic biomedical benchmark of unsolved questions without answer keys, scoring grounding and abstention rather than settled answers.The benchmark verifies openness through retrieval-grounded status judgments rather than assuming it.
  • 7 Conclusion: 15.9% of real citations are wrong-paper despite only ≈0.7% identifier invention across 4,863 citations.Wrong-paper citations attach resolvable PMIDs to unsupported claims, a failure invisible to identifier-only checking.
  • A Construction, Openness, and Difficulty Details: The corpus construction follows five stages: crawl, extract, refine, deduplicate, and export.Authoritative open-problem documents are processed into questions enriched with taxonomy, tools, and difficulty information before export.
  • A Construction, Openness, and Difficulty Details: Each source can yield several distinct open questions, which are rewritten to stand alone and mapped to taxonomy labels, difficulty hints, and medical MCP tools.The initial tool-less open-status judgment is later replaced by retrieval-grounded judgment.
  • A Construction, Openness, and Difficulty Details: Near-duplicate questions with MiniLM-L6 cosine similarity ≥ 0.90 are collapsed, while distinct questions from the same source are retained.Deduplication operates on question text rather than source documents.
  • A Construction, Openness, and Difficulty Details: The exported corpus contains four tracks: retrieval verified, expert consensus, priority setting, and expand.Their sizes and intended uses are specified under corpus statistics.
  • A.1 Self-Containment Audit: 85.4% of refined questions are self-contained versus 51.6% for the extractor-only baseline, a +33.8-point lift from refinement.This audit uses an LLM-judged random sample of n = 500.
  • A.1 Self-Containment Audit: 95% of retrieval-verified questions are self-contained compared with 75% in the expert-consensus track.The self-containment gain is therefore uneven across tracks.

A.2 Retrieval-Grounded Openness · A.3 Empirical Difficulty

OpenBioRQ grounds openness in retrieved follow-up evidence and derives difficulty empirically from failures by three roster models rather than assigning it by hand. Its hardest questions resist detectable resolution, show limited surface memorization, and are evaluated at question-level granularity within a snapshot-defined corpus.

  • A.2 Retrieval-Grounded Openness: Openness is re-judged from retrieved follow-up evidence, not source framing or model memory.Stage 1 gathers evidence from citing papers, ClinicalTrials.gov records, or Semantic Scholar citations; Stage 2 requires evidence IDs and rejects hallucinated citations.
  • A.2 Retrieval-Grounded Openness: 56.5% of 200 retrieval-verified questions changed status, and 14% received the previously unreachable unknown label.Grounded re-judgment made answered and unknown labels attainable, confirming that openness was evidence-grounded rather than inherited from source framing.
  • A.2 Retrieval-Grounded Openness: 0/657 core questions were confirmed resolved in the still-open audit, although narrative or guideline resolution remains undetectable and recall is unquantified.The audit covered trial-completed and no-follow-up flagged questions and found no detectable resolution under grounded re-judgment.
  • A.2 Retrieval-Grounded Openness: The frozen core shows limited surface memorization: median source overlap is 3 words, token Jaccard is 0.23, and 86.5% share ≤5 consecutive words.Under GLM-5.1, the core also has 1.61 nats/token higher perplexity than MedQA-USMLE, though these checks cannot exclude paraphrased exposure.
  • A.3 Empirical Difficulty: Difficulty is empirical: a question is core exactly when GLM-5.1, Qwen3.6, and DeepSeek-V4 all fail its frozen checklist with score < 0.5.Every question is answered with full tool access by the three roster models before core labeling.
  • A.3 Empirical Difficulty: 49% of 525 questions are core, 45% are discriminating, and 6% are easy in the question-granular priority-setting track.These correspond to 256, 236, and 33 questions, respectively.
  • A.3 Empirical Difficulty: The 12,553-question base corpus combines 6,648 retrieval-verified and 5,905 expert-consensus questions, while priority-setting and expand add 525 and 483 deduplicated questions.Openness labels and difficulty scores are as-of properties reproducible against their snapshot date; unique source id#k task IDs preserve distinct co-extracted questions.

B Definitions, Notation, and Harness · C Extended Experiments and Robustness

The evaluation formalizes checklist-based solving, empirical difficulty, citation support, and openness within a multi-round agentic harness. Extended analyses examine citation auditing, agentic collapse, rubric robustness, and reproducibility through the frozen-core design.

  • B Definitions, Notation, and Harness: Checklist scores weight binary-checkable criteria by importance, combining must mention, acknowledge, ground, and avoid requirements.Each question has 5–8 criteria with weights 1, 2, or 3; must-avoid criteria invert verdict polarity.
  • B Definitions, Notation, and Harness: Solve rate is the fraction of fixed questions solved in one deterministic attempt, using the threshold τ=0.5.The headline metric fixes decoding temperature T=0 and intentionally avoids the label pass@k.
  • B Definitions, Notation, and Harness: Empirical difficulty counts failures among GLM-5.1, Qwen3.6, and DeepSeek-V4, defining all3-fail, all3-pass, and split buckets.The core retains audited all-roster-fail items at T=0.3, while the frozen core further requires deterministic all-roster failure and satisfies 423 ⊆657.
  • B Definitions, Notation, and Harness: The citation audit separates identifier existence from evidential support, with wrong-paper cases defined as found citations whose support label is no.L1 treats notfound as fabrication but excludes transient outcomes; L2 assigns yes, partial, or no support labels and computes wrong-paper rates over found citations.
  • B Definitions, Notation, and Harness: The agentic harness interleaves reasoning and tool calls for up to ten rounds over ten biomedical REST APIs, then grades traces with checklist and citation measures.The trace includes the final answer, cited PMIDs, and tool calls; checklist scores of at least 0.5 count as solves.
  • C Extended Experiments and Robustness: Checklist scoring is designed to resist agentic collapse because nonattempts and zero-tool traces fail grounded-content criteria instead of earning credit for fluent hedging.Collapsed answers can also fail must-avoid criteria when they falsely claim that tools yielded nothing, stabilizing difficulty buckets and the core target.
  • C Extended Experiments and Robustness: Extended experiments unpack pooled headlines through per-model citation audits, robustness probes, behavioral breakdowns, no-tool ablations, checklist and domain checks, and reproducibility accounting.These analyses support evaluation of citation behavior, agentic collapse, judge consistency, domain effects, and the frozen-core construction.

C.1 Citation Audit: Robustness Probes · C.2 Behavioral Divergence

The citation audit shows that wrong-paper errors are understated by a conservative no-only definition and are concentrated more among trial-registry identifiers than PMIDs. Behavioral divergence is substantial, especially on the harder priority track, where two models exhibit sharp agentic collapse.

  • C.1 Citation Audit: Robustness Probes: Figure 5 reports the full per-model L1/L2 breakdown underlying the headline wrong-paper rate.The cited passage identifies this as the robustness-probe context for the headline rate.
  • C.1 Citation Audit: Robustness Probes: 40% of substantive partial citations flipped to “no” in a 25-item probe, implying that including partials would raise the wrong-paper upper bound to ∼25%.The partial bin comprises 28.1% of evaluable citations under the Opus judge, so the no-only definition under-reports rather than inflates failure.
  • C.1 Citation Audit: Robustness Probes: 20.3% of NCT trial IDs were wrong-paper under the Opus judge, versus 13.0% of PMIDs.The corresponding counts were 47/231 NCT IDs and 593/4,545 PMIDs; invented identifiers from GLM-5.1 were predominantly NCT IDs.
  • C.1 Citation Audit: Robustness Probes: 1.07 was the risk ratio linking wrong-paper citations with must-ground satisfaction across 879 jointly labeled answers, with a 95% CI of 0.88–1.31.Among must-ground-passing answers carrying a wrong-paper citation, 73% also included a genuinely supporting citation.
  • C.2 Behavioral Divergence: Cite-rate spanned roughly 10× across models, from GLM at 3.9% to DeepSeek-V4 at 38.5%, indicating sharp divergence in whether models ground answers.The behavioral analysis covers the 1,969-question set, with the full breakdown in Table 6 and Figure 6 in the main body.
  • C.2 Behavioral Divergence: 20.8/19.6/31.3% were the broad-set zero-tool rates for the three roster models.These rates occurred on the 1,969-question set, where zero-tool behavior was only moderate for all three models.
  • C.2 Behavioral Divergence: 65.3% zero-tool behavior for GLM-5.1 and 62.3% for DeepSeek-V4 marked sharp priority-track collapse, while Qwen3.6 reached 22.1%.GLM-5.1’s non-attempt rate was 69.1%, up from 26.2%; DeepSeek-V4’s was 62.3%, up from 0.8%; Qwen3.6’s was 26.3%.

C.3 No-Tool Ablation: Criterion Decomposition · C.4 Leaderboard Robustness to the Checklist Judge · C.5 Per-Domain Breakdown

The no-tool ablation shows that tools provide no criterion-level advantage for GLM-5.1, including on evidence-grounding. Leaderboard rankings remain robust under an independent judge, while per-domain analysis shows wrong-paper rates are broadly distributed across specialties.

  • C.3 No-Tool Ablation: Criterion Decomposition: No-tool GLM-5.1 scores are at least as high as tool-assisted scores on every criterion type across 656 matched core questions.The result is presented as evidence that tool parity is not merely a rubric artifact.
  • C.3 No-Tool Ablation: Criterion Decomposition: 32.5 → 40.2 and 77.7 → 84.7 are the largest no-tool gains, respectively, for must mention and must avoid.The smallest gain is for must acknowledge: 5.5 → 7.5.
  • C.3 No-Tool Ablation: Criterion Decomposition: 17.1 with tools vs 19.3 without shows no tool advantage on must ground, the criterion rewarding real supporting evidence.Must acknowledge rises from 5.5 to 7.5, the smallest gain among the listed criterion types.
  • C.4 Leaderboard Robustness to the Checklist Judge: At most 4.1 points separate per-model solve-rate shifts when six non-roster models are re-graded by independent Opus-4.7 rather than the GLM-5.1 checklist judge.The comparison covers 2,498 question-model pairs.
  • C.4 Leaderboard Robustness to the Checklist Judge: Cohen’s κ = 0.65 measures agreement between the original and independent judges on per-question solve labels.The passage also reports that the frontier ordering remains Gemini < Opus < GPT.
  • C.5 Per-Domain Breakdown: 8.1% in Surgical to 18.1% in Genomics is the Opus wrong-paper range, indicating no single specialty drives the pooled rate.The frozen core spans all twelve domains, with Clinical Medicine contributing the largest share at 21.5%.
  • C.5 Per-Domain Breakdown: Three pooled findings hold up under L1-taxonomy re-aggregation, while one does not.The supplied passage introduces this split before detailing the wrong-paper and frozen-core findings.

C.6 Reproducibility of the Frozen Core: Full Accounting

Re-decodes reproduce per-question scores closely, but frozen-core membership churns substantially, making it a single-T=0 snapshot rather than a seed-stable partition. Despite this churn, frontier comparisons preserve their ordering on stable items.

  • Reproducibility: 0.09–0.14 mean absolute checklist-score deltas and 0.78–0.81 fail-label agreement show close per-question reproducibility across five-day T=0 re-decodes.The model output itself is deterministic, while the test probes live-API and agentic nondeterminism.
  • Membership churn: 46.5% of boundary-proximal items flip membership, while 34.7% of near-threshold deep failures also flip to pass.Live-API drift and erratic tool use drive the flips; Qwen3.6 and DeepSeek-V4 account for 19 of 23 flips.
  • Membership churn: The authors retract the earlier 85.8% retention estimate and report no single retention figure because the frozen core is a single-T=0 snapshot.The earlier estimate assumed deep failures were fixed.
  • Comparison stability: 48.1/61.0/77.9% frontier solve rates on the 77 flipped items indicate that these items are easier for frontier agents.The ordering remains Gemini < Opus < GPT.
  • Core robustness: Jaccard 0.70–0.77 under single-model substitution shows the core is not idiosyncratic to the three roster models.This result comes from re-deriving the all-three-fail set using alternate triples.

D Extended Discussion, Release, and Ethics

The discussion frames wrong-paper citation as the medically consequential failure mode beyond citation existence, while positioning open questions as tests of abstention and faithfulness. It also details empirically calibrated difficulty, reproducibility safeguards, citation-validation limits, release practices, and dual-use risk.

  • Citation severity: 15.9% of real citations linked to unsupported claims, despite only ≈0.7% non-existent identifiers among 4,863 citations.The paper distinguishes citation existence from citation correctness and explicitly limits the claim to sourcing and attribution rather than patient impact, claim veracity, or reader trust.
  • Faithfulness and abstention: Open questions expose two failures: confident confabulation with unsupporting citations and agentic collapse that stops engagement instead of abstaining honestly.The benchmark treats these as opposite failures in the choice between abstention and assertion.
  • Difficulty construction: ≈1 in 4 questions became solvable after re-measuring the “solve rate ≈0%” core at T=0, showing stochastic decoding can distort empirical difficulty.Difficulty bucketing must therefore fix and report decoding temperature.
  • Difficulty construction: 34.7% of near-threshold “deep” failures flipped on re-decode, requiring difficulty measurement over a frozen tool environment with recorded and replayed responses.The released frozen replay cache instantiates this reproducibility correction.
  • Citation validation: ≈74% of LLM-synthesized gold-answer citations pointed to the wrong paper despite ≈100% real PMIDs, with independent judging estimating 72.8% versus 73.5%.The audit covered n ≈360 citations and was presented as not being a single-judge artifact.
  • Release and ethics: The release fixes 423 frozen-core task IDs and an ordered tool-response replay cache, because live reruns produce a 30–46%-different frozen core.The cache makes six audited runs replayable and re-gradable, but scoring new models still requires live tool access; the benchmark also acknowledges dual-use risks involving authoritative-looking medical misinformation.
Loading 2606.21959v1…