Source-linked AI summary
Quantifying Claim Smoothing by LLM Research Agents
Tiasa Singha Roy, Manya Wadhwa, Greg Durrett
TL;DR
LLM research agents produce fluent, cited reports whose conclusions can be more decisive than their sources warrant. SMOOTHSCORE scores claim calibration against evidence organized into aspects. Across four benchmarks and seven systems, smoothing is pervasive, and more thorough engagement with cited documents can help reduce it.
Problem
Long-form reports can misrepresent or overstate cited sources, while claim-verification methods do not assess calibration and scope across potentially conflicting evidence.
Source: §1 · pp. 1–2Method
SMOOTHSCORE extracts conclusive claims, gathers supporting and qualifying evidence from cited documents, organizes it into claim-specific aspects, and scores calibration.
Source: §1 · p. 2, §2.1 · p. 3, §2.2 · p. 4Results
Across four benchmarks and seven systems, smoothing is pervasive; on average, at least 25% of generated claims per system receive SMOOTHSCORE ≥3.
Source: §1 · p. 2Takeaways & Limitations
Limited interaction with cited documents contributes to smoothing, while actively engaging with full documents can help reduce it.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Deep research, or any use of large language models (LLMs) to read a large document collection and produce insights, requires synthesizing information from different sources. In this work, we find that LLMs produce long-form research reports that read as confident syntheses but which fail to accurately reflect the underlying documents. We call this failure smoothing. In collapsing diverse and potentially conflicting sources into a single narrative, the model discards the evidential "jaggedness" of its corpus and asserts conclusions more decisive than the citations can support. We introduce a claim level scoring pipeline, SmoothScore, to quantify this effect. Our pipeline extracts key conclusive claims from model generated reports, then finds evidence relevant to each claim and clusters that evidence along high level aspects. Finally, each claim is scored with respect to the evidence to quantify the degree of smoothing done by the agent. Across 7 systems and 4 datasets, we find that agents consistently produce stronger claims than warranted.
1 INTRODUCTION
LLM research agents synthesize large collections into fluent, cited reports, but their conclusions can overstate what the sources support. The paper introduces SMOOTHSCORE to measure this smoothing and finds it widespread across systems and benchmarks.
1 INTRODUCTION
- Fluent, well-organized, densely cited reports can be treated as substitutes for reading the literature even when they misrepresent or overstate source support.Source: §1 · p. 1
- Smoothing is when synthesized claims are more decisive than cited documents support, such as omitting mixed evidence or broadening a narrowly demonstrated result.Faithful claims should hedge when evidence is mixed and narrow their scope when support is narrow.Source: §1 · pp. 1–2
- SMOOTHSCORE measures whether synthesized claims are calibrated to cited documents, addressing a gap in verification methods that judge support but not calibration or scope across conflicting evidence.Figure 1 illustrates how a claim can appear supported by some citations while other retrieved evidence qualifies it.Source: Figure 1 · p. 1, §1 · p. 2
- For each conclusive claim, SMOOTHSCORE gathers supporting and qualifying or contradictory evidence, clusters it into aspects, and scores calibration of claim strength and scope on a 1–5 scale.Source: §1 · p. 2
- At least 25% of generated claims per system receive SMOOTHSCORE ≥3 on average, and most systems exceed 50%.Up to 35% of Gemini Deep Research’s claims show severe smoothing (≥4).Source: §1 · p. 2
2 SMOOTHING IN LLM RESEARCH AGENTS
Smoothing occurs when a report’s aggregated-evidence conclusion is partly supported but stronger or broader than the cited evidence warrants. SMOOTHSCORE evaluates such claims by extracting relevant evidence, organizing it by claim-specific aspects, and scoring how faithfully each claim represents that evidence.
2.1 DEFINING SMOOTHING
- A claim is smoothed when it draws a conclusion from aggregated evidence, receives some support, but is stronger or broader than the evidence warrants.A claim can appear supported by cited sources yet be true only under certain conditions, so checking individual citations alone can miss smoothing.Source: §2.1 · pp. 2–3
- Counting supporting versus refuting documents alone misses nuances in how a claim is supported or refuted.The framework instead organizes evidence around dimensions tailored to each claim and its cited evidence (Kargupta et al., 2025).Source: §2.1 · p. 3
2.2 MEASURING SMOOTHING WITH SMOOTHSCORE
- SMOOTHSCORE extracts conclusion-bearing claims from reports, then gathers relevant evidence from the cited documents using support and skeptic agents.The skeptic agent also gathers evidence about conditions, limitations, and relevant alternative approaches to provide context for scope.Source: §2.2 · pp. 3–4
- Claim-specific aspects group evidence about the same dimensions across documents, enabling like-for-like comparison of supporting and opposing findings.Aspects are consolidated across documents; evidence is assigned by what it discusses, regardless of stance, source, or strength.Source: §2.2 · p. 4
- A judge scores each claim against its aspect-organized evidence, assessing whether omitted conditions or scope change the claim’s central takeaway.The five-point rubric includes score 5 cases of pure hallucination, but the authors primarily frame SMOOTHSCORE as a method for analyzing smoothing.Source: §2.2 · p. 4
3 EXPERIMENTAL SETUP
The evaluation uses four research-report-generation benchmarks and seven systems spanning web-augmented agents, proprietary deep-research systems, and open research agents. It reports mean SMOOTHSCORE and the fraction of claims in each score bin, treating scores of at least 3 as indicating materially misleading claims.
3 EXPERIMENTAL SETUP
- Four datasets—ResearcherBench, ScholarQA-CS2 (Expert-Written), DeepResearch Bench II, and ResearchRubrics—provide tasks curated for research report generation.Queries were manually selected to ensure they require some degree of multi-document synthesis.Source: §3 · p. 5
- Seven systems span web-augmented agents, proprietary end-to-end deep-research systems, and open research agents with specialized retrieval and reasoning pipelines.The systems include GPT Luna 5.6, GPT Sol 5.6, Claude Sonnet 5, Claude Opus 5.5, Gemini Deep Research, DR Tulu, and QUEST.Source: §3 · p. 5
- The evaluation reports mean SMOOTHSCORE and the fraction of claims in each score bin, with scores ≥3 indicating materially misleading claims.Both metrics are reported for each system and benchmark.Source: §3 · p. 5
4 RESULTS
Smoothing is widespread across systems and datasets, but severity varies; qualitative observations describe recurring claim patterns and question framings where it is more pronounced.
4.1 SMOOTHSCORE RESULTS
- Open-source DR Tulu and QUEST have at least ∼59% and ∼55% of claims scoring ≥3, while 7–23% score ≥4.QUEST draws from among the smallest evidence pools, yet produces up to ∼7% and ∼4% of claims scoring 5 on two benchmarks.Source: §4.1 · p. 5
- Sonnet 5 and Opus 5.5 produce almost 1.4–2.5× the fraction of claims scoring 3–4 as Luna and Sol; Luna and Sol have at least 36% scoring 1.Neither Sonnet 5 nor Opus 5.5 produces score-5 claims above 1.1% on any dataset.Source: §4.1 · p. 5
- Gemini’s mean SMOOTHSCORE is 3.05–3.27, with 85–93% of its claims scoring ≥3.Gemini also draws roughly 2–5× more documents than Luna or Sol with websearch and produces the longest reports on every dataset.Source: §4.1 · p. 5
- At least 22% of claims per system-dataset case score ≥3; Gemini’s severe claims reach 22–35%, versus 1–4% for Luna and Sol.Table 1 reports SMOOTHSCORE ≥3 as material smoothing and ≥4 as severe smoothing.Source: §4.1 · p. 5, Table 1 · p. 6
- Across systems, more cited documents (ρ = 0.37, p = .056) and longer claims (ρ = 0.43, p = .02) correlate with more smoothing, but document-count effects disappear within systems.Within-system report-length correlation is small (ρ = 0.12) and not significant in any one system; other factors could also influence smoothing.Source: §4.1 · pp. 6–7, Figure 3 · p. 6
4.2 QUALITATIVE OBSERVATIONS
- Comparative and topic-overview queries are settings where smoothing is more pronounced, as agents may drop qualifications or generalize findings.Comparative questions can presuppose a choice; overview questions can prompt a single conclusion from multiple results or generalization from one finding.Source: §4.2 · p. 7
- Recurring patterns are unsupported composition, scope expansion, and premature closure in claims.Table 2 pairs illustrative claims and SMOOTHSCORE with summaries of supporting evidence and limitations.Source: §4.2 · p. 7, Table 2 · p. 7
- On ResearcherBench, evidence-only revisions can increase gold-rubric scores, matching or exceeding raw-document access at a fraction of the size.The comparison covers reports generated by Luna and Sonnet 5 and scores them with GPT-5.Source: Table 4 · p. 8
5 VALIDATING SMOOTHSCORE
The authors validate SMOOTHSCORE’s scores against human judgments and test whether its collected evidence improves report revision. The results support both judge-score agreement and the usefulness of the evidence for an independent expert-written evaluation.
5 VALIDATING SMOOTHSCORE
- The validation section checks both human agreement with the judge’s scores and the usefulness of evidence surfaced by the pipeline.The evidence usefulness check uses ResearcherBench gold rubrics for overall report quality.Source: §5 · p. 8
5.1 HUMAN VALIDATION
- SMOOTHSCORE scores were within one point of an annotator’s score on 93% of 73 claims, and differed by two or more points on 7%.At the ≥3 threshold, 62% of judge-flagged claims were also scored ≥3 by Annotator 1; annotators agreed with κ = 0.62.Source: §5.1 · p. 8
5.2 RELEVANCE OF EVIDENCE COLLECTED BY SMOOTHSCORE
- SMOOTHSCORE evidence improved expert-rubric scores for Luna by 4.4 and Sonnet 5 by 12.0 (both p < .001), matching or exceeding revision with full documents.The collected evidence used roughly one-fifth to one-tenth as many tokens as the full documents; the evaluation does not test evidence quality relative to generated claims.Source: §5.2 · p. 8
6 LACK OF ENGAGEMENT WITH DOCUMENT CONTENT
Luna and Sonnet 5 rarely open the cited documents in their WebSearch reports. Providing the cited documents together with calibration instructions reduces smoothing more than either intervention alone, while reports retain their central takeaway.
6 LACK OF ENGAGEMENT WITH DOCUMENT CONTENT
- Luna opened only 19% of cited pages, while Sonnet 5 opened only 2% of cited links in the evaluated WebSearch reports.Luna’s traces show it worked almost entirely from search snippets truncated by the tool (median 585 characters).Source: §6 · p. 8, Table 5 · p. 9
- With documents and calibration instructions together, materially smoothed claims fell from 29% to 12% for Luna and from 61% to 25% for Sonnet 5.Mean σ fell to 1.52 for Luna and 1.78 for Sonnet 5; severe smoothing (σ ≥4) fell to 0.5% and 2.2%, respectively. Table 6 compares WebSearch reports, cited documents in context, and each setting with instructions.Source: Table 6 · p. 9, §6 · p. 9
- Instructions alone or document access alone did not reduce smoothing as much as the combined intervention for either system.The paper concludes that calibration instructions are most effective when the agent has evidence available to act on them.Source: §6 · p. 9
- Reports written from full documents were 1.2–1.6× longer yet contained slightly fewer extracted claims, while their central takeaway remained the same.The claims tended to drop conclusive language and provide limitations as needed; the paper reports that smoothing tracks the added reasoning.Source: §6 · p. 9, Table 7 · p. 10
7 RELATED WORK
Prior work examines claim verification, multi-document summarization, and deep-research factuality, but often centers on whether claims have support. This paper focuses on whether reports preserve the scope and qualifications of cited evidence.
Claim decomposition and aspect-based verification
- Claim-level factuality checks can miss smoothing because a claim may be supported by every cited document while omitting relevant information.The paper organizes supporting and qualifying evidence by aspects such as viewpoint, strength, and certainty so they can be considered against each other.Source: §7 · p. 10
Multi-Document Summarization
- Prior studies report overgeneralization in scientific summaries and factual inaccuracies or certainty errors in medical summaries; this work extends the concern to agent-written multi-document research reports.The cited summary studies cover a single document or a small set of abstracts; this paper proposes a claim-level measure grounded in documents a report cites.Source: §7 · p. 10
Correctness evaluation for deep research systems
- Deep-research audits often operationalize factuality as citation validity, but individually supported claims can still present mixed or conditional evidence as decisive conclusions.The paper argues that faithful presentation requires claim-level analysis calibrated to underlying evidence.Source: §7 · p. 10
8 CONCLUSION
The conclusion is that LLM reports can smooth source evidence into oversimplified or insufficiently supported conclusions. SMOOTHSCORE quantifies this behavior, and the authors suggest that more thorough source engagement can improve it.
8 CONCLUSION
- The authors conclude that SMOOTHSCORE quantifies smoothing across state-of-the-art agents and suggest that more thorough engagement with cited documents can improve it.They characterize smoothing as oversimplified conclusions that are not fully backed by cited evidence.Source: §8 · p. 11
AI USE STATEMENT
The statement describes human-originated research hypotheses and experiment design, with generative AI used in the studied systems and to assist analysis. The method details describe merging overlapping aspect proposals into a hierarchy that preserves evidence traceability.
AI USE STATEMENT
- The research hypotheses and experiment design were human-originated; generative AI supported the studied systems, analyses, and revisions, but not substantive writing.The authors take responsibility for the final content.Source: AI USE STATEMENT · p. 11
A.1 METHOD DETAILS
- Aspect induction addresses duplicate or differently specific proposals by grouping evidence on the same topic across documents.The proposals may use different names or levels of specificity.Source: §A.1 · p. 14
- The merge step combines overlapping proposals, adds subaspects for distinct related findings, and retains each node’s proposing documents for traceability.The hierarchy consists of aspect and subaspect nodes linked by parent–child relations.Source: §A.1 · p. 14
- All pipeline stages use GPT-5-mini except the SMOOTHSCORE judge, which uses GPT-5; calls use medium reasoning effort with default sampling.Source: §A.1 · p. 14
A.2 DATASET CURATION
This section describes benchmark-query selection and claim/evidence extraction, then explains how SmoothScore derives claim-relevant aspects, assigns evidence, and judges the qualifications needed for claims to represent that evidence faithfully. It also reports generally weak and inconsistent relationships between generation features and SmoothScore, with only a small association for report length.
A.2 DATASET CURATION
- The curated datasets include 65 ResearcherBench questions and 25 ScholarQA-CS2 questions, plus candidate pools of 66 DeepResearch Bench II and 101 ResearchRubrics tasks filtered for high synthesis and high question openness.Queries were annotated for synthesis and whether they require one definitive answer before filtering.Source: §A.2 · pp. 14–15
A.3 MODELS
- The evaluation covers seven open- and closed-source systems, including frontier models.The supplied passage points to a table of model identifiers but does not provide those identifiers.Source: §A.3 · p. 15
A.4 RELATIONSHIP BETWEEN SMOOTHSCORE AND GENERATION FEATURES
- Only report length shows a small, consistent within-system association with SmoothScore; feature relationships are otherwise weak and vary in direction.The analysis covers report length and retrieved evidence across datasets and systems, with correlations measured across and within systems.Source: §A.4 · p. 15, Table 12 · p. 16
B.1 CLAIM GENERATION
- Claim extraction selects load-bearing, checkable conclusions in the report’s own wording, preserving their hedges, scope, and stated contrasts rather than extracting fragments or strengthening claims.The prompt favors complete conclusions and skips background, setup, and individual supporting findings that roll up into a larger verdict.Source: §B.1 · p. 17
B.2 SUPPORTING PASS
- The supporting pass extracts distinct findings from one document that support the claim as actually stated, including evidence for any stated limitations or concessions.Each finding is tied to the specific claim aspect it bears on; evidence against the claim is left to a separate process.Source: §B.2 · pp. 18–19
B.3 SKEPTIC PASS
- The skeptical pass labels findings as counters, concession-matches, or rival-wins according to how they relate to the claim’s assertions, scope, and stated qualifications.Carefully scoped claims may have no counter-items, while concession-match and rival-win items are still extracted when present.Source: §B.3 · pp. 19–20
B.4 ASPECT GENERATION
- Aspects are limited to substantive content a document actually discusses that bears on the claim; the prompt typically requests 2–3, but accepts one or an empty list when warranted.Each aspect includes a lowercase label, a description of what the document says and why it matters to the claim, and six keywords.Source: §B.4 · p. 21
B.5 ASPECT MERGE
- The merger targets 4–5 claim-relevant top-level aspects and roughly 4–8 total nodes, merging duplicates while grounding every node in document proposals.Subaspects are added only for distinct, separately evidenced subdivisions; each node retains its source document indexes.Source: §B.5 · p. 21
B.6 EVIDENCE ASSIGNMENT
- Evidence quotes are assigned by topic to the best-fitting leaf aspect; ROOT handles whole-claim evidence, while UNASSIGNED is reserved for quotes fitting neither an aspect nor ROOT.Assignment judges what each quote is about, not whether it supports or opposes the claim.Source: §B.6 · p. 22
B.7 SMOOTHSCORE JUDGE
- The judge evaluates the claim’s role and actual scope, weighs relevant evidence and objections, and resolves whether it survives as stated, needs a limitation, or needs substantial revision.It treats supplied evidence as the entire universe, weighs relevance and directness rather than quote counts, and checks whether apparent counterevidence addresses the claim as scoped.Source: §B.7 · pp. 22–23
- The judge assigns scores 1–5 using the minimum semantic repair needed: none, append, revise, recast, or replace, with each repair level mapped to its corresponding score.A needed caveat alone is not enough for revise; revise changes a material predicate or scope, recast substantially narrows or rebalances the conclusion, and replace requires a different takeaway.Source: §B.7 · pp. 23–24
- Scores reflect how evidence bears on the central takeaway: a claim may hold with qualifications, be contested or mixed, or lack evidential support for its asserted conclusion.Material qualifications count only when not already conveyed by the claim’s wording or context; subsidiary inaccuracies do not change the score unless they alter the takeaway.Source: §B.7 · p. 24
C JUDGE VALIDATION
The judge validation reports that GPT-5 agrees with an annotator about as closely as a second annotator does, while other judges rank claims similarly but differ in calibration. A worked example applies the pipeline from claim extraction through smoothing judgment, alongside report-generation setups and document-engagement examples.
C JUDGE VALIDATION
- GPT-5 agrees with Annotator 1 as closely as the second annotator does (ordinal α 0.62 vs. 0.64); Gemini 2.5 Pro and Claude Sonnet 4.5 agree less well (α 0.36 and 0.04) but rank claims similarly (ρ 0.60 and 0.51).All judges preserve the annotator’s system ranking, Luna < DR Tulu < Gemini; Gemini 2.5 Pro and Sonnet 4.5 score systematically higher.Source: C JUDGE VALIDATION · p. 25, Table 13 · p. 25, Table 14 · p. 25
1. Claim extraction
- The worked example extracts a claim about RL with verifiable rewards becoming mainstream and reconstructs the full text of its cited documents.Source: §1 · p. 25, Document reconstruction · p. 25
2a. Supporting-evidence extraction
- Across claim–document pairs, the supporting pass extracts 61 items, while the countervailing pass extracts 77 items that contradict, weaken, or restrict the claim.Source: 2a. Supporting-evidence extraction · p. 26
3b. Aspect consolidation
- The example proposes document-specific evaluation aspects, then merges overlapping proposals into a hierarchy including empirical effectiveness, verifier reliability, and adoption readiness.The hierarchy also distinguishes objective-check gains from settings where RL is unnecessary.Source: 3a. Local aspect induction · p. 26, 3b. Aspect consolidation · p. 26
4. Evidence assignment
- Evidence is assigned to the most specific applicable aspect; the objective-check gains node has 31 supporting and 9 countervailing items, the only node where support predominates.Examples include DeepSeek-R1-Zero’s AIME 2024 pass@1 increase from 15.6% to 77.9%, AlphaProof’s competition-math successes, and AlphaGeometry2’s 84% solve rate using a non-RL-centered pipeline.Source: §4 · p. 27
5. Smoothing judgment
- The example’s judgment says the forecast largely holds but needs limits: mainstream RLVR use is scoped to verifier-evaluable settings and constrained by verifier availability, risks, regimes, and non-RL alternatives.The repair is APPEND, with a smoothing score of 2.Source: §5 · p. 27
E.3 SETUP FOR DOCS IN CONTEXT, WITH INSTRUCTIONS
- The report prompts cover web search and provided-document workflows; calibration instructions ask reports to state conditions, surface disagreement, and avoid conclusions stronger than the sources support.The no-instructions provided-documents setup is also included.Source: §E.1 · p. 27, §E.2 · p. 28, §E.3 · p. 28
E.5 IMPROVED CLAIMS UNDER DOCUMENT ENGAGEMENT
- Selected Luna cases show the same conclusion changing in scope and strength when the cited documents are provided, while Table 16 describes same-conclusion claims across the two arms.Claims are extracted independently per report.Source: §E.5 · p. 29, Table 16 · p. 30
F DOCUMENT ENGAGEMENT DETAILS
Document engagement is measured through whether cited pages were opened, fetched, or visited, but system interfaces limit what can be observed. Open-source-system examples are collected in Table 18.
F DOCUMENT ENGAGEMENT DETAILS
- A cited link counts as opened when the system retrieves its page during generation; the analysis canonicalizes URLs and counts each link once per dataset.Table 17 reports the share of cited links opened by system and dataset.Source: F DOCUMENT ENGAGEMENT DETAILS · p. 31, Table 17 · p. 31
- Luna and Sol open 9–19% and 16–30% of cited links, Sonnet 5 and Opus 5.5 fetch 1.1–3.5% and 0.2–2.3%, and DR Tulu browses only five times overall.All five systems cite mostly from search results rather than pages they open; 45 of Luna’s 65 ResearcherBench reports open no cited page.Source: F DOCUMENT ENGAGEMENT DETAILS · p. 31, Table 17 · p. 31
- QUEST’s visits expose goal-conditioned extracts rather than raw page content, so its high visit rate does not mean it reads what it cites.The extract is generated by GPT-5-mini from a goal QUEST states for the visit.Source: F DOCUMENT ENGAGEMENT DETAILS · p. 31
- Gemini Deep Research engagement cannot be measured because its API returns only the final report and citation links, without browsing or tool steps.Source: F DOCUMENT ENGAGEMENT DETAILS · p. 31
G.1 OPEN-SOURCE SYSTEMS
- Table 18 provides examples of claims from the open-source systems DR Tulu and QUEST.Source: §G.1 · p. 31
G.2 CLOSED-SOURCE SYSTEMS
Examples from closed-source systems include high-severity claims, and one cited example is judged unsupported in key respects.
G.2 CLOSED-SOURCE SYSTEMS
- The example claim reports scores for image-text matching, captioning, question answering, and other tasks, and asserts DDT superiority in compositional edits and counterfactual interpolation.It attributes that superiority to recursive, attribute-rich tokens said to mimic natural-language structure.Source: §G.2 · p. 32
- The evidence does not show DDT outperforming all previously reported models on several listed benchmarks and provides no support for object detection.Source: §G.2 · p. 32
- Table 19 presents high-severity claims from closed-source systems alongside the SMOOTHSCORE judge’s verdict.Source: §G.2 · p. 31, Table 19 · p. 33