PaperlayerView exact revision on arXiv

Source-linked AI summary

Redesigning and Auditing Deep Research Writing for Faithful Reports

Hiroaki Hayashi, Pranav Narayanan Venkit, Prafulla Kumar Choubey, Chien-Sheng Wu

arXiv:2608.28643v1cs.CLcs.AI

TL;DR

Deep-research rubrics can obscure whether reports retain necessary evidence and attribute claims correctly. The paper introduces CLAIMPROBE for claim-level auditing and CLAIMWRITER for source-linked hierarchical writing, finding improved claim-level faithfulness and recall across three hosts while largely preserving rubric performance and supporting localized updates.

  • Problem

    Holistic deep-research rubrics often hide omitted key facts, unsupported claims, and source misattribution, limiting visibility into report faithfulness and completeness.

  • Method

    CLAIMPROBE audits claims against retrieved evidence, while CLAIMWRITER organizes extracted source facts into a query-derived hierarchy before drafting source-linked reports.

  • Results

    Across three deep-research hosts, replacing only the report writer improves claim-level faithfulness and necessary-fact recall while largely preserving rubric-based performance.

  • Takeaways & Limitations

    Fact-linked representations support report writing and localized revision that preserve evidence structure, attribution, and maintainability as sources change.

  • Takeaways & Limitations

    The dynamic-update study uses a controlled setting with five tasks, two source-change budgets, and one numerical fact update per changed source.

Abstract

from arXiv · show

Rubric-based evaluations of deep-research (DR) systems often obscure fine-grained factual failures in generated reports. We introduce CLAIMPROBE, a claim-level audit that decomposes DR reports into claims and measures hallucination, misattribution, citation hygiene, and necessary-fact recall against retrieved evidence. Using CLAIMPROBE, we find that strong DR pipelines can omit key evidence and misattribute claims even when their rubric scores remain stable. We then propose CLAIMWRITER, a hierarchical claim-based writer that extracts source facts, maps them to a query-derived outline, and drafts each section from a source-linked claim representation. Across three prior DR frameworks, replacing only the report writer with CLAIMWRITER reduces hallucination by 2.6 to 4.5 times and improves necessary-fact recall by 1.2 to 1.7 times, while largely preserving overall report quality. CLAIMWRITER also enables localized revision: when sources change, it propagates changed source facts into revised reports at the highest rate among update methods, while also being more cost-effective.

1 INTRODUCTION

CLAIMPROBE exposes claim-level failures that holistic rubrics can hide, while CLAIMWRITER reorganizes source facts before drafting to improve faithfulness and recall. Across three hosts, it preserves much of overall report quality and supports efficient updates when evidence changes.

  • Motivation: Holistic rubrics can hide omitted facts, unsupported claims, and source misattribution in deep-research reports.These failures matter because reports are used by downstream enterprise decision makers.
  • CLAIMPROBE: CLAIMPROBE decomposes reports into claims and measures hallucination, misattribution, citation hygiene, and necessary-fact recall against collected evidence.It preserves claim-level diagnostics while aggregating judgments into report-level metrics.
  • CLAIMWRITER: CLAIMWRITER organizes source-grounded facts into an explicit hierarchy before drafting, preserving links between claims and supporting sources.It replaces only the report-generation module of existing deep-research systems.
  • Evaluation: Across three hosts, CLAIMWRITER improves claim-level quality by reducing hallucination and misattribution while increasing necessary-fact recall, with only slight rubric-score movement.Upstream search and evidence collection remain fixed during the writer replacement.
  • Incremental revision: When sources change, CLAIMWRITER maps changed facts to affected claims, revises corresponding sections, and preserves unaffected report content.The incremental method is designed for settings where report structure remains stable while individual facts evolve.

2 CLAIMPROBE: A CLAIM-LEVEL REPORT AUDITING FRAMEWORK

CLAIMPROBE audits deep-research reports by aligning extracted report claims with source facts and computing four measures of factual support, attribution, citation completeness, and evidence coverage. Its local LLM judgments are calibrated against human annotations.

  • Overview: CLAIMPROBE evaluates each report claim against collected source evidence and computes four report-level measures.The measures target failure modes in long-form factuality and commercial answer-engine evaluation.
  • Metrics: HALL detects claims unsupported by any retrieved source, while MIS checks whether the cited source actually supports the claim.UNC measures supported claims that lack citations; REC measures coverage of highly relevant source facts.
  • Metrics: REC measures the fraction of highly relevant source facts that appear in the report.Source facts are labeled HIGH, RELATED, or UNRELATED, and only HIGH-relevance facts count toward REC.
  • Procedure: The pipeline separately processes sources and reports, aligns claims with source facts, and computes report-level metrics from cached judgments.For alignment, it retrieves the top-K=20 facts and applies an LLM judge in both claim-to-source and source-to-report directions.
  • Alignment: Claim-to-source matching requires precise factual support, whereas source-to-report matching accepts paraphrase, summarization, and preserved core coverage.Numeric values receive strict factual checking in claim-to-source matching.
  • Calibration: Human validation on n=100 annotations produced Cohen’s κ values of 0.484, 0.797, and 0.743 for hallucination, misattribution, and relevance.The judge scores one claim against a short evidence shortlist at a time.

3 CLAIMWRITER: HIERARCHICAL CLAIM-BASED WRITER MODULE

CLAIMWRITER treats source facts as first-class objects, organizes them under a query-derived hierarchy, and drafts reports with explicit claim-to-source links. Its fact-linked representation also localizes revisions to sections affected by source changes.

  • Writer design: CLAIMWRITER organizes source facts into a structured topic hierarchy before drafting and resolves claim-to-source links during final report construction.This replaces concatenating retrieved content for direct long-context synthesis.
  • Fact organization: The writer derives an initial outline from the user query, extracts and deduplicates source facts, scores relevance and specificity, and assigns each fact to one outline topic.The outline is not conditioned on extracted evidence, keeping it aligned with the query’s information requirements.
  • Representation: Each extracted fact is linked to a topic assignment, citation marker, and supported report section.This creates a fact-linked report representation for downstream revision.
  • Source updates: For changed sources, CLAIMWRITER identifies updated, removed, and newly added facts, then revises sections associated with affected topics.New facts are assigned to appropriate topics in the existing outline.
  • Source updates: Restricting rewriting to sections whose assigned facts changed reduces token cost and avoids unnecessary edits to unaffected text.The procedure preserves the report structure during updates.

4 EXPERIMENTS

The experiments test whether CLAIMWRITER preserves holistic quality, improves claim-level faithfulness and recall, and updates reports effectively under source changes. They compare writer variants across three host pipelines and multiple audit settings.

  • Evaluation goals: The experiments evaluate holistic quality, necessary-fact retention without fabrication, and report updating under source changes.These questions target deployment concerns for deep-research stacks.
  • Setup: The setup uses 50 English and 50 Chinese deep-research tasks across Enterprise Deep Research, NVIDIA AI-Q, and OpenDeepResearch.CLAIMWRITER replaces each pipeline’s original report-writing step.
  • Evaluation setup: CLAIMPROBE evaluation uses text-embedding-3-small, K=20 retrieval in both directions, and GPT-5.4-mini on 100 tasks plus GPT-5.5 on a 10-task subset.RACE scores use the original GPT-5.5 configuration on 50 English tasks.

5 RESULTS

CLAIMWRITER largely preserves holistic report quality while exposing substantial claim-level gains that RACE scores alone do not reveal.

  • RACE results: CLAIMWRITER leaves comprehensiveness and insight essentially unchanged across hosts, while readability decreases by 3.8–5.9 points.The resulting net decrease in Overall is at most 1.8 points in every host.
  • CLAIMPROBE results: CLAIMWRITER lowers hallucination and misattribution while increasing necessary-fact recall, despite only small changes in holistic scores.On EDR with GPT-5.4-mini, HALL falls from 15.89 to 5.02, MIS from 18.94 to 5.43, and REC rises from 36.83 to 45.85.
  • CLAIMPROBE results: Across hosts and judge models, CLAIMWRITER and CLAIMWRITER-BU are consistently the strongest writers, while the baseline is weakest.GPT-5.4-mini produces nearly the same relative writer ranking as GPT-5.5, although absolute scores differ moderately.

6 DISCUSSION

Claim-level audits reveal that holistic quality and source faithfulness can diverge, while fact-linked structure supports more accurate, localized updates at manageable cost.

  • Holistic quality and faithfulness: Holistic report scores can remain close to baseline while claim-to-source relationships change substantially.RACE and CLAIMPROBE together show that surface-level quality and source faithfulness are partly separable.
  • Holistic quality and faithfulness: A claim-level audit should supplement holistic rubrics by checking source attribution and necessary-fact retention.CLAIMPROBE is designed to expose which claims fail and why.
  • Claim-level judging: CLAIMPROBE reduces page-level confounds by evaluating local claim–evidence comparisons rather than either artifact end to end.HALL and MIS compare one report claim against a top-K=20 evidence shortlist, while REC checks coverage of one highly relevant source fact.
  • Dynamic report update: The update evaluation perturbs one numerical fact per changed source under budgets of 5 and 25 changed sources.Perturbations cover years, quantities, trends, or combinations and are validated against surrounding source context.
  • Dynamic report update: Update yield is the clean rate at which a regenerated report adopts each new perturbed value and removes the old value.Output-token cost is emphasized because these updates are typically scheduled rather than latency-critical.
  • Dynamic report update: CLAIMWRITER’s fact graph localizes revisions and achieves substantially higher update yield near diff-style output-token costs.Full replay sets the cost ceiling and misses many changes, whereas monolithic diffs lack a stable changed-fact handle.
  • Cost trade-offs: CLAIMWRITER’s higher first-run input cost is amortized across repeated updates through a lower incremental-update budget.For one-shot reports, users must weigh that initial cost against claim-level structure and maintainability.
  • Interpreting secondary metrics: High UNC should be interpreted as a citation-hygiene gap rather than a fabrication signal when HALL and MIS are low and RECstr rises.The uncited residue increasingly consists of grounded background and connective tissue.

7 RELATED WORK

Related work spans holistic deep-research benchmarks, deployed research pipelines, and claim-level methods for factuality, attribution, and retrieval-grounded faithfulness.

  • Deep-research systems: Production deep-research systems emphasize planning, search, and stopping criteria, while report generation is often treated as final long-context synthesis.Examples include Google Gemini Deep Research, OpenAI Deep Research, Enterprise Deep Research, NVIDIA AI-Q, and Open Deep Research.
  • Evaluation research: DeepResearch Bench and related benchmarks primarily evaluate deep-research reports at the document level.This work complements those benchmarks by focusing on claim-level report faithfulness and attribution.
  • Claim-level factuality and attribution: FActScore and SAFE decompose long-form generations into atomic facts or claims, while ALCE, attributed QA, and RAGAS assess citation quality or retrieval-grounded faithfulness.Prior studies also document hallucinations and citation errors in deployed search and question-answering systems.

8 CONCLUSION

The paper introduces claim-level auditing and fact-linked writing for faithful deep-research reports, then extends the same representation to localized revision under source drift.

  • Contributions: CLAIMPROBE audits hallucination, attribution, citation quality, and coverage, while CLAIMWRITER improves claim-level faithfulness and recall across three deep-research hosts.Replacing only the writer preserves rubric-based performance while improving claim-level outcomes.
  • Dynamic revision: The fact-linked representation supports localized report revision that produces more accurate updates at lower output-token cost.The conclusion motivates report-writing and update systems that preserve evidence structure, attribution, and maintainability.

LIMITATIONS

The evaluation has limitations in judge calibration, controlled update conditions, and cost measurement. These constraints bound how broadly the findings should be interpreted.

  • Judge calibration: CLAIMPROBE uses an LLM judge validated against human annotations, but deployment in new domains requires recalibration.The judge scores one claim against a top-K=20 evidence shortlist; users should recalibrate it for their task distributions.
  • Update scope: The dynamic-update study covers five tasks, two source-change budgets, and one numerical fact update per changed source.The controlled design isolates changed-fact propagation but does not cover new sources, removed evidence, qualitative changes, or report restructuring.
  • Cost boundary: The reported cost advantage is strongest for repeatedly updated reports because CLAIMWRITER pays a higher first-run cost to materialize its fact graph.For one-shot reports, users must weigh the higher initial cost against claim-level structure and maintainability.
  • Evaluation design: The audit metrics rely on retrieved evidence and direction-specific alignment judgments, with strict numeric matching for report-to-source evaluation.The supplied evaluation materials define HALL, HALLstr, MIS, and UNC over claim-to-evidence judgments.

C ROBUSTNESS TO THE JUDGE MODEL

Using an independent Gemini-3.5-flash judge on a 10-task subset produces the same qualitative result: CLAIMWRITER variants outperform baselines across audit metrics.

  • Independent judge: CLAIMWRITER systems significantly improve over baselines in every CLAIMPROBE metric under the Gemini-3.5-flash judge.The comparison uses 10 tasks and holds writers, sources, and extraction steps fixed.
  • AI-Q example: On AI-Q, CLAIMWRITER-BU drops HALL from 57.14 to 19.81 and MIS from 15.89 to 0.86, while raising REC from 15.97 to 41.22.These values are percentages from the independent-judge audit.

D PROBE UNDER PERTURBATION

Under controlled source perturbations, CLAIMPROBE evaluates whether updates preserve report quality and whether systems propagate changed evidence without introducing drift. The audit uses claim-level alignment, relevance labels, and update-specific fact changes.

  • Perturbation results: EDR update methods remain within ±1 percentage point of the unperturbed baseline on every CLAIMPROBE metric.The result indicates that the surgical updates do not introduce measurable audit drift in the tested perturbations.
  • Perturbation results: AI-Q systems show HALLstr ≈66−68, MIS ≈44−45, and REC ≈31−38 across configurations, independently of update strategy.These systems hallucinate and misattribute more while recalling fewer HIGH facts in every tested configuration.
  • Alignment procedure: CLAIMPROBE compares report claims with retrieved evidence using strict report-to-source judgments and more permissive source-to-report coverage judgments.Numeric mismatches are penalized in the report-to-source direction, while paraphrase, summarization, and subsumption count as source-to-report coverage.
  • Update mechanism: The incremental-update path returns only changed facts as identifier-and-change pairs from updated source text and prior report-used facts.This representation supports localized revision of affected claims rather than rewriting the entire report.
Loading 2608.28643v1…