Source-linked AI summary
FrankenReport: Early Exiting in Long-Form Generation Using Expected Value of Computation
Zhengping Jiang, Gonzalo Ramos, Jina Suh, Shiqian Rachel Ng, Elias Stengel-Eskin, Justin Svegliato, Benjamin Van Durme, Andy Huntington, Sam Thomson
TL;DR
Deep-research systems generate useful long-form reports, but their iterative retrieval and reasoning pipelines incur substantial cost and latency, and the value of extra computation is difficult to predict. FRANKENREPORT uses intermediate section drafts to estimate the gains from further refinement and allocate computation adaptively. Experiments and user studies report better quality than random allocation under limited budgets and adaptation to diverse user preferences with modest feedback.
Problem
Deep-research reports rely on costly, latency-inducing retrieval and reasoning pipelines, while the subjective, multidimensional value of additional computation makes resource allocation difficult to predict in advance.
Source: §1 · pp. 1–2Method
FRANKENREPORT uses intermediate drafts and a learned predictor of future quality gains to adaptively allocate refinement across report sections.
Source: §1 · p. 2, §2.4 · p. 3Results
Takeaways & Limitations
The results show that early drafts provide useful signals for predicting quality gains and that user preferences can be learned online for adaptive computation allocation.
Source: §1 · p. 2, §5 · p. 9Evaluation focuses on English-language report benchmarks; the full orchestration study uses Llama-3.1-70B-Instruct, and user preferences are modeled as linear weights over five quality dimensions.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
While deep research systems address interactive information-seeking needs impressively, their real-world deployments face latency and resource-consumption challenges. We present FrankenReport, an interface for long-form knowledge-seeking report generation that supports adaptive early exiting per section: it evaluates intermediate outputs during generation and predicts whether further targeted computation will yield significant quality gains. In a simulation study, FrankenReport outperforms random allocation baselines by a large margin (up to 4x) under low budgets and smoothly recovers full-pipeline quality as the budget grows, showing that future quality gains are predictable from intermediate drafts. Through experiments and user studies, we further show that despite varying preferences across users and topics, FrankenReport adapts to simple, natural user feedback as efficiently as methods requiring much costlier supervision such as generated drafts and explicit rationales.
1 Introduction
Deep-research reports incur substantial computation, cost, and latency, while additional compute may yield diminishing or subjective quality gains. FRANKENREPORT uses intermediate drafts to predict the value of further refinement; experiments find these drafts informative and show adaptive allocation can improve quality under limited budgets.
Motivation
- Deep-research systems produce structured, cited reports through iterative retrieval and reasoning, but those pipelines incur substantial computational and monetary costs and user-facing latency.Examples named in the paper include OpenAI Deep Research, Gemini Deep Research, and DeerFlow.Source: §1 · p. 1
- More computational effort does not guarantee better results: it can add low-value text or factual errors as well as accurate facts.FRANKENREPORT predicts the gain from additional effort and spends it when estimated quality gain is high.Source: Table 1 · p. 1
- In report generation, unlike binary correctness, the marginal value of additional computation is subjective and multidimensional, complicating a priori resource allocation.Users may tolerate delays for noticeable, justified improvements, while additional computation can yield diminishing returns.Source: §1 · pp. 1–2
Approach
- FRANKENREPORT uses outlines, gists, and partial drafts as early signals of organization, coverage, and factual grounding to predict gains from further computation.A fine-tuned LLM with a regression head estimates potential quality improvements and supports adaptive early exiting; the final report combines sections refined to different depths.Source: §1 · p. 2
Results
- Intermediate sketches predict multidimensional quality gaps better than the query alone, and FRANKENREPORT improves quality over random exiting under limited budgets.A 100-participant study found diverse but learnable user preferences, and the system adapted with sample efficiency comparable to methods using full drafts or rationales.Source: §1 · p. 2
2 Related Work
Prior work studies early stopping in neural inference and reasoning, and routes queries or refinement effort to manage cost and quality. FrankenReport frames early exiting for long-form reports as section-level resource allocation across stages and tools.
2 Related Work
- Deep-research workflows target knowledge-seeking tasks such as business analysis, market research, and clinical summarization, where comprehensive reports and short turnaround times motivate compute-aware generation.The paper positions its setting as single-turn report generation.Source: §2 · p. 2
2.1 Early Exiting in Transformer Models
- Early-exit methods for Transformers halt inference using intermediate classifiers, building on evidence that shallow decoder-only LLM layers can already encode many correct predictions.DeeBERT, FastBERT, and PABEE report latency reductions with minimal accuracy loss [Xin et al., 2020; Liu et al., 2020; Zhou et al., 2020].Source: §2.1 · p. 2
2.2 Early Exiting from Reasoning Chains
- Reasoning-chain early stopping uses signals such as intermediate-answer convergence, answer entropy, token-level logits, or redundant reflection to decide when to stop.Other approaches use injected exit instructions or an external completion verifier.Source: §2.2 · p. 2
2.3 Semantic Routing for LLM Cascades
- LLM cascades balance cost and quality by routing queries to models or refinement steps, using predicted difficulty, semantic intent, disagreement, uncertainty, or speculative computation.FrugalGPT escalates to stronger models when needed, while other methods learn routing policies or selectively apply multi-agent refinement.Source: §2.3 · p. 2
2.4 Wikipedia-style Report Generation
- The paper builds on STORM, which gathers perspectives through parallel Questioner–Expert dialogue threads, aggregates retrieved passages into an outline, and expands it into section prose.The original pipeline’s Sketch and Verify checkpoints occur later than the corresponding gist-primed checkpoints shown in Figure 1.Source: §2.4 · pp. 2–3, Figure 1 · p. 3
- FRANKENREPORT formulates budgeted exiting across report sections, refinements, and tool use, and evaluates quality-cost trade-offs, especially under low budgets.Unlike per-token or per-layer stopping, its future-utility predictor estimates section-level gains to guide selective refinement; gist-primed checkpoints appear earlier than the original STORM checkpoints.Source: §2.4 · p. 3, Figure 1 · p. 3
3 FRANKENREPORT
FRANKENREPORT turns a deep-research pipeline into staged, section-level generation and uses predicted quality gains to decide where further computation is worthwhile. Its interface streams drafts as refinement continues, while its evaluation cost is a latency-calibrated proxy rather than a measure of exact resource savings.
3 FRANKENREPORT
- A trained action predictor monitors intermediate section drafts and estimates whether further computation is likely to improve perceived quality, enabling adaptive stopping and personalized latency-quality trade-offs.The orchestration policy selects configurations of the research pipeline, which determine the modules or reasoning steps applied to a query.Source: §3 · p. 3, Figure 2 · p. 4
- Experimental cost is a normalized, latency-calibrated stage cost: Sketch-to-Verify costs one unit, while Sketch-to-Expand costs roughly four, based on logged mean transition times.The simulator sums promotion units to impose a common allocation budget across sections.Source: §3 · p. 3
- Parallel and speculative execution can reduce perceived waiting without reducing aggregate model work; the reported curves therefore do not measure exact token, energy, or GPU-hour savings.Only refinements never launched or terminated early reduce both latency and actual compute.Source: §3 · p. 3
3.1 Step-wise Report Generation
- Gist-priming retrieves documents from the prompt, creates inexpensive section gists, and allocates evidence-gathering threads across them before staged generation.Sketch drafts from model knowledge conditioned on the gists; Verify adds one retrieval round guided by the sketch and gist; Expand runs the full STORM workflow with multiround retrieval.Source: §3.1 · p. 4
3.2 Score-based Orchestration
- The score-based orchestrator estimates the conditional expected quality gain from advancing a section between stages and weighs it against transition cost and the overall budget.A regression-head LLM conditions its estimate on the query, section gist, and current draft; the draft is available because later stages build on earlier ones.Source: §3.2 · pp. 4–5
3.3 User Interface Design
- The interface streams available drafts while deeper reasoning continues, lets users select a draft for the report, and can skip later stages when predicted gains do not justify further computation.When further compute is warranted, the interface can prefetch later stages; Verify and Expand drafts are usually paired with citations.Source: §3.3 · p. 5
4 Experiment Results
The experiments test whether extra computation improves report quality, whether its gains can be predicted early, and how allocation can adapt to user preferences. The learned allocator achieves up to 4× the gains of random allocation at low budgets, while prediction and user-study results support adaptive section-level refinement.
4.1 Datasets
- Five datasets contribute 1,000 prompts and 17,106 section examples: 100 FreshWiki topics plus 900 prompts from four auxiliary pools.The collection emphasizes newer topics lacking complete Wikipedia coverage; the auxiliary samples include 100 LongFact, 100 FActScore, 200 CORE, and 500 WildHallucinations topics.Source: §4.1 · p. 5
4.2 Quality Assessment
- Later drafts tend to score higher across five textual-quality metrics, but gains vary by metric and are modest for Coherence and Engagingness.Verify can disrupt fluent style with loosely relevant evidence even while improving grounding.Source: §4.2 · p. 6, Table 3 · p. 6
4.3 Predicting quality differences
- Organization and Coherence are the most predictable dimensions, while query-only Factuality prediction is especially weak because supporting evidence is not yet available.Predicting Expand’s total gain from Sketch differs from predicting its residual gain after Verify, which is smaller, rarer, and noisier.Source: §4.3 · pp. 6–7
- Up to 4× the gains of random allocation are reached in the low-budget regime; for Coherence, 10% of the budget recovers over 90% of full-STORM performance.For Informativity, gains rise smoothly as more compute is added.Source: Figure 3 · p. 7, §4.3 · p. 7
4.4 Online Learning from User Feedback
- Simulated preference accuracy reaches a high level within 10 examples, and information weighting encourages exploration of drafts hidden by an earlier predicted exit.A user sees only drafts up to the predicted exit point, so an early prediction can obscure later alternatives.Source: §4.4 · pp. 7–8
4.5 User Study
- Later drafts are preferred more often overall, but Sketch receives the maximum rating for a nontrivial subset of sections.Automated metrics align meaningfully but incompletely with human ratings; replayed online personalization improves selected ratings by .042 on the 1–5 scale over uniform weighting.Source: §4.5 · p. 8
5 Conclusion
FRANKENREPORT exposes intermediate drafts to reduce compute and latency by estimating quality from early signals; its orchestrator learns user preferences online with modest budgets.
5 Conclusion
- FRANKENREPORT exposes intermediate drafts so a deep research pipeline can estimate final-draft quality from early signals and reduce compute and latency.The conclusion describes an extension to variable-depth refinement under the same expected-gain accounting.Source: §5 · p. 9
- Under fixed costs, FRANKENREPORT consistently outperforms random expansion baselines.The orchestrator learns user preferences online with modest budgets, although preferences vary across topics and generally favor later drafts.Source: §5 · p. 9
- Future work includes topic-conditioned features and extending the Sketch/Verify/Expand setup to variable-depth refinement.The extension would retain the same expected-gain accounting.Source: §5 · p. 9
Limitations
The evaluation is limited to English-language benchmarks, model- and metric-dependent quality gains, a linear preference model, and automatable metrics.
Limitations
- Experiments use only English-language report-generation benchmarks, so generalization to other languages is not guaranteed.The limitation is a boundary on the evaluated language scope, not evidence that the approach fails in other languages.Source: Limitations · p. 9
- The magnitude and shape of staged quality gains depend on the base model's capability and metric ceilings.The GPT-4.1 study covers only non-search-backed metrics, while the full orchestration study uses Llama-3.1-70B-Instruct.Source: Limitations · p. 9
- Individual user preferences are modeled as linear weights over five quality dimensions, although real preferences may be nonlinear and context-dependent.The authors identify feature-interaction or contextual-bandit preference models as possible extensions.Source: Limitations · p. 9
- Evaluation is restricted to reliably automatable metrics, which may not capture real-world user preferences as well as heterogeneous or human-centric metrics.The authors identify incorporating such metrics as future work.Source: Limitations · p. 9
A Interface Design Details
The appendix maps the interface's main panels and controls to the paper's Sketch, Verify, and Expand stages, then describes evaluation prompts for quality and preference tasks.
A Interface Design Details
- Figure 6 labels the gist, initial draft, grounding control, and elaboration control; the initial draft uses a retrieval-informed gist but omits section-level evidence retrieval.Initial Draft/First Draft correspond to Sketch, Grounding to Verify, and Elaborating/Elaborated to Expand; Figure 7 shows the override menu for inspecting alternatives and giving corrective feedback.Source: A Interface Design Details · p. 12
B Evaluation Prompts
- The evaluation prompts cover coherence, organization, engagingness, preference summarization, and preference-conditioned draft selection.The corresponding prompt figures are Figures 8–12.Source: B Evaluation Prompts · p. 12
C Generalization to a Second Chat Model
GPT-4.1 retains metric-dependent gains across later passes, with informativity benefiting from deeper exploration; the appendix also describes budget-allocation baselines and evaluation scope.
C Generalization to a Second Chat Model
- Figure 6 shows the interface's gist, initial draft, grounding, and elaboration controls; the initial draft omits section-level evidence retrieval.The interface terminology maps Initial Draft/First Draft to Sketch, Grounding to Verify, and Elaborating/Elaborated to Expand.Source: Figure 6 · p. 13
- The GPT-4.1 experiments pair generation with non-search-backed metrics, excluding Factuality, which depends on retrieval and evidence integration.The experiments complement the Llama-3.1-70B-Instruct orchestration experiments, which use search-backed evaluation across thousands of topics.Source: C Generalization to a Second Chat Model · pp. 12–13, Table 6 · p. 13
- Informativity rises from 0.831 to 1.020 with GPT-4.1, while later-pass gains are not uniform across metrics.Factuality is omitted from the GPT-4.1 results because it requires search-backed evaluation.Source: C Generalization to a Second Chat Model · p. 13, Table 6 · p. 13
- GPT-4.1 starts with Coherence at 4.93 versus 4.39 for Llama, near the 1–5 scale ceiling; Verify scores 4.982 versus 4.739 for Expand.With GPT-4.1, Informativity gains remain substantial, while some style-oriented metrics can score lower after further refinement.Source: C Generalization to a Second Chat Model · p. 13
- With a stronger base model, the orchestrator can skip refinement on dimensions near saturation and focus compute on dimensions with more room to improve.Future-utility estimation remains useful for directing refinement toward dimensions with available headroom.Source: C Generalization to a Second Chat Model · p. 13
D Budget Allocation
- Figure 13 reports budget-allocation results for Factuality and Engagingness.The supplied description identifies the evaluated metrics but does not state their comparative outcomes.Source: D Budget Allocation · p. 14
- Random-uniform samples Sketch, Verify, or Expand without section features, while random-greedy assigns the highest-effort feasible action in random section order.Random-greedy prioritizes Expand over Verify over Sketch, based on average quality gains at later stages.Source: D Budget Allocation · p. 14
- Shortest-first promotes the currently shortest draft without learned quality predictions; GPT-5-mini scoring instead promotes the lowest-scoring feasible draft.The GPT-5-mini router scores the visible draft but does not predict the marginal gain of the next stage.Source: D Budget Allocation · p. 14
E Additional Details on Quality Assessment
The appendix reports stable factuality scores under claim subsampling and describes preference adaptation that trades some local optimality for faster early learning. Larger information weights speed adaptation but have little effect on final accuracy.
E Additional Details on Quality Assessment
- Factuality scores remain stable when claims are subsampled, motivating evaluation on a subset of claims.Figure 14 presents factuality scoring results on the left and preference adaptation under different α values on the right.Source: E Additional Details on Quality… · p. 14
F User Preference Adaptation
- An information-gain term biases early action selection toward Expand, collecting more informative user feedback at the cost of locally suboptimal predictions.The policy selects actions using the estimated preference vector and expected information gain.Source: F User Preference Adaptation · p. 14
- Larger α values lead to faster preference adaptation but have little effect on final accuracy.The authors attribute this to the default policy already exploring sufficiently for efficient adaptation.Source: F User Preference Adaptation · p. 14
G.1 Annotation
The annotation study collected human quality ratings of reports at different early-exit points from 100 participants. Participants evaluated multiple drafts for each section and reported their familiarity with each topic and confidence in their ratings.
G.1 Annotation
- 100 participants evaluated generated report content at different early-exit points, with five independent evaluations per test prompt.The study used the same 100 test prompts and generations from the full dataset.Source: §G.1 · p. 14
- Participants were recruited through Prolific and screened for English use, U.S. location, prior AI-evaluation experience, approval rate, and education.Eligibility required at least 20 prior AI-evaluation tasks, 100% approval, and an undergraduate degree or higher.Source: §G.1 · p. 14
- Participants rated Sketch, Verify, and Expand drafts for each section, then reported topic familiarity and confidence in their assessments.Draft options were shuffled; participants evaluated five topics per task and copied their ratings into the survey.Source: §G.1 · p. 15
- The survey took approximately 90 minutes, and each participant who completed it was compensated USD $19.The survey also collected demographic information after participants evaluated the five topic reports.Source: §G.1 · p. 15
G.2 Inter-Annotator Agreement Analysis
Agreement among section-draft ratings was mild, with only modest increases at higher reported familiarity or confidence.
G.2 Inter-Annotator Agreement Analysis
- Krippendorff’s α was 0.43, with slightly higher agreement at greater reported topic familiarity or confidence.The increase with familiarity or confidence was modest.Source: §G.2 · p. 15
G.3 Oracle Preference Modeling
The oracle preference analysis uses user feedback to construct preference profiles and predict preferred drafts, comparing its results with an online orchestrator. The reported oracle results exceed a majority-vote baseline, while the online orchestrator approaches oracle accuracy after several interaction rounds.
G.3 Oracle Preference Modeling
- The oracle profile uses up to 10 annotated reports, holding one out and summarizing the user's rankings and explanations to predict preferred drafts.The preference summary is generated from the remaining reports and applied to each section of the held-out report.Source: §G.3 · p. 15
- The oracle sees all three drafts and explicit rationales, unlike the online orchestrator's sequential drafts and implicit single-session feedback, so it upper-bounds deployed prediction accuracy.The oracle procedure is described as having access to more information than the online preference model.Source: §G.3 · p. 15
- GPT-5.1 reaches .676 draft-selection accuracy and .820 “Needs Improvement” accuracy, versus .406 and .716 for majority vote; adding familiarity yields .640 and .815, respectively.GPT-5-mini scores .649 draft-selection accuracy and .811 “Needs Improvement” accuracy; online orchestration approaches oracle accuracy within roughly five interaction rounds.Source: §G.3 · p. 15
- The draft-selection dropdown lets users inspect alternative generations before confirming an exit decision.Source: Figure 7 · p. 16