Source-linked AI summary
LLM$\times$MapReduce-V2: Entropy-Driven Convolutional Test-Time Scaling for Generating Long-Form Articles from Extremely Long Resources
Haoyu Wang, Yujia Fu, Zhu Zhang, Shuo Wang, Zirui Ren, Xiaorong Wang, Zhili Li, Chaoqun He, Bo An, Zhiyuan Liu, Maosong Sun
TL;DR
Long-to-long generation remains challenging because LLMs must integrate and analyze information from extensive resources. LLM×MapReduce-V2 uses entropy-driven convolutional test-time scaling and integrative representations; on SurveyEval, it improves reference utilization and outperforms representative baselines.
Problem
Long-to-long generation remains relatively underexplored, and effectively integrating and analyzing relevant information from extensive inputs remains difficult for LLMs.
Source: §1 · pp. 1–2Method
Results
At least 32.9% improvement in the reference utilization rate, with better performance than the extractive baseline in other dimensions; the method also consistently outperforms representative baselines on SurveyEval.
Source: §1 · p. 2Takeaways & Limitations
The information-bottleneck analysis indicates that more informative intermediate elements can theoretically improve the lower bound on information in the final output.
Source: §1 · p. 2LLM×MapReduce-V2 has only been verified on survey writing, and the authors identify extension to other tasks as future work.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Long-form generation is crucial for a wide range of practical applications, typically categorized into short-to-long and long-to-long generation. While short-to-long generations have received considerable attention, generating long texts from extremely long resources remains relatively underexplored. The primary challenge in long-to-long generation lies in effectively integrating and analyzing relevant information from extensive inputs, which remains difficult for current large language models (LLMs). In this paper, we propose LLM$\times$MapReduce-V2, a novel test-time scaling strategy designed to enhance the ability of LLMs to process extremely long inputs. Drawing inspiration from convolutional neural networks, which iteratively integrate local features into higher-level global representations, LLM$\times$MapReduce-V2 utilizes stacked convolutional scaling layers to progressively expand the understanding of input materials. Both quantitative and qualitative experimental results demonstrate that our approach substantially enhances the ability of LLMs to process long inputs and generate coherent, informative long-form articles, outperforming several representative baselines. Both LLM$\times$MapReduce-V2 and SurveyEval are publicly available at https://github.com/thunlp/LLMxMapReduce .
1 Introduction
Long-to-long generation requires integrating extensive resources, but current models struggle to synthesize their key insights, and query-based extraction can omit relevant information outside the query. The paper proposes an integrative framework and SurveyEval to address and evaluate resource utilization in this setting.
1 Introduction
- Long-to-long generation must integrate collected materials into informative, cohesive results, but extensive resources challenge LLMs and human experts alike.The task includes both resource collection and resource utilization; this work focuses on utilization.Source: §1 · pp. 1–2
- Query-based extraction may omit relevant critical analysis, nuanced insights, or broader context that does not directly align with the query.The paper motivates integrative approaches as a way to synthesize broader information and draw connections across content.Source: §1 · p. 2
- An information-bottleneck analysis identifies informative intermediate representations as central, introducing a skeleton and resource digests to improve the theoretical lower bound on information in the output.The skeleton is aligned with the output, while digests summarize the resources.Source: §1 · p. 2
- LLM×MapReduce-V2 uses randomized convolutional test-time scaling guided by entropy estimation to progressively aggregate information and enhance result informativeness.The approach draws inspiration from convolutional networks that abstract local features into higher-level global representations.Source: §1 · p. 2
- SurveyEval pairs academic surveys with their complete reference papers to evaluate resource utilization in survey writing.The paper describes it as the first benchmark in computer science to pair surveys with complete reference papers.Source: §1 · p. 2
- At least 32.9% improvement in the reference utilization rate; LLM×MapReduce-V2 also performs better than the extractive baseline in other dimensions.On SurveyEval, the method consistently outperforms several representative baselines.Source: §1 · p. 2
2 Information Bottleneck Analysis
The analysis frames long-to-long generation as an information-bottleneck problem: resources must be compressed for the context window, while intermediate representations support construction of the final output.
2 Information Bottleneck Analysis
- The information bottleneck frames long-to-long generation as compressing lengthy resources to fit LLM context windows while relying on intermediate representations to construct the output.In the notation, X is the input source, Z the intermediate representation, and Y the output.Source: §2 · p. 2
- The paper defines a skeleton S aligned with the output and digests D as compressed summaries derived from the resources R.The input materials X include the required article topic and the provided resources.Source: §2 · p. 3
- The bound yields objectives to increase digest information, increase skeleton information, reduce digest information unused in the output, and incorporate information beyond the digest when writing.The paper focuses on the first three objectives and leaves the fourth for future work.Source: §2 · p. 3
- Optimization of the objective for incorporating additional information beyond the digest when writing the survey is left for future work.The paper focuses on the other three objectives to improve the information-bottleneck lower bound.Source: §2 · p. 3
3 LLM×MapReduce-V2
The method treats the skeleton as a bridge between input resources and the final survey: it guides digest extraction and organizes digests for writing. LLM×MapReduce-V2 combines skeleton-guided digests, entropy-driven convolution and Best-of-N refinement, then generates content with the survey tree structure.
3.1 Initialization
- The method parses the skeleton and paper digests into a tree mirroring the markdown document, with each skeleton node specifying digest construction and digest analysis.Digest Construction guides digest creation; Digest Analysis specifies how digests are used in writing.Source: §3.1 · p. 3
3.2 Skeleton Improvement
- Skeleton improvement updates the skeleton with digest feedback rather than replacing it, ranking candidates by estimated entropy split into title-structure and chapter-description components.An LLM-as-judge estimates the information entropy.Source: §3.2 · p. 4
3.2.1 Entropy-Driven Convolution
- Cluster-level partial feedback enters randomized multi-layer convolution: entropy-driven sampling forms feedback sets, which are integrated in parallel and top-k propagated before selecting the best final-layer skeleton.Source: §3.2.1 · pp. 4–5
3.2.2 Best-of-N Self-Refinement
- Best-of-N independently generates candidate feedback from Srefine and selects the highest-entropy candidate for repeated global skeleton adjustments.The selected skeleton is intended to exhibit global information integration beyond the references.Source: §3.2.2 · p. 5
3.3 Topology-Aware Content Generation
- Using the optimized skeleton and digests, generation proceeds node by node with node-specific analysis and reference digests; leaves emphasize details and comparisons, while parent sections incorporate subsection content.Source: §3.3 · p. 5
4 Experiment
On SurveyEval, LLM×MapReduce-V2 outperforms baselines in most reported dimensions. The experiments compare survey-writing systems, evaluate automatic and human judgments, and analyze component settings.
4.1 Dataset
- SurveyEval pairs surveys with complete reference papers; the collection contains 384 surveys citing over 26,000 references, and 20 articles form the test set.The test-set size was chosen to align with AutoSurvey and limit the time and labor of evaluation.Source: §4.1 · p. 5
4.2 Baselines
- The comparison includes direct full-input decoding, skeleton-first decoding, and AutoSurvey, with each baseline given the test-set titles and full reference papers.All baselines use Gemini-2.0-flashthinking-exp-1219.Source: §4.2 · pp. 5–6
4.3.1 Automatic Metrics
- Automatic evaluation covers reference use, structure, content quality, and claim richness or density; human assessors compare surveys on the same topic and yield a win rate.Figure 4 reports the human-evaluated win rate against AutoSurvey; the supplied passages do not state its value.Source: §4.3.1 · p. 6, §4.3.2 · p. 7, Figure 4 · p. 7
4.4 Main Results
- Across four SurveyEval dimensions, LLM×MapReduce-V2 outperforms baselines in most dimensions, scoring 95.00 on structural metrics and 97.22 on faithfulness, with reference precision 95.50 and recall 95.80.The reported precision and recall are the highest among the baselines.Source: Table 1 · p. 7, §4.4 · p. 7
4.3.2 Human Evaluation
- In a human evaluation, assessors judge which survey performs better on the same topic; win rate is computed from these comparisons, and Figure 4 reports LLM×MapReduce-V2 versus AutoSurvey on the test set.Source: §4.3.2 · p. 7, Figure 4 · p. 7
4.4 Main Results
- LLM×MapReduce-V2 produces the largest number of informative claims and higher claim density than AutoSurvey, while standard decoding yields fewer unique claims despite high density.The authors associate these comparisons with the entropy-driven test-time scaling mechanism and integrative approach.Source: §4.4 · p. 7
4.5 Analysis of the Components
- With convolution settings fixed, skeleton entropy peaks at seven layers and kernel width three; three self-refinement iterations also peak, while excessive depth, width, or refinement can hurt.The component analysis uses normalized information entropy as a proxy for skeleton informativeness.Source: §4.5 · p. 8, Figure 5 · p. 8
5 Related Work
Prior long-to-long generation work often uses retrieval and extraction, while survey-writing systems also use retrieval, clustering, or skeleton construction. Integrating extensive information remains a central consideration.
5 Related Work
- STORM and Co-STORM use multi-agent question generation to retrieve documents, while OmniThink and IRP extract relevant paragraphs for article writing.These are presented as extractive approaches to long-to-long generation.Source: §5 · p. 8
- The cited end-to-end generation work still struggles to achieve satisfactory results given model capability limitations.Source: §5 · p. 8
- Survey-writing systems include AutoSurvey's retrieval and iterative refinement, HiReview's title clustering and skeleton generation, and PaSa's paper-search agent; integrating vast information is increasingly crucial.Source: §5 · p. 8
6 Conclusion
The paper presents an entropy-driven convolutional test-time scaling framework and evaluates it with SurveyEval. It reports superiority over existing baselines for generating comprehensive surveys.
6 Conclusion
- The proposed framework uses entropy-driven convolutional test-time scaling to process and synthesize extremely long input materials.Source: §6 · p. 9
- SurveyEval is introduced as a benchmark for evaluating the method's effectiveness.Source: §6 · p. 9
- The paper reports that its method is superior to existing baselines in generating comprehensive surveys.Source: §6 · p. 9
Limitations
The stated limitation is that LLM×MapReduce-V2 has so far been verified only on survey writing. The accompanying information-bottleneck material defines the task representations and states assumptions relating them.
Limitations
- LLM×MapReduce-V2 has only been verified on the survey task, so its reported scope does not yet include other practical tasks such as research reports.Source: Limitations · p. 9
- The experiments mainly use Gemini-flash-thinking, and the authors plan future validation on newer, more powerful models; base-model hallucinations may introduce errors or misleading information.Source: Limitations · p. 9
A Information Bottleneck in Survey Generation
- The information-bottleneck setup defines resources R and topic T as inputs, with digests D and skeleton S as intermediate representations; it assumes R contains all information in D and S.It also assumes D and S contain all information in T, and survey Y contains all information in S.Source: A Information Bottleneck in Survey… · p. 11
B.1 Dataset Construction
Existing survey datasets often provide only abstracts or outlines, while full-text datasets are scarce or narrowly scoped. SurveyEval addresses this gap with literature surveys paired with full references.
B.1 Dataset Construction
- Many available survey datasets contain only reference abstracts or outlines, and existing full-text datasets are limited in scope or size.SurveySum, for example, contains six literature surveys.Source: §B.1 · p. 11
- SurveyEval combines comprehensive literature reviews with full references and is designed for long-to-long generation.Its dataset comparison covers components, input lengths, and dataset sizes.Source: §B.1 · p. 12, Table 2 · p. 12
- The dataset was assembled from arXiv cs.CL surveys, filtered for suitability, and quality-checked through automated screening and manual verification.Reference searches used sources including ACL, NeurIPS, CVPR, and Google Scholar; MinerU converted PDFs to structured Markdown.Source: §B.1 · p. 12
B.2 Test Dataset
The test set contains 20 SurveyEval papers selected to reflect reference-retrieval completeness and varied input sizes. The vanilla baseline fits full reference content to the model window through proportional cropping.
B.2 Test Dataset
- Twenty SurveyEval papers were selected using reference-retrieval completeness and reference-list token-count diversity as criteria.These criteria were intended to make the test set representative of real-world scenarios.Source: §B.2 · p. 12
C.1 Implementation of Vanilla
- The vanilla baseline supplies the topic and full reference contents, proportionally cropping papers to fit the model’s context window.The crop allocated to each paper is proportional to its length relative to the total reference length.Source: §C.1 · p. 12
C.2 Implementation of Vanilla with skeleton
The section describes a skeleton-based baseline, the AutoSurvey implementation, and LLM×MapReduce-V2 settings, followed by SurveyEval’s evaluation framework and quality criteria.
C.2 Implementation of Vanilla with skeleton
- The skeleton baseline first creates an outline from the topic and reference abstracts, then combines it with reference full texts to generate the survey.Source: §C.2 · p. 12
C.3 Implementation of AutoSurvey
- AutoSurvey was adapted to SurveyEval while retaining its original parameters, with retrieval configured for 1,200 papers and outline generation based on abstracts.Subsections were fixed at eight; generation used the first 1,500 body tokens from 60 retrieved papers, held constant through reflection and polishing.Source: §C.3 · p. 13
C.4 Implementation of LLM×MapReduce-V2
- LLM×MapReduce-V2 uses six convolution layers, counting Digest-Based Feedback Clustering as one layer.Source: §C.4 · p. 13
D Details of SurveyEval Evaluation
- SurveyEval combines automatic and human evaluation on a 100-point scale, adapting AutoSurvey metrics and adding criteria for language and criticalness.Language assesses clarity, formality, and redundancy; criticalness assesses analytical depth, originality, and future research directions.Source: D Details of SurveyEval Evaluation · p. 13, §D.1.2 · pp. 14–15
D.1.1 Structure Quality criteria
- Structure quality follows AutoSurvey criteria, with raw scores multiplied by 20 to convert the original 0–5 scale to 100 points.Source: §D.1.1 · p. 14
D.1.2 Content Quality criteria
- Faithfulness checks citation accuracy and relevance against cited-paper full texts, while relevance retains AutoSurvey criteria and is rescaled from 0–5 to 100.The evaluation separates claim and source checks for citations and assesses multiple citations individually.Source: §D.1.2 · p. 14
D.1.3 Claim Evaluation Details
The evaluation measures claim quantity and redundancy, reference utilization, and human preferences through randomized survey comparisons.
D.1.3 Claim Evaluation Details
- Claims are extracted separately by survey section and deduplicated within 300-claim batches and across groups; density is unique claims divided by pre-deduplication claims.Source: §D.1.3 · p. 15
D.1.4 Reference Evaluation Details
- Reference precision checks whether each input reference supports at least one cited sentence, while recall measures how many input references appear in the generated survey.Source: §D.1.4 · p. 16
D.2 Human Evaluation Details
- Human evaluators selected their preferred survey (“Document 0” or “Document 1”) or marked “Tie”; evaluation randomized topics, and 17 university volunteers provided 217 valid data points across 20 test-set topics used in automatic evaluation.Source: §D.2 · p. 16