Source-linked AI summary
TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review
Yuan Chang, Ziyue Li, Hengyuan Zhang, Yuanbo Kong, Yanru Wu, Hayden Kwok-Hay So, Zhijiang Guo, Liya Zhu, Ngai Wong
TL;DR
LLM-based peer-review methods can struggle to provide thorough, insightful feedback efficiently. TreeReview builds and resolves a hierarchical question tree, adding follow-up questions when needed. On ICLR and NeurIPS benchmarks, it outperforms strong baselines on review quality while reducing token use.
Problem
Current LLM-based review methods can miss details dispersed across long papers, produce superficial feedback, and incur substantial coordination costs.
Source: §1 · p. 1Method
TreeReview recursively decomposes review questions into fine-grained sub-questions, aggregates answers from leaves to root, and dynamically adds follow-up questions for deeper probing.
Source: §1 · p. 2Results
TreeReview improves full-review quality, including specificity by 12.27% over the best baseline, and beats MARG by 5.7% in Jaccard while reducing token usage by 80.2%.
Source: §1 · p. 2Takeaways & Limitations
The ICLR and NeurIPS benchmark evaluates both full-review generation and actionable-feedback-comment generation.
Source: §1 · p. 2TreeReview does not incorporate figures as model input, so it cannot capture feedback about visual elements.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
While Large Language Models (LLMs) have shown significant potential in assisting peer review, current methods often struggle to generate thorough and insightful reviews while maintaining efficiency. In this paper, we propose TreeReview, a novel framework that models paper review as a hierarchical and bidirectional question-answering process. TreeReview first constructs a tree of review questions by recursively decomposing high-level questions into fine-grained sub-questions and then resolves the question tree by iteratively aggregating answers from leaf to root to get the final review. Crucially, we incorporate a dynamic question expansion mechanism to enable deeper probing by generating follow-up questions when needed. We construct a benchmark derived from ICLR and NeurIPS venues to evaluate our method on full review generation and actionable feedback comments generation tasks. Experimental results of both LLM-based and human evaluation show that TreeReview outperforms strong baselines in providing comprehensive, in-depth, and expert-aligned review feedback, while reducing LLM token usage by up to 80% compared to computationally intensive approaches. Our code and benchmark dataset are available at https://github.com/YuanChang98/tree-review.
1 Introduction
Current LLM-based review methods can miss details in long papers, produce superficial feedback, and incur high coordination costs. TreeReview decomposes review into focused questions and aggregates answers, and evaluations report stronger feedback quality with lower token use.
1 Introduction
- Long papers’ dispersed technical details challenge LLMs’ long-range reasoning, while existing methods also risk shallow feedback and costly, error-prone multi-agent coordination.These issues can leave reviews incomplete or short of the depth needed to evaluate technical nuances.Source: §1 · pp. 1–2
- TreeReview recursively decomposes broad review questions into focused sub-questions, answers them, and aggregates answers from leaves to root, adding follow-up questions when deeper probing is needed.The framework uses focused paper chunks and structured decomposition and aggregation to address these challenges while minimizing token usage.Source: §1 · p. 2
- Specificity rises 12.27%, comprehensiveness 11.22%, and technical depth 6.45% over the best baseline; TreeReview also beats MARG by 5.7% in Jaccard while reducing token usage by 80.2%.Expert evaluators also preferred TreeReview reviews over baseline methods with high consistency.Source: §1 · p. 2
- The benchmark covers full-review generation and actionable-feedback-comment generation using papers and human reviews from ICLR and NeurIPS.For full reviews, the reported gains span specificity, comprehensiveness, and technical depth.Source: §1 · p. 2
2 Related Work
Prior automated peer-review research spans specialized tools, review-generation models, and more complex frameworks; separate work studies decomposition as a strategy for complex reasoning. TreeReview applies dynamic hierarchical decomposition to review tasks.
2.1 Automated Scientific Peer Review
- Earlier automated-review work addressed reviewer assignment, plagiarism detection, and rating recommendation, while small fine-tuned models have faced limits in producing nuanced reviews of full papers.Recent LLM review studies can generate meaningful feedback but often lack the critical analysis, depth, and specificity of human reviews.Source: §2.1 · pp. 2–3
- Research to improve review quality beyond simple prompting follows two directions: dataset curation and fine-tuning for review generation, and richer frameworks using multi-agent systems, multimodal processing, or external knowledge; review generation also serves as feedback in scientific-discovery workflows.Source: §2.1 · p. 3
2.2 Decomposition of Complex Tasks
- Task decomposition breaks challenging reasoning into manageable subproblems, from implicit Chain-of-Thought steps to explicit sequential or iterative solutions and tree-based reasoning.The cited tree-based approaches model hierarchical dependencies and aggregate results.Source: §2.2 · p. 3
- TreeReview applies dynamic hierarchical decomposition so different aspects of scientific papers can be assessed in a focused manner.The paper frames peer review as a complex cognitive task requiring comprehensive evaluation across multiple aspects.Source: §2.2 · p. 3
3 Method
TreeReview builds a question tree through top-down decomposition, then resolves it bottom-up to produce a review. Its answer synthesizer can add follow-up questions when current evidence is insufficient, and final root-level synthesis uses the full paper.
3.1 Overview
- The framework uses question generation to decompose high-level review questions and answer synthesis to aggregate answers up the tree, with follow-up questions available for deeper probing.This bidirectional structure supports focused local analysis and systematic aggregation into global assessments.Source: §3.1 · p. 3
3.2 Top-down Question Generation Stage
- The question generator adaptively decomposes non-leaf questions into sub-questions, using paper metadata rather than the full paper to encourage exploratory questions.Decomposition is depth-aware and aims for mutually exclusive, collectively exhaustive sub-questions; sufficiently specific questions become leaves.Source: §3.2 · pp. 3–4
3.3 Bottom-up Answer Aggregation Stage
- Leaf questions are answered using the top-k relevant paper chunks rather than the full text, with answers grounded in explicit evidence from those chunks.The paper is segmented into chunks, and question-aware filtering selects relevant content for each leaf question.Source: §3.3 · pp. 4–5
- For intermediate questions, the synthesizer checks whether existing sub-question answers suffice and, if not, adds follow-up questions to probe unaddressed aspects before synthesis.New questions enter the tree and may be further decomposed; synthesis is deferred until their answers are obtained.Source: §3.3 · p. 5
- At the root, TreeReview generates the final review using the full paper alongside its collected sub-question answers, which serve as explicit reasoning traces.Intermediate aggregation relies on sub-answers, whereas final synthesis adds the full paper for holistic context.Source: §3.3 · p. 5
4 Experiments
TreeReview’s experiments evaluate full reviews and actionable comments, finding stronger review quality and expert preference alongside lower token use. Results also show gains on a multi-domain sample and identify contributions from question decomposition and dynamic expansion.
4.1 Experimental Settings
- The benchmark includes 40 ICLR-2024 and 40 NeurIPS-2023 papers with human reviews, balanced by acceptance status, and evaluates full reviews and actionable feedback comments.Full reviews include textual assessments and ratings; comments target substantive weaknesses and improvement areas.Source: §4.1 · pp. 5–6
4.2 Full Review Generation Task.
- TreeReview scored 8.18 overall and exceeded the best baseline by 12.27% in specificity, 11.22% in comprehensiveness, and 6.45% in technical depth.The authors attribute these gains to focusing attention on paper details while systematically aggregating answers.Source: Figure 3 · p. 6, §4.2 · p. 6
- TreeReview and SEA-E had the lowest MAE and MSE across rating dimensions; TreeReview’s Overall Rating MSE was 2.12 versus SEA-E’s 2.30.The evaluation compares system ratings with average ground-truth ratings; Table 1 reports the rating analysis.Source: §4.2 · pp. 6–7, Table 1 · p. 7
4.3 Feedback Comments Generation Task
- TreeReview had the highest LLM-based alignment precision (32.10%), exceeded MARG by 5.7% in pseudo-jaccard, and led semantic alignment with 47.99% SN-Precision.Its ITF-IDF score was 4.62, second only to MARG without refinement (5.37%), which sacrificed alignment for diversity.Source: §4.3 · p. 7
4.4 Human Evaluation
- Human evaluators preferred TreeReview across both tasks: full-review win-rates ranged from 66.25% to 90.00%, and it beat MARG 57.50% to 42.50% on comments.LLM overall-quality scores correlated with aggregated human win-rates (ρ ≈ 0.90, p < 0.05).Source: Figure 5 · p. 8, §4.4 · p. 8
4.5 Domain Generalization Analysis
- On papers from biology, climate science, and quantum science, TreeReview scored 8.31 overall for full reviews and led feedback generation in precision (34.62%) and Jaccard similarity (17.68%).The evaluation used 40 Nature Communications papers; performance broadly followed the earlier results across almost all metrics.Source: §4.5 · p. 8
4.6 Ablation Study
- Removing decomposition degraded quantitative accuracy and review quality; removing dynamic expansion lowered comment Jaccard from 15.33% to 12.98% and ITF-IDF from 4.62 to 4.01.Expansion triggered for 38.54% of non-leaf questions on average, adding 25.6 questions per review; rating prediction remained relatively robust without it.Source: §4.6 · p. 9
4.7 Cost Analysis
- TreeReview used 80.2% fewer tokens per paper than MARG and 54.4% fewer than MARG-BASE while retaining superior or competitive evaluation performance.The authors report shorter processing times and lower API costs.Source: §4.7 · p. 9, Table 5 · p. 9
Conclusion
TreeReview uses a dynamic, hierarchical question-answering architecture to generate in-depth review feedback efficiently. Benchmark experiments show superiority over baselines, and ablations highlight the importance of hierarchical decomposition and dynamic expansion.
Conclusion
- TreeReview’s benchmark experiments show superior in-depth and helpful review feedback compared with baselines while maintaining efficiency.The paper also reports that ablations highlight the importance of hierarchical decomposition and dynamic question expansion.Source: Conclusion · p. 9
- TreeReview structures paper review as a dynamic, hierarchical question-answering architecture.The framework is presented as addressing key challenges in LLM-based paper review.Source: Conclusion · p. 9
- The authors suggest TreeReview may also benefit tasks requiring deep comprehension of long text.Source: Conclusion · p. 9
Limitations
TreeReview’s review quality depends on backend LLM capabilities, including coverage of specialized domains, and LLM hallucinations can propagate factual errors. The work excludes figures as model input and leaves evaluation of multimodal inputs and their computational costs to future work.
LLM Limitations and Risks
- Backend LLM capabilities constrain review quality, including uneven knowledge coverage in highly specialized domains.LLM hallucinations can propagate factual inaccuracies; evidence-based aggregation mitigates but does not eliminate misleading feedback risk.Source: Limitations · p. 10
Multimodal Input Consideration
- TreeReview does not use figures as model input, relying instead on captions and paper analysis for essential information.The authors leave evaluation of multimodal models and whether their benefits outweigh computational costs to future work.Source: Limitations · p. 10
Ethical Considerations
TreeReview is intended to assist, not replace, human reviewers, and its outputs should not serve as standalone official reviews or final recommendations. Its appendix describes question-aware chunk selection and intermediate-question processing that synthesizes answers when evidence is sufficient or expands questions to address gaps.
Ethical Considerations
- TreeReview outputs are intended as supplementary feedback subject to human interpretation and oversight, not official standalone reviews or final recommendations.The paper warns that misuse as a substitute for expert assessment could undermine fairness, transparency, and trust in peer review.Source: Ethical Considerations · p. 10
A.1 Question-aware Chunk Reranking
- Question-aware reranking selects the three chunks with lowest question perplexity as context for answering each leaf question.The paper segments text into roughly 1,024-token chunks while allowing chunks to exceed that size to avoid cutting paragraphs; section hierarchy is prepended for positional cues and citation.Source: §A.1 · p. 14
A.2 Intermediate Question Processing
- For an intermediate question, the synthesizer integrates subanswers when they suffice or generates follow-up questions targeting evidence gaps.It is instructed to reason about sufficiency before producing the answer or follow-up questions.Source: §A.2 · p. 14
B.1 Benchmark Construction
The benchmark samples ICLR-2024 and NeurIPS-2023 papers and reviews with balanced acceptance decisions and topic diversity, then provides references for full reviews and actionable comments. The appendix describes several review-generation baselines, including fine-tuned, prompt-based, and multi-agent approaches.
B.1 Benchmark Construction
- The benchmark contains 40 ICLR-2024 and 40 NeurIPS-2023 papers with corresponding human reviews, sampled from SEA’s test set.The source papers’ preprocessed Markdown retains text, tables, and equations but excludes visual elements such as figures.Source: §B.1 · p. 14
- Sampling balances accepted and rejected papers within each venue and uses semantic diversity to select papers with distinct topics.The Min-Max procedure begins with a random paper in each venue-decision category, then favors candidates without keyword overlap and with maximal minimum embedding distance from selected papers.Source: §B.1 · pp. 14–15
- The sampled papers span topics across both venues and acceptance decisions, as illustrated by the benchmark’s t-SNE visualization.The visualization projects concatenated titles and abstracts using multilingual-e5-small embeddings; the figure caption identifies sampled-paper topic diversity across venues and decisions.Source: §B.1 · p. 15, Figure 6 · p. 15
- Full-review references use original human reviews and ratings, while actionable-comment references consolidate reviewers’ major feedback points.The comment extraction and merging instructions appear in Figures 7 and 8; merging combines similar comments while preserving distinct perspectives, and manual checks on 15 cases found reliable extraction and merging.Source: §B.1 · p. 15, Figure 7 · p. 16, Figure 8 · p. 16
- The benchmark’s average paper length is about 20K tokens, posing a challenge for capturing nuanced details while maintaining comprehensive understanding.Table 6 summarizes paper, review, and comment distributions across venues and evaluation settings.Source: §B.1 · p. 15, Table 6 · p. 15
B.2 Baselines Implementation Details
- REVIEWER2 uses separate fine-tuned models for aspect-prompt generation and review writing, while SEA-E fine-tunes on standardized reviews integrated from multiple reviewers.REVIEWER2’s PGE pipeline annotates existing review datasets with aspect prompts; SEA standardization uses GPT-4 distillation.Source: §B.2 · pp. 15, 17
- DGE prompts with conference review guidelines and accept- and reject-leaning examples, whereas SORT uses section templates to elicit outline-format reviews.SORT also extracts rejection-focused comments, enabling its use for both full-review and actionable-comment tasks.Source: §B.2 · p. 17
- MARG coordinates leader, worker, and expert agents across review aspects; the implementation refines its messaging protocol and includes a no-refinement variant.The refinement removes SEND FULL MESSAGE, limits agents to one SEND MESSAGE per output, and delays broadcasting until planning is complete.Source: §B.2 · p. 17
C.1 Text Similarity-based Evaluation
The appendices describe text-similarity, LLM-judge, specificity, and alignment evaluations. TreeReview performs strongly on the reported alignment measures, while text-similarity scores are close to DGE.
C.1 Text Similarity-based Evaluation
- TreeReview scored 49.94 on ROUGE-1 and 85.27 on BERTScore, closely matching or slightly exceeding DGE, which led the baseline methods on most metrics.Differences across ROUGE and BERTScore were marginal, and these metrics may not capture differences in critical insights or actionable suggestions.Source: §C.1 · p. 18, Table 7 · p. 18
C.2 LLM-as-Judge Evaluation
- Three LLM scoring runs produced an average-rater absolute ICC of 0.9642, indicating consistency across independent LLM-based evaluations.The scores were averaged across runs to smooth minor judge variance.Source: §C.2 · pp. 18–19
C.3 Specificity Evaluation
- ITF-IDF is a reference-free specificity measure that discourages repeated comments within a review and generic comments shared across papers.Higher scores indicate comments are more specific to an individual paper and less generic across papers.Source: §C.3 · p. 19
C.4 Semantic Similarity-based Alignment
- TreeReview reached 47.99% SN-Precision and 48.83% SN-F1, indicating strong semantic alignment with human feedback compared with baselines.The authors describe SN metrics as rough estimates because semantic distance may miss nuanced comment relationships and depends on the embedding model.Source: §C.4 · p. 19
C.5 LLM-based Alignment
- TreeReview achieved 32.10% precision and 15.72% highly related comments in LLM-based alignment, with a competitive pseudo-jaccard score.The authors also report the highest proportion of “more specific” comments among methods.Source: §C.5 · p. 20
C.6 Human Evaluation
The human evaluation uses expert pairwise comparisons of anonymized reviews and feedback comments. Evaluators showed substantial agreement.
C.6 Human Evaluation
- Five experienced reviewers assessed paired outputs for 20 papers, with two distinct evaluators independently judging each paper and presentation order randomized.Outputs were anonymized; criteria covered thoroughness, constructiveness, and specificity for reviews, and accuracy, specificity, and helpfulness for comments.Source: §C.6 · pp. 20–21
- Expert evaluators reached 0.75 overall agreement and a Cohen κ of 0.70, indicating substantial consistency in their judgments.The reported agreement measures assess evaluation reliability.Source: §C.6 · p. 21
D Case Study
In a sampled-paper case study, TreeReview generated feedback that was more specific and closely aligned with reviewers’ concerns than baseline comments. Its question tree illustrates how decomposition and dynamic expansion support targeted critique.
D Case Study
- TreeReview identified missing parameter choices and design justifications, questioned mini-ImageNet’s suitability for knowledge transfer, and alone flagged missing computational-complexity discussion.The case study reports these as more concrete critiques than those offered by comparison methods.Source: D Case Study · p. 21
- Across the case study examples, TreeReview’s comments were more specific and more closely aligned with the concerns expressed by human reviewers.The paper also presents a full generated review as an example of comprehensive assessment.Source: D Case Study · p. 21
- Many highly aligned comments traced to fine-grained questions and dynamically expanded follow-ups that probed ambiguous or insufficiently addressed areas.The question tree covered aspects including novelty, methodology, and limitations.Source: D Case Study · p. 21
E.1 Tree Structure Configuration Analysis
The appendix examines robustness to initial question-tree size, recurring failure modes, performance by paper type, and the prompts used across TreeReview’s stages.
E.1 Tree Structure Configuration Analysis
- Reducing first-level branching from five questions to four still let dynamic expansion recreate an omitted dataset-statistics inquiry, illustrating robustness to initial-tree omissions.The compact configuration reduced subsequent branching limits from four and three to three and two, respectively.Source: §E.1 · pp. 21–22
E.2 Error Mode Analysis
- The analysis identifies two boundaries: visual feedback is outside scope because multimodal inputs are not processed, and subtle cross-section inconsistencies remain difficult to track.The latter can require following detailed information scattered across the manuscript despite final aggregation.Source: §E.2 · p. 22
E.3 Performance Analysis by Paper Type
- Across the full dataset, paper type had no significant correlation with alignment; among the lowest-performing 20%, Application and Theoretical papers were slightly overrepresented.The authors hypothesize that domain-specific context and intricate reasoning may contribute to these patterns.Source: §E.3 · p. 22
F Prompts used in TreeReview
- The appendix provides prompts for question decomposition, leaf answering, intermediate answer aggregation and follow-up generation, and root-level review or feedback generation.The prompts are presented in Figures 13–17.Source: F Prompts used in TreeReview · p. 22