PaperlayerView exact revision on arXiv

Source-linked AI summary

ReGround: Grounding Reviewer Comments in Multimodal Evidence

Serwar Basch, Lizhen Qu, Iryna Gurevych

arXiv:2609.11460v1cs.CLcs.IR

TL;DR

Reviewer comments are often underspecified and must be grounded in heterogeneous evidence across long scientific papers, a setting existing benchmarks do not directly support. ReGround builds a rebuttal-derived multimodal grounding dataset and evaluates retrieval methods, finding that unified retrieval is difficult, evidence-type inference is a major bottleneck, and text and visual signals are complementary. The dataset frames reviewer-comment grounding as a demanding problem for scientific document understanding.

  • Problem

    Existing benchmarks do not directly capture grounding naturally occurring, evaluative reviewer comments in heterogeneous multimodal evidence from original scientific-paper submissions.

  • Method

    ReGround uses explicit paper references in author rebuttals to construct comment–evidence links and casts grounding as retrieval over multimodal paper evidence.

  • Results

    Unified retrieval peaks at 21% Recall@10; evidence-type inference is the dominant bottleneck, while text and image signals succeed on different instances.

  • Takeaways & Limitations

    Reviewer-comment grounding requires type-aware routing, multi-evidence aggregation, and multimodal fusion for scientific document understanding.

  • Takeaways & Limitations

    The dataset reflects author-cited evidence, may underrepresent unresolved comments and contain coarse links, and is limited to NLP papers with separate text and image candidate pools.

Abstract

from arXiv · show

Reviewer comments naturally relate to specific parts of the reviewed paper, yet grounding these comments to the underlying evidence is difficult due to long multimodal documents. Existing benchmarks do not capture this setting and largely focus on explicit, information-seeking queries. We introduce ReGround, a large-scale dataset for reviewer comment grounding that links 10,267 reviewer comments to 16,274 evidence in the original anonymous submission of 3,656 papers. We build on a simple observation: author rebuttals often include explicit references to content of the submission used to address reviewer comments, providing a high-precision annotation source. We cast grounding as a retrieval task and evaluate a wide range of retrieval methods. Results show that retrieval over the entire paper content performs poorly, evidence-type inference is a major bottleneck, and multimodal evidence provides complementary signals that text alone misses. Our dataset exposes grounding reviewer comments as a difficult and practically important problem for scientific document understanding.

1 Introduction

Grounding reviewer comments requires linking underspecified comments to heterogeneous evidence in long scientific papers, a setting existing benchmarks do not directly capture. ReGround addresses this gap with a rebuttal-derived dataset and retrieval evaluation.

  • Motivation: ReGround targets grounding evaluative, underspecified reviewer comments in heterogeneous paper evidence spanning text, tables, and figures.Existing scientific QA resources focus on information-seeking questions with relatively well-defined answer spans, while revision datasets link comments to later paper edits.
  • Approach: Author rebuttals provide high-precision links between reviewer comments and evidence in the original anonymous submission.Authors explicitly point to submitted-paper content when addressing concerns, unlike links to revised or camera-ready versions.
  • Dataset: ReGround contains 3,656 papers and 16,274 reviewer comment–evidence pairs, covering paragraphs, sections, tables, and figures.The dataset is designed for multimodal evidence grounding and supports retrieval-based evaluation.
  • Findings: Unified grounding remains difficult: the best LLM-based ranker reaches only 21% Recall@10, evidence-type inference is a major bottleneck, and multimodal signals are complementary.Models also struggle with comments grounded in multiple evidence units, while text and visual modalities exhibit different failure modes.

2 Related Work

Related datasets study scientific QA, peer-review interactions, revisions, or document-level links, but do not directly model fine-grained grounding of reviewer comments to original paper content. ReGround differs by targeting these comment–evidence links at fine granularity.

  • Peer-review corpora: Prior peer-review corpora model reviewing processes and discourse, but generally operate at the document level rather than linking comments to specific paper content.These resources include review–rebuttal exchanges and discourse or argument annotations.
  • Revision-based resources: ARIES and RE3 align reviewer comments with revisions, whereas ReGround links comments to evidence in the original submission.Revision-based datasets capture how feedback manifests in later papers rather than the evidence available during review.
  • Scientific QA: PEERQA reframes reviewer questions as document-level QA with manually annotated answer spans, but uses camera-ready papers and limited evidence scope.This differs from grounding naturally occurring comments in multimodal evidence from the reviewed submission.
  • Evidence retrieval: Scientific evidence-retrieval and QA benchmarks provide related foundations, but reviewer comments remain naturally occurring, underspecified inputs requiring fine-grained multimodal grounding.The setting therefore differs from explicit information-seeking queries and conventional document-level QA.

3 Dataset Construction

ReGround constructs reviewer-comment evidence links by extracting paper references from rebuttals, aligning them to review spans, and resolving them in the original submission. The resulting dataset spans multiple evidence types and exposes retrieval challenges caused by heterogeneous evidence and weak semantic overlap.

  • Link construction: The construction pipeline detects paper references in rebuttals, aligns them to reviewer comments, and resolves them to evidence units in the original submission.The task focuses on implicit comments without explicit references that could be resolved by string matching.
  • Evidence extraction: The retrieval pool includes paragraphs, sections, captions, and figure or table images, with line references mapped to containing paragraphs.Structured PDF representations provide textual and visual evidence units at multiple granularities.
  • Validation and difficulty: Alignment validation accepted 96% of sampled links, while filtering achieved 97% precision and 93% recall against a manually annotated subset.The reported checks support the precision of the automated construction pipeline, although some alignment errors involve under-selected reviewer spans.
  • Dataset statistics: The final dataset contains 3,656 papers, 10,267 reviewer comments, and 16,274 evidence units, with 25.77% of comments linked to multiple evidence.This motivates evaluating both single- and multi-evidence retrieval.
  • Dataset statistics: Nearly 90% of evidence references target lines, sections, figures, or tables, while textual units vary widely in length and candidate-pool size.Uneven information density and span length complicate retrieval across evidence types.
  • Validation and difficulty: Reviewer comments and gold evidence have limited semantic overlap, with average comment–evidence similarity of 0.377 versus 0.431 for rebuttal sentence–evidence pairs.This weak overlap motivates methods beyond semantic matching alone.

4 Experimental Setup

The experiments evaluate reviewer-comment grounding through unified, type-aware, joint, and visual retrieval settings. They compare text and multimodal retrieval models using exact evidence-unit matching, Recall@k, and MRR.

  • Unified text retrieval (RQ1): Unified retrieval ranks paragraphs, sections, and figure/table captions in one heterogeneous pool without evidence-type hints.This setting tests relevance retrieval and evidence-type inference jointly.
  • Type-aware text retrieval (RQ2): Type-aware retrieval restricts candidates to the gold evidence type, isolating retrieval difficulty from evidence-type inference.Sections, tables, figures, and other evidence types are evaluated against matching candidate pools.
  • Type-aware joint evidence retrieval (RQ3): Joint retrieval evaluates whether models recover the full set of relevant evidence for comments grounded to multiple non-redundant units.The evaluation uses oracle type information to focus on multi-evidence retrieval.
  • Visual evidence retrieval (RQ4): Visual retrieval compares raw figure/table images with images augmented by their captions.The broader model suite includes vision–text encoders, late-interaction retrieval, and multimodal LLMs.
  • Models: The study compares sparse, dense, cross-encoder, LLM-based, vision–text, late-interaction, and multimodal retrieval models.LLM rankers score comment–evidence pairs using binary relevance probabilities converted into continuous scores.
  • Evaluation metrics: Performance uses Recall@k and MRR with exact evidence-unit matching, giving no partial credit for hierarchically related units.Recall measures relevant units retrieved in the top k, while MRR emphasizes the rank of the first relevant unit.

5 Results

ReGround retrieval remains difficult across heterogeneous evidence, with evidence-type uncertainty and multi-evidence grounding limiting performance. Visual and textual signals are complementary, while oracle type information substantially improves retrieval.

  • 5.1 Unified Text Retrieval (RQ1): R@10 peaks at 21.15, showing that unified retrieval over heterogeneous textual evidence remains difficult across model families.LLM-based rankers perform best, but their MRR advantage over dense models is modest: 10.87 versus 9.41.
  • 5.1 Unified Text Retrieval (RQ1): Cross-encoder MRR drops from 44 at Q5 to near zero at Q10, whereas LLM-based rankers improve through Q8 and then degrade more gradually.Long section-level candidates in the heterogeneous pool disproportionately penalize cross-encoders.
  • 5.2 Type-aware (oracle) Text Retrieval (RQ2): Restricting retrieval to the gold evidence type substantially improves performance, identifying evidence-type uncertainty as a major error source.However, paragraph- and section-level retrieval remain challenging, with strongest models reaching only around 31% Recall@2.
  • 5.2 Type-aware (oracle) Text Retrieval (RQ2): Hard filtering by predicted evidence type reduces Qwen-3 30B Instruct’s R@10 from 21.15 to 6.11, so type prediction is better used for soft reranking.Prediction errors on figure and table types can remove the correct evidence from the candidate pool.
  • 5.3 Type-aware joint evidence retrieval (RQ3): For comments with multiple evidence units, Qwen-3 30B Instruct retrieves only 12.69% of the full evidence set at R@1 and 23.53% at R@2.The result holds under favorable oracle type-aware conditions, indicating difficulty aggregating non-redundant evidence units.
  • 5.4 Visual Evidence Retrieval (RQ4): Caption augmentation consistently improves visual retrieval, while image-only retrieval remains competitive, indicating complementary textual and visual signals.These results support multimodal fusion rather than relying exclusively on either captions or images.

6 Error Analysis

Error analysis finds failures caused by imprecise or coarse references, generic captions, and underspecified comments requiring local context. Text and image retrieval also fail on substantially different instances.

  • 6 Error Analysis: Imprecise line references often point to section headers, while coarse section references can miss evidence localized in subsections.References may also navigate toward another evidence unit, such as a table, leaving the cited text insufficient on its own.
  • 6 Error Analysis: Generic or duplicated captions cause plausible-but-wrong matches because important details may remain only in the associated figure or table.Caption failures therefore reflect both insufficient content and similarity across multiple visual candidates.
  • 6 Error Analysis: Underspecified comments remain difficult because successful grounding requires identifying referenced local context before locating the explanatory evidence.The analysis gives questions such as how an error was introduced at a cited line as an example.
  • 6.2 Text vs. image retrieval disagreement: For tables, image retrieval reaches 60.34% Recall@2 versus 53.29% for caption text, with 29.7% disagreement between modalities.For figures, image retrieval is 58.47% versus 55.16%, and the modalities disagree on 24.1% of instances.

7 Conclusion

ReGround grounds reviewer comments in fine-grained multimodal evidence from original anonymous submissions using author rebuttals as annotations. The task remains difficult: unified retrieval peaks at 21% Recall@10, while evidence-type inference and multimodal, multi-evidence reasoning remain challenging.

  • Author rebuttal references scale expert-like grounding annotations to 3,656 papers and 16,274 reviewer comment–evidence pairs.
  • 21% Recall@10 is the peak for unified retrieval, underscoring the difficulty of grounding reviewer comments in heterogeneous evidence.
  • Evidence-type inference is a dominant bottleneck, and comments linked to multiple evidence items remain difficult even under oracle conditions.
  • Text and image signals succeed on different instances, motivating multimodal fusion rather than replacing one modality with the other.

Ethical considerations

The study uses voluntarily donated NLPeer peer-review data and releases it under CC-BY-NC 4.0. The dataset contains no personally sensitive information beyond what authors and reviewers publicly shared.

  • The data consist of original anonymous submissions, reviews, and rebuttals voluntarily donated by authors and reviewers.
  • The dataset was collected, processed, and will be released under the CC-BY-NC 4.0 license.
  • The dataset contains no personally sensitive information beyond material publicly shared by authors and reviewers.

Limitations

ReGround focuses on implicitly related reviewer comments and author-cited evidence, while several methodological and domain boundaries constrain coverage and interpretation. Its supervision can omit plausible evidence, use coarse links, and does not yet evaluate fully unified multimodal or end-to-end retrieval.

  • The dataset excludes comments that explicitly reference paper content, so it does not capture the full spectrum of reviewer behaviors.
  • Author rebuttals may underrepresent unresolved or weakly grounded comments because they emphasize content authors chose to address.
  • Author-provided evidence links can be coarse, such as section-level links where a more specific subsection would be appropriate, adding noise to supervision.
  • Evaluation omits task-specific training and end-to-end systems that jointly select evidence types and retrieve evidence.
  • Text and image retrieval use separate candidate pools, leaving fully unified multimodal retrieval for future work.
  • The dataset is limited to NLP papers, whose evidence types and granularities may differ from those in other domains.

F Manual Inspection

Manual inspection found that author-cited evidence was generally resolved correctly and relevant to reviewer comments. However, evidence boundaries remain subjective, and author-cited links do not guarantee complete coverage.

  • Approximately 1.58% of inspected paper references may be incorrectly resolved, indicating high but imperfect reference-resolution accuracy.
  • Authors are treated as expert annotators because they best understand whether cited evidence addresses a paper-specific reviewer concern.
  • All manually checked referenced evidence was relevant, and reviewers acknowledged the rebuttal in 84% of sampled comments.
  • Figure 3 presents the distribution of cosine similarity between reviewer comments and their target content units.

G Cosine Similarity Analysis

The analysis characterizes reviewer-comment grounding through similarity, evidence composition, and relation directionality. It finds modest, overlapping similarity across evidence types and emphasizes that retrieval precedes relation classification.

  • Cosine similarity: 0.377 mean cosine similarity across reviewer comment–evidence pairs, with substantial overlap and no consistently high-similarity evidence category.Similarities use Sentence-Transformers embeddings and caption text for figure/table evidence.
  • Evidence composition: Multi-evidence comments require retrieving and combining same-type, cross-granularity, and multimodal evidence units.The reported combinations cover 55% of multi-evidence combinations.
  • Relation directionality: ReGround retrieves paper content linked to reviewer comments and author responses without assuming whether that evidence supports, qualifies, or contradicts the comment.Relation typing is treated as a separate downstream task after evidence retrieval.
  • Relation directionality: A descriptive single-annotator sample suggests explanatory or clarifying evidence is more common than corrective evidence.The authors retain retrieval framing rather than treating the sample as a definitive relation analysis.
  • Comment taxonomy: Reviewer comments receive one mutually exclusive primary category from a descriptive taxonomy spanning conceptual, technical, empirical, data, presentation, and deployment concerns.The category assignment is performed by one author and summarized in Table 9.

K Coverage and Representativeness Analysis

The coverage analysis finds that ReGround includes a minority of review spans, while retrieval experiments expose distinct limits from section length, multi-evidence aggregation, and evidence-type prediction. In particular, figure and table type errors can eliminate the correct candidates, favoring soft reranking over hard filtering.

  • Coverage and representativeness: 17,682 of 115,989 review spans are covered, corresponding to 15.24% of weakness and suggestion-related spans.Coverage is measured over summary_of_weaknesses and comments_suggestions_and_typos fields.
  • Coverage and representativeness: The included comments differ from excluded comments especially for administrative or vague remarks and conceptual comments that often lack localized paper content.These exclusions are described as properties of the grounding task rather than evidence of content-related sampling bias.
  • Section length: Cross-encoder MRR improves through mid-length sections before dropping sharply, whereas the LLM retriever improves through Q8 and declines only in the longest quantiles.The comparison uses BGE-M3-Reranker and Qwen3-30B-Instruct across ten section-length quantiles.
  • Multi-evidence retrieval: Oracle-type multi-evidence retrieval remains limited: models struggle to retrieve all relevant units early, even when the gold evidence types are known.Larger LLMs improve overall recall modestly, but no model reliably retrieves the complete evidence set near the top.
  • Evidence-type prediction: Evidence-type prediction identifies dominant text classes more reliably than figures or tables, so predicted types work better as soft reranking signals than hard retrieval constraints.Hard filtering can remove the correct evidence before ranking when figure or table prediction is wrong.
Loading 2609.11460v1…