LimGen: Probing the LLMs for Generating Suggestive Limitations of Research Papers

1 Introduction

• Identifying limitations is tedious and time-consuming, requiring broad knowledge, close attention, and analytical thinking.Source: §1 · p. 1
• SLG aims to generate diverse, paper-specific potential limitations for reviewers, drawing on reasoning and associations across previously encountered papers.Source: §1 · pp. 1–2
• The example shows author-like limitations alongside novel, valid suggestions about adversarial examples.Table 1 marks matching content in magenta and novel content in blue.Source: Table 1 · p. 2, §1 · p. 2
• The authors present SLG as a first task, release LimGen with 4,068 papers and limitations, and test LLM schemes through automated, human, and LLM-based evaluation.Source: §1 · pp. 2–3

2 Related work

• Author-stated limitations offer researchers’ own perspective on study constraints, complementing external critiques from peer reviewers.The passage also associates limitations with transparency, credibility, guidance for future research, and managing reader expectations.Source: §2 · p. 3
• The authors position their work as the first to generate research-paper limitations using ACL Anthology papers and author-written limitations.Source: §2 · p. 3

3.1 Dataset collection

• The dataset comprises 4,068 peer-reviewed short and long papers from three ACL venues; each paper’s actual limitations section serves as reference, while source text retains methodology, experiments, results, and discussions.Source: §3.1 · p. 4
• About 5,000 tokens and 187 sentences per paper contrast with about 230 tokens and 9 sentences per limitations section.The paper notes that the longer paper length challenges LLMs processing long contexts.Source: §3.1 · p. 4

3.2 Nature of the limitations

• In a diverse manual sample of 60 papers, over 50% had limitations that were not straightforward to discern from the paper text.Limitations often described future prospects, varied from sentences to lengthy paragraphs, and were predominantly associated with methodology; few covered multiple sections.Source: §3.2 · p. 5

4.1 Task formulation

• The task treats a research paper R as input and generates a structured limitation block L containing n limitations in sequence.Source: §4.1 · p. 5

4.2 Methodology

• The methodology compares summarization models with generative LLMs, which must infer and recommend limitations rather than simply condense source content.The explored models include BART and PEGASUS for summarization, and T5, Cerebras-GPT, and Llama 2 for generation.Source: §4.2 · pp. 5–6

Non-truncated research paper

• Full-paper prompting and fine-tuning use the paper and its limitations, but maximum input-token limits can prevent comprehensive paper context.The scheme also fine-tunes summarization models and applies zero-shot prompting or fine-tuning to generative models.Source: Non-truncated research paper · p. 6

Dense Passage Retrieval (DPR)

• DPR selects the top three passages with cosine similarity of at least 0.5 for each limitation sentence, then uses retrieved passages as fine-tuning inputs.Because passage-level inputs may lack broader paper context, the approach can produce irrelevant limitations.Source: Dense Passage Retrieval (DPR) · pp. 7–8

Chain Modeling

• Chain modeling generates limitations from passages alongside a paper summary, then refines them by removing obvious duplicates and standardizing language.Its variants use two fine-tuned models, or pass each iteration’s result forward; hallucinations or repetitions can affect subsequent steps.Source: Dense Passage Retrieval (DPR) · p. 8

4.3 Experimental Setup

• Cerebras-GPT 1.3B and Llama 2 7B are fine-tuned with LoRA and 8-bit quantization, using an 80-10-10 dataset split and three training epochs.DPR inputs are capped at two passages for Llama 2 and three for Cerebras-GPT.Source: §4.3 · pp. 8–9

5 Experimental Results and Analysis

• Summarization-trained models often extract source sentences instead of generating insightful limitations, while Llama2 and Cerebras generation quality declines without the “Limitations:” prompt keyword.Both generative models are also constrained by input truncation and limited context across paper sections.Source: §5 · pp. 9–10

Chain modeling.

• DPR-based generation can derive limitations from relevant passages, but passage-level retrieval can also yield less pertinent limitations without the paper’s overall context.The chain-modeling approach adds a full-paper summary alongside passages and removes duplicate limitations during refinement.Source: §5 · p. 10, Chain modeling · p. 10

Human evaluation.

• GPT-4 ranks Llama2-FT-Distilled above the other models and T5-base lowest, but may score generic limitations highly even when they do not fit the paper.GPT-4 can miss incorrect limitations unless an issue is singled out for re-evaluation.Source: Human evaluation · pp. 11–12
• GPT-4o’s completeness and accuracy penalties are inconsistent, and its scoring can disproportionately penalize systems that generate more limitations.The authors report that XGPTScore’s formatted output often omits scores from the initial freeform assessment, requiring manual score transfer.Source: Human evaluation · pp. 12–13
• Llama2-FT-Distilled outperforms the other systems overall and for completeness, producing fewer, more coherent limitations with less non-limitation and speculative content.The authors attribute quality gains to the full-paper summary and distillation step; the DPR-distilled model trails and sometimes produces generic limitations.Source: Human evaluation · p. 13
• Human evaluation found that Llama2-DPR and T5-base often produced noisy, irrelevant, or summary-like outputs, while over half of T5-base outputs were hallucinated or repetitive.The evaluation used four questions covering sense, overlap, whether outputs were actual limitations, and hallucinations, repetitions, or grammatical errors.Source: Human evaluation · pp. 13–14

Complexity of the SLG task

• Limitations can be difficult to infer from paper text alone because the context surrounding them is often not detailed.The authors emphasize fine-tuning to learn similarities and differences across papers, and note that citations could provide auxiliary information.Source: §6 · p. 14

Evaluation metrics suitability

• ROUGE’s lexical matching is imperfect for novel limitation phrasing, while BERTScore showed no notable variation despite human-rated quality differences.The paper identifies a tailored evaluation metric as the most promising path forward.Source: §6 · p. 14

Multi-modal content; Coherence and relevance

• The study does not use images or tables, although these could provide context for nuanced potential limitations.The paper also reports that DPR-based models can lack coherence, while its best Llama2-FT-Distilled model sometimes produces speculative or insufficiently relevant limitations.Source: §6 · p. 15

Open-ended generation of LLMs; Controllability

• Around 20% of LLM-generated limitations are susceptible to hallucinations, repetitions, or grammatical errors.The models also sometimes produce structured limitations inconsistently, switching between bullet points and paragraphs.Source: §6 · p. 15

7 Limitations

• The study uses only paper text, so its generation does not incorporate tables or images that could improve contextual relevance.The experiments were also limited by compute constraints to Llama 2 7B and Cerebras-GPT 1.3B; ACL Anthology sourcing may limit applicability to physics, chemistry, and mathematics.Source: §7 · p. 15
• The automated evaluation metrics are insufficient for the task’s complexity, so the authors supplemented evaluation with manual assessments.The best model’s outputs also included some sub-standard limitations.Source: §7 · p. 15

8 Conclusions

• SLG aims to generate potential limitations of research papers to assist reviewers.The paper describes the task as novel and introduces an LLM-based baseline with ablation studies and automatic, LLM-based, and human evaluations.Source: §8 · pp. 15–16
• LimGen contains 4,068 research papers and corresponding limitations collected from the ACL Anthology.The authors plan to incorporate images and tabular content in addition to text.Source: §8 · pp. 15–16

• The LimGen dataset was created without collecting personal data.Source: §9 · p. 16
• The dataset relies on publicly accessible data from the ACL Anthology.Source: §9 · p. 16
• LimGen is released under CC BY 4.0, while its code is released under the MIT license.The dataset license matches the original ACL Anthology license.Source: §9 · p. 16