Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation

1 Introduction

• LLMs struggle to produce introductions that balance coherent motivation, scope, significance, academic rigor, and grounding in current literature, while existing benchmarks and evaluations leave their capabilities underexplored.Prior work also often misses faithfulness, narrative consistency, citation accuracy, and alignment with academic standards.Source: §1 · pp. 1–2
• The study presents a large-scale benchmark for scientific introduction generation and evaluates outputs across 30 metrics spanning automatic, LLM-judge, and human assessments.The evaluation covers lexical overlap, semantic similarity, content coverage, faithfulness, consistency, citation correctness, and narrative quality.Source: §1 · p. 2
• A new end-to-end pipeline combines PDF text extraction, LLM-assisted citation identification, and SemanticScholar retrieval to curate 3,900 samples from NAACL 2025 and ICLR 2025 papers.The datasets are intended to provide a benchmark based on recent papers rather than collections likely to have appeared in LLM training data.Source: §1 · p. 2
• The paper systematically explores few-shot prompting strategies and conducts a small-scale human evaluation to compare generated introductions with expected standards.The human study complements the automatic evaluations and aims to provide practical insights for LLM-assisted scientific writing.Source: §1 · p. 2

Multi-Document Summarization (MDS)

• MDS approaches use tailored pretraining, hierarchical reference trees, or graph representations, but often require extra linguistic annotations or parsing pipelines.Examples include PRIMERA (Xiao et al., 2022), Nandy and Bandyopadhyay (2025), and graph-based methods (Liao et al., 2018).Source: §2 · p. 2

Large Language Models for Scientific Summarization

• General-purpose LLMs perform well on summarization benchmarks, but multi-input, constraint-driven generation such as meta-review synthesis remains underinvestigated.The passage names GPT-3.5, GPT-4 (Achiam et al., 2023; Ouyang et al., 2022), PaLM (Chowdhery et al., 2023), LLaMA (Touvron et al., 2023), Bloom (Workshop et al., 2022), and GLaM (Du et al., 2022).Source: §2 · p. 2

LLM-based Academic Text Generation

• Liebling et al. (2025) study interactive introduction authoring with retrieved citation suggestions, whereas this work benchmarks one-pass generation from fixed inputs without retrieval, citation recommendations, or human scaffolding.The inputs are Title, Abstract, and Related Work.Source: §2 · p. 2
• AutoSurvey (Wang et al., 2024) and SurveyGen (Bao et al., 2025) address broad survey generation, which prioritizes comprehensive coverage over introductions’ precise argumentation and contextual positioning.SurveyGen pairs human-written surveys with the quality-aware retrieval-augmented framework QUAL-SG.Source: §2 · p. 3

3.1 Problem Formulation

• SciIG asks an LLM to generate a coherent, academically styled introduction from a paper’s title, abstract, and related papers.Each related paper is represented by its title, abstract, and author list; the model is conditioned on a task-specific prompt.Source: §3.1 · p. 3
• The three inputs are chosen to provide sufficient information for establishing research territory, identifying a gap, and positioning the contribution.The authors report that manual inspection of 20 NAACL and ICLR papers confirmed this structure; other sections are excluded for reproducible benchmarking, lower token and computational costs, and easier extraction across variable PDF formats.Source: §3.1 · p. 3

3.2 Datasets

• The NAACL 2025 and ICLR 2025 datasets contain 800 and 3,100 samples, respectively, with a 94.0% citation resolution rate.The corpus is restricted to 2025 ML/NLP papers for community-standard editorial quality, domain consistency, and publication after the evaluated models’ pretraining cutoffs.Source: §3.2 · p. 3
• NAACL introductions average 597.1 ± 228.4 words and 13.4 citations, while ICLR introductions average 771.1 ± 414.3 words and 15.3 citations.Abstract median lengths are 162 words for NAACL and 194 for ICLR; fewer than 2% of samples have abstracts under 60 words, mostly due to parsing artifacts.Source: §3.2 · pp. 3–4
• The construction pipeline retrieves and parses conference papers, extracts and maps in-text citations, then enriches cited works with Semantic Scholar metadata.Citation strings are mapped to bibliographic entries using LLaMA 4-Maverick; Figure 1 depicts the three-phase process and the pairing of inputs with published introductions.Source: §3.2 · p. 3, Figure 1 · p. 4
• The initial study is limited to ML because recruiting domain-expert annotators is challenging; the authors annotated outputs themselves.They report minimal author-bias risk because no model was trained on the data and the study is benchmarking-only.Source: §3.2 · p. 4

4.1 Methods

• Five models—DeepSeek-V3-0324, LLaMA 4-Maverick, Mistral-Small-3.1-24b, Gemma-3-12b-it, and GPT-4o—were evaluated on NAACL 2025 and ICLR 2025.The models span open-source and commercial systems and have knowledge cutoffs before January 2025.Source: §4.1 · p. 4
• Seven prompting strategies range from Short, Medium, and Elaborate to One-Shot, Two-Shot, Three-Shot, and iterative self-refining AutoCoT.Prompts use the paper’s title, abstract, and JSON-formatted related papers; Elaborate enforces four paragraphs of 100–150 words, while few-shot strategies use held-out examples.Source: §4.1 · pp. 4–5

4.2 Evaluation

• Evaluation spans lexical overlap, semantic similarity, content coverage, faithfulness, consistency, citation correctness, and narrative quality.Automated metrics are complemented by GPT-4o judge scores for content coverage, faithfulness, consistency, citation contextual quality, and narrative quality.Source: §4.2 · pp. 5–6
• Coverage uses complementary keyphrase measures: RBC compares generations with the published introduction, while RFC checks grounding in the title and abstract.Keyphrases match only when their stemmed forms are identical; the published introduction is one accepted reference, not a unique gold standard.Source: §4.2 · p. 5
• Faithfulness measures whether generated keyphrases are grounded in the title and abstract, whereas consistency compares generated text with the published introduction.Both measures also use an LLM-as-a-Judge score.Source: §4.2 · p. 5
• Citation correctness measures citation precision and recall against the original Related Works list and judges whether cited claims fit paper metadata and rhetorical context.Citation strings are extracted using LLaMA 4-Maverick; Citation Contextual Quality is an LLM-as-a-Judge score.Source: §4.2 · p. 5
• Narrative quality combines GPT-2 perplexity, where lower values indicate more natural text, with judge scores for grammar, flow, tone, and readability.The judge score complements perplexity by assessing those qualities directly.Source: §4.2 · p. 5

5 Results and Analysis

• LLaMA4-Maverick leads overall, with the strongest performance across lexical overlap, content coverage, citation quality, and several judge-rated dimensions.The benchmark covers five models on the NAACL 2025 and ICLR 2025 SciIG datasets.Source: §5 · p. 6
• Gemma excels in semantic alignment and competes on content coverage and citation correctness, while Gemma and Mistral show competitive faithfulness and consistency.GPT-4o and DeepSeek trail on most metrics, with challenges in factual accuracy and lexical alignment.Source: §5 · p. 6

5.1 Analysis on Few-shot Experiments

• Three-shot prompting consistently outperforms the other few-shot settings on lexical overlap and reference-free content coverage, with diminishing returns after two-shot.Three-shot setups also score higher on SummaC-ZS faithfulness, suggesting exemplars help preserve input factual correctness.Source: §5.1 · pp. 6–7, Table 3 · p. 7

5.2 Analysis with Various Prompt Strategies

• Medium-length prompts achieve higher ROUGE scores, whereas AutoCoT yields superior METEOR and keyphrase-based faithfulness but increases perplexity.Prompt complexity should be adjusted to the priority: lexical fidelity, conceptual alignment, or narrative fluency.Source: §5.2 · p. 7, Table 3 · p. 7

6 Ablation Study

• The study compares Title-only, Title + Abstract, and Title + Abstract + Related Work inputs under ELABORATE and MEDIUM prompting, using lexical, semantic, and LLM-based metrics.It tests these configurations on the overall front-runner, LLaMA4-Maverick.Source: §6 · p. 7
• A full-paper-context case study reports gains in methodological detail and result-driven framing, while reinforcing the realism–scalability trade-off behind restricted input.The full paper context excludes the Introduction.Source: §6 · p. 7
• Adding related work improves nearly all metrics, including BERTScore (0.8448 →0.8471), citation precision (0.3610 →0.9357), recall (0.1199 →0.5574), and LLM-judge citation quality (0.5184 to 0.8051).The citation-overlap jump reflects penalties for diverging from the original citation set, which is not treated as a unique oracle; Section 7 human evaluation checks citation relevance and non-fabrication.Source: §6 · p. 7, Table 4 · p. 8

Human Evaluation

• The human evaluation covered 80 outputs from 20 source examples, with four generations per example and three authors each annotating 6–7 papers.No paper received multiple annotations because domain-expert annotators were unavailable.Source: §7 · p. 8
• GPT-4o outperforms 0-shot LLaMA-4 on most criteria, 3-shot prompting scores highest on content coverage, idea flow, and clarity, and Elaborate 0-shot beats AutoCoT across dimensions.LLaMA-4 shows stronger hallucination resistance than GPT-4o, while Elaborate 0-shot has notable gains in citation quality and faithfulness over AutoCoT.Source: Table 5 · p. 8, §7 · p. 8
• In a detailed NAACL 2025 example, both models produced fluent, faithful introductions with strong narrative flow and accurate citations but missed structure and deeper details absent from the inputs.The missing information included explicit research questions and contribution-style summaries.Source: §7 · p. 8
• The study also computed pairwise Spearman correlations between human ratings, LLM-as-Judge scores, and formula-based automated metrics, with full analysis in Appendix D.Source: §7 · p. 8

8 Conclusion and Discussion

• Structured prompts enable LLMs to generate coherent, well-structured introductions that align with academic norms.The prompts include clear instructions, a meaningful title and abstract, and relevant related works.Source: §8 · p. 9
• LLM-generated introductions are not ready to use as-is because they miss fine-grained technical details, contextualize citations inaccurately, and imprecisely state contributions.These shortcomings necessitate substantial post-editing and expert input.Source: §8 · p. 9
• LLMs are promising assistants in scholarly writing, but they are not yet suitable replacements for human authorship.Source: §8 · p. 9

• Scientific images and diagrams are outside scope because generating abstract, non-natural diagrams is challenging.The authors plan to address diagram generation in future work.Source: Limitations · p. 9
• Conditioning on titles, abstracts, and related work omits methods/results signals; future iterations may add full sections or extracted keyphrases, though full sections increase token use and require consistent extraction.The current input choice keeps the setup reproducible and scalable.Source: Limitations · p. 9

Ethical Considerations

• The authors reject autonomous scholarly use of LLMs and require human review because generations can contain factual errors, improper citations, and weak domain grounding.They also caution that proprietary-model use may transmit research content to third-party servers, raising privacy, confidentiality, and security concerns.Source: Ethical Considerations · p. 9

A Justification for Model Choices

• The evaluated models were selected for complementary reasoning, context-handling, academic-prose, and efficiency strengths across diverse architectures and access types.The set spans open- and closed-source models, including mixture-of-experts and dense architectures, and was chosen over alternatives such as Claude or Qwen.Source: A Justification for Model Choices · p. 12

B Implementation Details

• Generation used nucleus sampling at temperature T = 0.7 and topp = 0.95 with a fixed random seed across models and prompting strategies.These settings were held constant within experiments so observed differences would reflect model or prompt rather than sampling variation.Source: B Implementation Details · p. 12

B.1 AutoCoT Prompting

• AutoCoT uses a single-pass prompt for internal reasoning before introduction generation, without external stopping criteria or iterative critique–revision.This setup isolates autonomous reasoning from additional engineering machinery and keeps prompting-strategy comparisons controlled.Source: §B.1 · p. 12

B.2 Contextual Relevance

• Contextual Relevance averages cosine similarities between Longformer embeddings of generated introductions and their title, abstract, and cited-works contexts.Mask-aware mean pooling over non-padding tokens produces the vectors; this metric does not use BERTScore or Sentence-BERT.Source: §B.2 · pp. 12–13

B.3 Keyphrase Extraction and Matching

• Both reference-based and reference-free coverage use top-five KeyBERT phrases, Porter-stemmed and matched by exact equality, differing only in reference set.Phrases use all-MiniLM-L6-v2, an n-gram range of 1–3, and English stopword removal; references are respectively the ground-truth introduction or title and abstract.Source: §B.3 · p. 13

C Semantic Scholar Metadata Resolution

• Missing citation metadata triggered title-normalized unofficial-endpoint queries, then a Graph API fallback (Kinney et al., 2023) requiring title similarity ≥0.85.Related papers without usable metadata were excluded from citation context, and targets with all references unresolved were removed.Source: C Semantic Scholar Metadata Resolution · p. 13
• 59,619 related-paper lookups yielded 94.0% citation resolution, and roughly 10% of resolved entries lacked abstracts but retained title and author metadata.Targets whose cited references all remained unresolved were excluded from the final dataset.Source: C Semantic Scholar Metadata Resolution · p. 13

D Metric Agreement Analysis

• LLM-as-Judge agrees with formula-based metrics on Faithfulness (ρ=0.35), Consistency (ρ=0.45), Content Coverage (ρ=0.30), and Narrative Quality (ρ=0.34), but diverges on citation quality (ρ=−0.19).The analysis uses Spearman rank correlations on the human-evaluated subset (n=79–80).Source: D Metric Agreement Analysis · p. 13
• Both formula-based metrics and LLM-as-Judge show weak agreement with human ratings (|ρ| ≤0.20), motivating multichannel evaluation rather than reliance on one metric family.Source: D Metric Agreement Analysis · p. 13

E Dataset Construction Pipeline: Detailed Steps

• The five-step dataset pipeline retrieves NAACL 2025 and ICLR 2025 papers, parses PDFs, extracts and maps related-work citations, then enriches them with Semantic Scholar metadata.Parsing uses grobid2json from S2ORC (Lo et al., 2020); citation mapping uses LLaMA 4-Maverick, and enrichment uses official and unofficial Semantic Scholar API endpoints.Source: E Dataset Construction Pipeline:… · p. 14

F.1 Lexical Overlap

• Lexical overlap uses ROUGE-1/2, ROUGE-L, BLEU, and METEOR to assess n-gram overlap, longest common subsequences, modified precision, and stemming or synonym matches.BLEU includes a brevity penalty; METEOR captures morphological and lexical variation that ROUGE and BLEU may miss.Source: §F.1 · pp. 14–15

F.2 Semantic Similarity

• Semantic measures include BERTScore’s contextual-token cosine similarity, BLEURT’s human-rating-trained quality regression, and Contextual Relevance to input-context embeddings.Contextual Relevance averages similarity to the title, abstract, and cited papers, reducing sensitivity to a drifting or very short context element.Source: §F.2 · p. 15

G.1 Citation Correctness

• Faithfulness compares generated keyphrases with the title and abstract, consistency compares them with the ground-truth introduction, and citation precision and recall evaluate citation identifiers against the paper’s related-work list.Faithfulness and consistency use stemmed exact keyphrase matching; citation contextual quality separately judges whether cited metadata is accurately represented and appropriately placed.Source: G Faithfulness and Consistency Metrics · p. 15, §G.1 · p. 15

G.2 Narrative Quality

• GPT-2 perplexity serves as a surface-fluency proxy, while GPT-4o judges grammar, logical flow, academic tone, and readability using structured scores and rationales.The same judge model scores all five LLM-judge dimensions on a 0–1 scale to keep the rubric internally consistent.Source: §G.2 · pp. 15–16, §G.3 · p. 16

H A Case Study

• Annotators praised LLaMA-4 Maverick and GPT-4o for faithful, coherent introductions and contextually appropriate citations, even under zero-shot conditions.All configurations also produced strong motivation sections.Source: H A Case Study · p. 16
• In the restricted-input case study, annotators found that models often omitted paper-specific details and produced conclusion-like contribution summaries.The three research questions were absent from the title, abstract, and related works; annotators also noted GPT-4o sometimes drifted into irrelevant discussion and citations, whereas LLaMA-4 produced strong background sections.Source: H A Case Study · p. 16

H.1 Full-Paper-Context Case Study

• Full-paper context let GPT-5.1 recover the three research questions, summarize methodology, and incorporate findings, but required unavailable-at-writing-time sections and increased input costs.The paper notes full-paper inputs of 5,000–10,000 words and structural-extraction challenges; restricted inputs were chosen as a trade-off for realism, scalability, and reproducibility.Source: §H.1 · p. 16