PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation

Krishna Garg, Firoz Shaik, Sambaran Bandyopadhyay, Cornelia Caragea

arXiv:2508.14273v3cs.CL

TL;DR

Existing LLM writing systems remain inadequately benchmarked for producing introductions that are coherent, faithful to sources, consistent, and citation-accurate. SciIG benchmarks introduction generation on 3,900 NAACL 2025 and ICLR 2025 samples using seven prompting strategies and automated, judge-based, and human evaluations. LLaMA4-Maverick is the overall front-runner across lexical overlap, content coverage, citation quality, perplexity, faithfulness, and consistency, while added related work improves nearly all metrics.

  • Problem

    Existing LLM writing systems remain inadequately benchmarked for producing introductions that are coherent, faithful to sources, consistent, and citation-accurate.

  • Method

    SciIG benchmarks introduction generation on 3,900 NAACL 2025 and ICLR 2025 samples using seven prompting strategies and automated, judge-based, and human evaluations.

  • Results

    LLaMA4-Maverick is the overall front-runner across lexical overlap, content coverage, citation quality, perplexity, faithfulness, and consistency, while added related work improves nearly all metrics.

  • Takeaways & Limitations

    LLMs can produce strong first-draft introductions from structured prompts, but generated text should not be used as-is without further attention to critical aspects.

  • Takeaways & Limitations

    The benchmark conditions generation on Title, Abstract, and Related Work, omitting methodological and results-level signals that may inform introductions.

Abstract

from arXiv · show

As researchers increasingly adopt LLMs as writing assistants, generating high-quality research paper introductions remains both challenging and essential. We introduce Scientific Introduction Generation (SciIG), a task that evaluates LLMs' ability to produce coherent introductions from titles, abstracts, and related works. Curating new datasets from NAACL 2025 and ICLR 2025 papers, we assess five state-of-the-art models, including both open-source (DeepSeek-v3, Gemma-3-12B, LLaMA 4-Maverick, MistralAI Small 3.1) and closed-source GPT-4o systems, across multiple dimensions: lexical overlap, semantic similarity, content coverage, faithfulness, consistency, citation correctness, and narrative quality. Our comprehensive framework combines automated metrics with LLM-as-a-judge evaluations. Results demonstrate LLaMA-4 Maverick's superior performance on most metrics, particularly in semantic similarity and faithfulness. Moreover, three-shot prompting consistently outperforms fewer-shot approaches. These findings provide practical insights into developing effective research writing assistants and set realistic expectations for LLM-assisted academic writing. To foster re- producibility and future research, we publicly release all code and datasets.

1 Introduction

LLM-assisted introduction writing is promising but remains difficult to benchmark for coherence, academic rigor, faithfulness, consistency, and citation accuracy. SciIG addresses this gap with new datasets and a multidimensional evaluation framework combining automated, judge-based, and human assessment.

  • LLMs must produce introductions that articulate motivation, scope, and significance while maintaining academic rigor and grounding claims in current literature.
  • Existing academic-writing systems often struggle with source faithfulness, narrative consistency, citation accuracy, and nuanced quality assessment.
  • SciIG evaluates introduction generation on lexical overlap, semantic similarity, content coverage, faithfulness, consistency, citation correctness, and narrative quality.
  • The benchmark establishes a large-scale study, a 30-metric evaluation suite, two datasets totaling 3,900 samples, and systematic few-shot prompting experiments.

2 Related Works

Prior work has explored summarization and academic text generation, but SciIG isolates single-pass introduction generation from fixed paper context without retrieval or human-in-the-loop support.

  • Multi-document summarization methods integrate information from related sources, while general-purpose LLMs have mainly been studied for single-document generation.
  • Liebling et al. (2025) study interactive introduction authoring with retrieved citations and human-written drafts, whereas SciIG generates complete introductions in one pass.
  • SciIG uses fixed Title, Abstract, and Related Work inputs without external retrieval, citation recommendation, or human-in-the-loop scaffolding.
  • Survey-generation systems such as AutoSurvey and SurveyGen target broad coverage, while SciIG emphasizes precise argumentation and contextual positioning in introductions.

3 Task

SciIG formulates introduction writing as conditional generation from a paper’s Title, Abstract, and Related Work, using 3,900 NAACL 2025 and ICLR 2025 samples paired with published introductions.

  • 3 Task: SciIG maps a Title, Abstract, and set of Related Papers to a coherent, academically styled Introduction using an LLM-conditioned task prompt.
  • 3 Task: Title, Abstract, and Related Work are selected as minimal yet sufficient inputs for establishing research territory, identifying gaps, and positioning contributions.
  • 3 Task: The benchmark excludes other sections to preserve reproducibility and scalability despite increased token costs and extraction variability.
  • 3.2 Datasets: The datasets contain 800 NAACL 2025 samples and 3,100 ICLR 2025 samples, totaling 3,900 long-form ML and NLP papers.
  • 3.2 Datasets: The five-step pipeline retrieves and parses papers, extracts and resolves citations, and enriches cited titles with Semantic Scholar metadata, achieving a 94.0% citation resolution rate.
  • 3.2 Datasets: The datasets vary substantially in introduction length and citation density, with mean introductions of 597.1 ± 228.4 words for NAACL and 771.1 ± 414.3 for ICLR.
  • 3.2 Datasets: The initial study is limited to machine learning because recruiting domain experts is difficult, and the authors performed the annotations themselves.

4 Methods and Evaluation

The study compares five language models under seven prompting strategies and evaluates generated introductions across surface similarity, semantic alignment, coverage, faithfulness, consistency, citation correctness, and narrative quality.

  • 4.1 Methods: The evaluation compares DeepSeek-V3, LLaMA 4-Maverick, Mistral-Small-3.1-24b, Gemma-3-12b-it, and GPT-4o on the SciIG datasets.
  • 4.1 Methods: Seven prompts range from minimal and structured instructions to one-, two-, and three-shot examples plus iterative AutoCoT refinement.
  • 4.2 Evaluation: The evaluation covers lexical overlap, semantic similarity, content coverage, faithfulness, consistency, citation correctness, and narrative quality using automated and LLM-as-a-Judge metrics.
  • 4.2 Evaluation: ROUGE is included for comparability with prior work rather than treated as a sole proxy for generation quality.
  • 4.2 Evaluation: Reference-based Coverage measures recovery of published-introduction content, while Reference-free Coverage measures grounding in Title and Abstract context.
  • 4.2 Evaluation: Citation precision measures generated citations matching the original list, recall measures recovered original citations, and contextual quality assesses factual representation and rhetorical placement.
  • 4.2 Evaluation: Perplexity evaluates fluency with lower values indicating more natural text, while judge scores assess grammar, flow, academic tone, and readability.

5 Results and Analysis

Across automated and LLM-as-a-Judge evaluations, LLaMA4-Maverick is the overall front-runner, while prompting strategy substantially affects introduction quality. Three-shot prompting performs best among few-shot settings, whereas AutoCoT improves some faithfulness measures at a perplexity cost.

  • LLaMA4-Maverick leads overall across lexical overlap, content coverage, citation quality, perplexity, and LLM-as-a-Judge scores.Its strongest LLM-as-a-Judge performance is reported for faithfulness, consistency, and citation quality.
  • Gemma excels in semantic alignment, while Gemma and Mistral remain competitive on faithfulness and consistency.GPT-4o and DeepSeek trail on most metrics.
  • Few-shot experiments: Three-shot prompting consistently outperforms other few-shot settings in lexical overlap, content coverage, and SummaC-ZS faithfulness.The gains suggest that additional exemplars improve lexical detail, coverage, and adherence to input facts, although improvements plateau after Two-Shot.
  • Prompt elaboration: Medium prompts achieve higher ROUGE overlap, whereas AutoCoT improves METEOR and keyphrase-based faithfulness but increases perplexity.The comparison indicates a trade-off between lexical overlap, conceptual faithfulness, and fluency-related perplexity.

6 Ablation Study

The ablation study shows that richer input context improves generated introductions. Adding related work produces the strongest results across nearly all evaluated settings, while full-paper context offers additional detail at a realism–scalability trade-off.

  • Adding related work to the title and abstract yields the best scores across nearly all metrics and prompting strategies.The T+A+Related configuration improves semantic similarity, citation precision, and citation recall over narrower inputs.
  • Citation precision rises from 0.3610 to 0.9357 and citation recall from 0.1199 to 0.5574 when related work is added.BERTScore also increases from 0.8448 to 0.8471 under the richer input configuration.
  • A full-paper context case study further improves methodological detail and result-driven framing but reinforces a realism–scalability trade-off.The study therefore motivates restricting the benchmark inputs to selected paper metadata and related work.

7 Human Evaluation

Human evaluation finds differentiated strengths across models and prompting strategies. GPT-4o generally exceeds zero-shot LLaMA-4 on several qualitative criteria, three-shot prompting performs best overall, and model outputs still miss source-specific structure when inputs omit it.

  • GPT-4o outperforms zero-shot LLaMA-4 on consistency, citation contextual quality, and contribution summary, while LLaMA-4 resists hallucinations better.
  • Three-shot prompting achieves the strongest overall human-evaluation performance in content coverage, idea flow, and clarity of methods and contributions.
  • Elaborate zero-shot prompting surpasses AutoCoT across all evaluated dimensions, especially citation quality and faithfulness.
  • Models produce fluent, faithful introductions with strong narrative flow and accurate citation usage but miss explicit research questions and contribution summaries absent from inputs.The human evaluation used 80 outputs, with single annotations because domain-expert annotators were unavailable.

8 Conclusion and Discussion

The paper concludes that LLMs can generate coherent, academically appropriate first-draft introductions from structured inputs, but these drafts are not ready for use without substantial expert revision. Their main shortcomings concern technical detail, citation contextualization, and precise contribution statements.

  • Structured prompts containing a title, abstract, and related works enable coherent, well-structured introductions aligned with academic norms.
  • Generated introductions require substantial post-editing and expert input because they omit fine-grained technical details, contextualized citations, and precise contributions.
  • LLMs are promising scholarly-writing assistants but are not yet suitable replacements for human authors.

Limitations

The benchmark excludes scientific images and diagrams from introduction generation, while conditioning generation only on Title, Abstract, and Related Work. These choices preserve a reproducible, scalable setup but omit visual and methodological or results-level signals.

  • Scientific images and diagrams are outside the benchmark’s scope because generating these abstract, non-natural visuals is challenging.The paper leaves diagram generation for future work.
  • Generation is conditioned only on Title, Abstract, and Related Work, omitting methodological and results-level signals that may inform introductions.The authors identify full Methods and Results inputs or structured signals from them as future extensions.

Ethical Considerations

The paper frames SciIG as a benchmarking tool rather than an autonomous scholarly-writing system. Its evaluation documents unresolved factual, citation, grounding, privacy, and reproducibility considerations that require human oversight and careful deployment.

  • Generated introductions can contain factual inaccuracies, improper citation use, and limited domain grounding, requiring human validation.The authors recommend review and revision by human authors to preserve accuracy, integrity, and academic standards.
  • Proprietary LLMs outperform locally deployable models in the evaluation, but transmitting research content to third-party servers raises privacy, confidentiality, and security concerns.
  • Evaluation decoding uses nucleus sampling with T = 0.7, topp = 0.95, and a fixed random seed held constant within each experiment.The controlled settings are intended to attribute observed differences to models or prompts rather than sampling variation.
  • AutoCoT generates intermediate reasoning steps internally in one pass, without external stop criteria or multiround critique–revise cycles.
  • Contextual Relevance averages cosine similarities between generated-introduction embeddings and Title, Abstract, and cited-works embeddings.Longformer representations use mask-aware mean pooling over non-padding tokens.
  • Citation metadata can remain incomplete or unresolved, so the pipeline uses endpoint fallback and similarity matching before excluding fully unresolved citations.The fallback accepts records only when title similarity reaches 0.85, and targets with entirely unresolved references are removed.
  • 94.0% of 59,619 related-paper lookups were resolved, while roughly 10% of resolved entries lacked abstracts but retained title and author metadata.

D Metric Agreement Analysis

LLM-as-a-Judge agrees moderately with formula-based metrics for several quality dimensions but diverges on citation quality. Both automated channels correlate weakly with human ratings, supporting a multichannel evaluation design.

  • LLM-as-a-Judge moderately agrees with formula-based metrics for Faithfulness (ρ=0.35), Consistency (ρ=0.45), Content Coverage (ρ=0.30), and Narrative Quality (ρ=0.34).
  • Citation quality is the exception, with negative agreement between LLM-as-a-Judge and overlap-based citation metrics (ρ= −0.19).The divergence is consistent with overlap metrics penalizing contextually valid but non-identical citations.
  • Both formula-based metrics and LLM-as-a-Judge show weak agreement with human ratings (|ρ| ≤0.20), motivating multiple evaluation channels.

E Dataset Construction Pipeline: Detailed Steps

The pipeline retrieves and parses NAACL 2025 and ICLR 2025 papers, extracts Related Works citations, and enriches cited titles with Semantic Scholar metadata. It then evaluates introductions with surface, semantic, contextual, faithfulness, consistency, fluency, and human-aligned measures, while case analyses expose limitations of restricted inputs.

  • Metadata Resolution: 94.0% of 59,619 related-paper lookups were resolved, while 5.5% remained unmatched and 150 target papers were excluded because all citations were unresolved.Among resolved entries, 90.0% had titles, abstracts, and authors; 10.0% had titles and authors but lacked abstracts.
  • Pipeline Steps: The pipeline retrieves accepted conference PDFs, converts them into structured JSON, extracts Related Works citations, and enriches cited titles with abstracts and author details.PDF parsing uses grobid2json and citation extraction links in-text citation patterns to reference entries before Semantic Scholar enrichment.
  • Evaluation Metrics: Evaluation combines ROUGE, BLEU, METEOR, BERTScore, BLEURT, contextual relevance, keyphrase faithfulness and consistency, perplexity, LLM judging, and human evaluation.Faithfulness compares generated keyphrases with Title and Abstract context, while consistency compares them with the ground-truth introduction.
  • Qualitative Findings: Annotators found LLaMA-4 Maverick and GPT-4o fluent, faithful, narratively coherent, and citation-accurate, but weaker at recovering research questions and paper-specific structure absent from inputs.Contribution summaries often resembled conclusions, and missing methodological nuances limited consistency with the published introductions.
  • Context Trade-off: Full-paper context enabled GPT-5.1 to recover research questions, summarize methodology, and incorporate findings, but restricted inputs preserve scalability, reproducibility, and realistic writing-time conditions.Full-context evaluation increases token usage and depends on reliable extraction and alignment across heterogeneous PDFs.
Loading 2508.14273v3…