PaperlayerView exact revision on arXiv

Source-linked AI summary

LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA

Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, Juanzi Li

arXiv:2409.02897v3cs.CL

TL;DR

Long-context LLMs often lack precise citations, making their outputs difficult to verify. LongCite introduces LongBench-Cite, the CoF data-construction pipeline, and LongCite-45k to train models for sentence-level citations; LongCite-8B and LongCite-9B achieve higher citation quality than GPT-4o.

  • Problem

    Long-context LLMs typically lack citations to specific context snippets, making their outputs difficult to verify despite ongoing hallucination and faithfulness problems.

  • Method

    The paper introduces LongBench-Cite, uses CoF to construct LongCite-45k, and fine-tunes long-context models on this citation-focused dataset.

  • Results

    LongCite-8B and LongCite-9B generate accurate responses with fine-grained citations in one pass and outperform GPT-4o in citation F1.

  • Takeaways & Limitations

    Citation-focused supervised fine-tuning improves citation generation and further improves response correctness for long-context question answering.

  • Takeaways & Limitations

    On GovReport, proprietary models’ coarser citations create an unfair advantage in citation recall and precision evaluation.

Abstract

from arXiv · show

Though current long-context large language models (LLMs) have demonstrated impressive capacities in answering user questions based on extensive text, the lack of citations in their responses makes user verification difficult, leading to concerns about their trustworthiness due to their potential hallucinations. In this work, we aim to enable long-context LLMs to generate responses with fine-grained sentence-level citations, improving their faithfulness and verifiability. We first introduce LongBench-Cite, an automated benchmark for assessing current LLMs' performance in Long-Context Question Answering with Citations (LQAC), revealing considerable room for improvement. To this end, we propose CoF (Coarse to Fine), a novel pipeline that utilizes off-the-shelf LLMs to automatically generate long-context QA instances with precise sentence-level citations, and leverage this pipeline to construct LongCite-45k, a large-scale SFT dataset for LQAC. Finally, we train LongCite-8B and LongCite-9B using the LongCite-45k dataset, successfully enabling their generation of accurate responses and fine-grained sentence-level citations in a single output. The evaluation results on LongBench-Cite show that our trained models achieve state-of-the-art citation quality, surpassing advanced proprietary models including GPT-4o.

1 INTRODUCTION

LongCite addresses the difficulty of verifying long-context LLM outputs by benchmarking citation quality, constructing fine-grained citation data, and training models that generate responses with sentence-level citations.

  • Existing retrieval-based and post-hoc citation methods can lose context, add latency, or cite whole pages and coarse chunks rather than precise evidence.These limitations make users locate supporting evidence themselves during verification.
  • LongBench-Cite reveals that current long-context LLMs produce many irrelevant, incomplete, or overly coarse citations.Generating citations on the fly also generally lowers response correctness relative to vanilla long-context QA.
  • CoF automatically constructs long-context QA instances with precise sentence-level citations for supervised fine-tuning.The pipeline begins by generating queries and answers from long text, then uses the answer to retrieve relevant context.
  • LongCite-8B and LongCite-9B outperform GPT-4o in citation F1 while producing twice finer citation granularity.Their 8B/9B models improve citation F1 over GPT-4o by 6.4%/3.6%, respectively.
  • The authors construct LongCite-45k as a large-scale supervised fine-tuning dataset for long-context question answering with citations.The dataset is produced using the CoF construction method.
  • LongCite-8B and LongCite-9B generate accurate responses and fine-grained citations in one pass, and citation-focused fine-tuning further improves response correctness.The models support context windows up to 128,000 tokens.

2 LONGBENCH-CITE: BENCHMARK LONG-CONTEXT QA WITH CITATIONS

LongBench-Cite evaluates long-context question answering with citations across bilingual, single- and multi-document QA, summarization, and real-world queries. It measures both answer correctness and citation quality, revealing that current models often struggle with relevant, sufficiently fine-grained evidence while citations can also reduce QA correctness.

  • Data Collection: LongBench-Cite covers bilingual single-document QA, multi-document QA, summarization, and real-world long-context queries, requiring long-form responses with citations.Its sources include MultiFieldQA-en/zh, HotpotQA, DuReader, GovReport, and LongBench-Chat queries with contexts ranging from 10k to 100k in length.
  • Evaluation Metrics: Citation quality measures whether cited snippets fully support statements, avoid irrelevant evidence, and remain fine-grained.The benchmark evaluates citation recall, citation precision, citation F1, and citation length.
  • Evaluation Metrics: Citation length is the average token count of cited snippets, with lower values indicating finer-grained citations and helping prevent citing the whole context for every statement.This metric also captures citation granularity beyond citation F1.
  • Benchmarking Results: Open-source LLMs, especially smaller models, trail proprietary models in citation quality despite achieving similar correctness.The authors attribute this gap to weaker instruction-following, in-context learning, and evidence-searching abilities.
  • Benchmarking Results: Proprietary models still show limited citation quality, while generating answers and citations together often reduces correctness relative to vanilla long-context QA.Citation F1 is around 0.5 on LongBench-Chat and HotpotQA, GPT-4o’s cited snippets average 220 tokens, and most correctness ratios are below 100%.

3 COF: AUTOMATIC SFT DATA CONSTRUCTION FOR LQAC

CoF is a post-hoc retrieval-and-extraction pipeline that constructs long-context QA data with precise sentence-level citations, producing LongCite-45k for LQAC training.

  • 3.1 PIPELINE DETAILS: CoF generates QA pairs, retrieves relevant context chunks, creates chunk-level citations, extracts sentence-level evidence, and filters insufficiently cited instances.The pipeline first constructs answers, then adds citations to preserve answer correctness while refining citation granularity.
  • 3.1 PIPELINE DETAILS: Aggregating retrieved chunks for all answer statements reduces LLM calls and improves citation recall over individually generating citations.
  • 3.2 EFFECTIVENESS OF COF: CoF preserves vanilla long-context QA answers with 100% correctness while achieving the highest citation F1 among post-hoc strategies.Its focused evidence search and sentence-level extraction produce precise citations with relatively short citation length.
  • 3.3 LONGCITE-45K: A LARGE-SCALE SFT DATASET FOR LQAC: LongCite-45k contains 44,600 filtered LQAC instances generated from 50,000 documents spanning nine domains and lengths up to 128k tokens.Each instance pairs a long document and query with an answer containing sentence-level citations.

4 LONGCITE: TEACH LONG-CONTEXT LLMS TO GENERATE CITATIONS

The authors fine-tune long-context base models on citation-augmented data and evaluate citation quality, correctness, and their relationship. LongCite-8B and LongCite-9B achieve leading citation quality while also improving response correctness.

  • 4.2.1 MAIN RESULTS: LongCite-8B and LongCite-9B achieve the best citation quality, improving overall citation F1 over GPT-4o by 6.4 and 3.6 points, respectively.Their citations are also shorter than proprietary-model and chunk-level citations, indicating finer granularity.
  • 4.2.1 MAIN RESULTS: On GovReport, proprietary models retain a performance advantage that the authors attribute to their coarser citations, which receive an evaluation benefit.GLM-4 and LongCite-9B have citation lengths of 188 and 86, respectively, on GovReport.
  • 4.2.1 MAIN RESULTS: SFT with citation information consistently improves correctness over vanilla long-context SFT, increasing LongCite-8B/9B correctness by 16%/28%.The authors attribute gains mainly to better evidence locating, reduced hallucination, and more uniform context use.
  • 4.2.2 ABLATION STUDIES: LongCite models outperform standard long-context SFT on LQAC, whereas LongSFT-9B often produces nonconforming or absent citations.
  • 4.2.2 ABLATION STUDIES: Filtering improves citation quality, while training on post-RAC-S citations performs worse because discontinuous citation numbers make subsequent training difficult.
  • 4.2.3 ANALYSIS: Higher correctness is associated with higher citation F1 for LongCite-9B, indicating a mutually promoting relationship between response correctness and citation quality.

5 RELATED WORKS

Related work covers long-context LLMs and citation-aware question answering. Existing approaches rely heavily on automatically synthesized QA data, retrieval, or post-hoc processing, creating faithfulness, information-loss, latency, and citation-granularity concerns.

  • LONG-CONTEXT LLMS: Long-context LLMs extend context windows through continued pretraining and long-context QA alignment, but automatically synthesized QA can lack faithful answers and permit hallucinations.
  • QUESTION ANSWERING WITH CITATIONS: Citation-aware QA commonly uses retrieval-augmented generation or post-hoc processing, which can lose information or add latency in long-context settings.These methods also tend to provide citations at coarse granularity.

6 CONCLUSION

The paper introduces a benchmark, a citation-construction pipeline, and a large SFT dataset, then trains models that generate accurate answers with fine-grained citations in one pass. Experiments and human evaluation support the approach’s effectiveness for LQAC.

  • 6 CONCLUSION: LongBench-Cite exposes limited current performance, while CoF constructs LongCite-45k with precise sentence-level citations for LQAC training.
  • 6 CONCLUSION: Training LongCite-8B and LongCite-9B enables accurate responses and fine-grained citations in one pass, with experiments and human evaluation supporting the approach.

A MODEL CARDS

The evaluated models are listed in Table 8.

  • Table 8 lists the models evaluated in this work.

B CASE STUDY

Three case studies are presented to interpret improvements in correctness.

  • Three case studies interpret the improvement in correctness.The detailed interpretation appears in Section 4.2.1.

D EVALUATION COST

The evaluation uses automated prompts for correctness, citation recall, and citation precision, while case studies examine how citation-aware models use long contexts. A GPT-4o evaluation run costs about $4 for correctness and $25 for citation quality.

  • $4/$25 is the reported GPT-4o evaluation cost for correctness/citation quality on LongBench-Cite.
  • Case study: LongCite-9B uses more middle-context information than LongSFT-9B, producing a more detailed response.
  • Case study: Citation numbers help LongCite-9B cover context more uniformly, whereas LongSFT-9B focuses on the front and ignores later parts.
  • Evaluation prompts: Evaluation prompts assess response correctness, citation recall, and citation precision.
  • CoF prompts: CoF prompts generate diverse questions, add citations to answers, identify supporting chunks, and extract sentence-level citations.
Loading 2409.02897v3…