Source-linked AI summary
Data Citation for Large Language Models: A Challenge
Gianmaria Silvello
TL;DR
The paper addresses the open problem of citing data in LLM outputs, beyond document-level citation grounding, so verification, provenance, and credit remain supported. It organizes the problem into three research directions—training attribution, inference-time data citation, and KG-fact citation—and calls for coordinated work across several fields. The central conclusion is that these directions require citation models, provenance mechanisms, credit distribution, evaluation benchmarks, and standards designed for structured data.
Problem
LLM citation research largely focuses on textual-document verification, leaving unresolved how to cite data for verifiability, provenance tracking, and fair credit.
Method
The paper frames data citation as three research directions: training-data attribution, inference-time citation of datasets and query results, and citation of knowledge-graph facts.
Results
The paper identifies data citation for LLMs as an open, technically harder challenge requiring unified models, provenance-aware architectures, credit mechanisms, benchmarks, and standards.
Takeaways & Limitations
Progress depends on joint work across database, information retrieval, knowledge representation, and artificial intelligence communities.
Abstract
from arXiv · showhide
Large language models increasingly mediate access to information, and a growing body of work asks whether they cite the sources behind their outputs. That work treats citation as a verification device and applies it to textual documents. Scholarly citation serves two further functions, credit and provenance, and it applies to data as much as to text. This paper argues that data citation for large language models is an open challenge, distinct from document-level citation grounding and harder to solve. We ask how such models should cite data so that outputs stay verifiable, provenance stays traceable, and credit reaches data creators and curators. We set out three research directions. Training data attribution has to turn influence estimates into references for corpora absorbed into model parameters. Data citation at inference time has to identify datasets, subsets, and query results at the right granularity and fixity. Citing knowledge graph facts has to define what a reference to a single triple denotes and how credit propagates along provenance. Progress on all three depends on joint work across the database, information retrieval, knowledge representation, and artificial intelligence communities.
1 Motivation
LLMs increasingly mediate information access through diverse sources, while existing citation work focuses mainly on verifying textual documents. This paper asks how LLMs should cite data to preserve verifiability, provenance, and fair credit.
- 1 Motivation: LLMs increasingly serve as interfaces for information access, synthesis, and decision support.Their outputs may draw on training corpora, retrieved documents, knowledge graphs, and tool-augmented workflows.
- 1 Motivation: An answer’s origin may be a dynamic process spanning several sources rather than one retrieved document.
- 1 Motivation: Scholarly citation supports verification, credit, and provenance, not verification alone.
- 1 Motivation: Current LLM citation efforts largely target verification for textual documents while neglecting training datasets, structured context, and knowledge graphs.
- 1 Motivation: The paper asks how LLMs should cite data to ensure verifiability, provenance tracking, and fair credit attribution to data creators and curators.
- 1 Motivation: Data citation for LLMs is technically harder and socially more consequential than document-level citation grounding.
2 Background
Data citation frameworks recognize data as scholarly objects supporting verification, provenance, and credit, but important problems remain for evolving and structured data. Prior LLM citation research mainly addresses textual grounding, leaving data citations and KG-fact credit unresolved.
- 2 Background: Data citation principles treat data as first-class scholarly objects with human-understandable, machine-actionable citations.These citations are intended to support credit, provenance, and verifiability.
- 2 Background: Persistent challenges include citing subsets and query results over evolving databases, maintaining fixity under updates, and generating references at the right granularity.
- 2 Background: Credit-distribution methods propagate credit through relational structures and provenance chains, including indirect and transitive credit.
- 2 Background: LLM citation research distinguishes parametric training content from non-parametric retrieved content, but attribution methods often misidentify supporting documents.
- 2 Background: Existing work evaluates textual citation grounding and faithfulness, while data citations remain largely absent from evaluation landscapes.
- 2 Background: Although knowledge graphs increasingly ground LLMs, citation of the KG facts used and credit to their contributors remain unaddressed.
3 Challenges and Research Directions
The paper identifies three interconnected data-citation challenges for LLMs: attributing training influence, citing inference-time data at suitable granularity and fixity, and referencing KG facts with provenance-aware credit. It proposes coordinated development of citation models, provenance architectures, credit mechanisms, benchmarks, and standards.
- 3 Challenges and Research Directions: The paper identifies three interconnected challenges: training-data attribution, inference-time data citation, and citation of knowledge-graph facts.
- Training Data Attribution: Training-data citation must map outputs to influential training items with contribution weights, but scalable and rigorous methods remain open.The challenge includes referencing specific tables, columns, or tuples in structured training data.
- Training Data Attribution: Training-data attribution is realistically feasible only for open models with documented training corpora.
- Data Citation in LLMs: Inference-time data citations must identify datasets, subsets, and query results with appropriate granularity, metadata, and fixity.The citable object may be a query and its result over a versioned source rather than a static document.
- Citing Knowledge Graph Facts: KG-fact citation must define what a single triple denotes, preserve provenance, and distribute credit across contributing sources.Shared upstream sources make credit distribution a network-propagation problem over provenance graphs.
- 3 Challenges and Research Directions: A comprehensive solution requires unified citation models, provenance-aware architectures, scalable credit distribution, data-level benchmarks, and machine-actionable standards.
- 3 Challenges and Research Directions: In RAG, provenance enables links from generated spans to retrieved data, citation granularity enables credit distribution, and benchmarks enable evaluation.The stages are interdependent rather than merely sequential.
- 3 Challenges and Research Directions: Inference-time data citation is presented as the first and most immediately tractable challenge because sources are available and can be anchored to versioned artifacts or producing queries.RAG is identified as the natural entry point.