Source-linked AI summary
FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation
Sebastian Hofstätter, Jiecao Chen, Karthik Raman, Hamed Zamani
TL;DR
Retrieval-augmented generation provides provenance alongside answers, but its complexity and inference costs motivate a more efficient approach. FiD-Light compresses the information passed from encoder to decoder and adds source-pointer reranking; it improves the latency-effectiveness frontier and achieves new combined retrieval-generation state-of-the-art results on six KILT tasks.
Problem
Retrieval-augmented generation offers provenance and updateable knowledge, but these systems have greater complexity and training and inference costs.
Source: §1 · p. 1Method
FiD-Light compresses the encoded information passed to the decoder, while FiD-LightSP uses textual source pointers to rerank retrieved passages.
Source: §1 · p. 2Results
Takeaways & Limitations
The reported results indicate that highest effectiveness does not always require scaling up, with larger T5 backbones yielding Pareto-optimal results on six KILT tasks.
Source: §5 · p. 9The discrete selection of top-R+ passages departs from pairwise reranking and introduces biases about expected R+ distributions that the paper does not study.
Source: §2.1 · p. 4
Abstract
from arXiv · showhide
Retrieval-augmented generation models offer many benefits over standalone language models: besides a textual answer to a given query they provide provenance items retrieved from an updateable knowledge base. However, they are also more complex systems and need to handle long inputs. In this work, we introduce FiD-Light to strongly increase the efficiency of the state-of-the-art retrieval-augmented FiD model, while maintaining the same level of effectiveness. Our FiD-Light model constrains the information flow from the encoder (which encodes passages separately) to the decoder (using concatenated encoded representations). Furthermore, we adapt FiD-Light with re-ranking capabilities through textual source pointers, to improve the top-ranked provenance precision. Our experiments on a diverse set of seven knowledge intensive tasks (KILT) show FiD-Light consistently improves the Pareto frontier between query latency and effectiveness. FiD-Light with source pointing sets substantial new state-of-the-art results on six KILT tasks for combined text generation and provenance retrieval evaluation, while maintaining reasonable efficiency.
1 INTRODUCTION
Retrieval-augmented generation can provide attributable answers from updateable knowledge bases, but FiD’s high inference costs motivate reducing latency without sacrificing effectiveness. FiD-Light compresses information passed to the decoder and adds source-pointer reranking; its KILT results improve efficiency-effectiveness tradeoffs and set new combined-score bests on six tasks.
1 INTRODUCTION
- FiD synthesizes answers from separately encoded retrieved passages, but its long decoder input makes inference costly; simply reducing passage count also substantially reduces effectiveness.Retrieval-augmented generation offers attributable answers and updateable knowledge bases, alongside higher system complexity and training and inference costs.Source: §1 · pp. 1–2
- FiD-Light compresses the encoded vectors passed to the decoder, allowing 40 passages with decoding time substantially reduced relative to FiD.FiD-Light’s decoding time is described as on par with encoding time, with lower total latency than FiD using 10 passages; source pointers add reranking for provenance retrieval.Source: §1 · p. 2
- Simple supervised retrieval training outperforms a zero-shot retrieval baseline and moves FiD-LightSP toward a realistic oracle-retriever ceiling.The paper notes that downstream result quality is strongly bound by retriever recall.Source: §1 · p. 2
- FiDSP loses effectiveness as input passages are reduced, while FiD-LightSP maintains passage precision at lower latency.The source-pointer workflow trains on relevant retrieved passages, then uses pointers to rerank rather than discard the remaining retrieved list.Source: §1 · pp. 2–3
- FiD-LightSP is Pareto optimal on six of seven tested tasks, retaining most of FiDSP’s maximum effectiveness while running 3× faster.Larger T5 backbones further improve efficiency while outperforming the initial FiDSP baseline in efficiency.Source: §1 · p. 3
- FiD-LightSP improves previous bests by +11.1 K-EM on HotpotQA, +7.5 K-EM on NQ, +5.8 K-EM on TriviaQA, +10.8 K-AC on zsRE, +0.5 K-AC on T-REx, and +6.0 K-AC on FEVER.These are combined KILT-score results across six tasks.Source: §1 · p. 3
2 BACKGROUND AND RELATED WORK
FiD independently encodes each retrieved passage with the query, then concatenates the encoded representations for decoding. The section outlines FiD’s provenance and efficiency limitations, FiD-Ex’s textual passage markers, and related approaches to retrieval-enhanced modeling and efficient generation.
2.1 FID (FUSION IN DECODER) WITH EXPLANATIONS
- FiD concatenates independently encoded query-passage representations and feeds them to a decoder to generate one output sequence.The retriever is independently trained and supplies candidate passages.Source: §2.1 · p. 3
- FiD’s text-only output omits the passages used, while its O(n ∗(|q| + |p|)) decoder input makes autoregressive decoding inefficient.The expected output is dozens of tokens, while the decoder input can contain thousands.Source: §2.1 · p. 4
- FiD-Ex adds passage indices to inputs and relevant indices to training targets; its discrete selection can induce biases in expected passage counts.Lakhotia et al. (2021) adapted sentence markers to passage indices for a more general formulation [p27].Source: §2.1 · p. 4, §2.2 · p. 4
2.2 RELATED WORK
- Related work covers efficient generation, retrieval-enhanced models, and FiD adaptations, including early exiting (Schuster et al., 2022), LongT5 (Guo et al., 2021), and relevance prediction (Asai et al., 2021).Other cited approaches include task-specific retrieval-enhanced models and systems that retrieve supporting documents for FiD.Source: §2.2 · p. 4
3 FID-LIGHT WITH SOURCE POINTERS
FiD-Light reduces the number of encoded vectors passed to the decoder, while its source-pointer workflow re-ranks retrieved passages. The section describes how these changes address decoding efficiency and the robustness of provenance selection.
3 FID-LIGHT WITH SOURCE POINTERS
- Reducing each encoded query-passage pair to k vectors changes decoder input length from O(n ∗(|q| + |p|)) to O(n ∗k), with k ≪ |q| + |p|.Experiments use compression ratios from 6x to 384x; the implementation takes the first k vectors of each sequence.Source: §3 · p. 5
- FiD-Light treats source pointers as a parallel passage re-ranking task, allowing their output to be processed further rather than treated as explanations.Directly using pointer selections can produce instability in the number of returned passages.Source: §3 · p. 5
- The workflow moves pointer-selected passages to the top of the initial ranked list; when several are selected, it preserves model-output order.With graded relevance annotations, training can follow the grades; with binary relevance, selected passages retain their initial candidate ranking.Source: §3 · p. 5
- Table 1 reports downstream KILT scores for FiD-Light using different retrievers, presenting results under both realistic and oracle evaluation settings.Source: Table 1 · p. 6
4 RESULTS
The results examine retriever training, source-pointer re-ranking, efficiency–effectiveness tradeoffs, and performance against KILT leaderboard systems. FiD-LightSP combines efficiency gains with strong combined retrieval-and-generation results across tasks.
4.1 INFLUENCE OF THE RETRIEVER
- In-domain retriever fine-tuning consistently improves results over a zero-shot retriever, while training with all known relevant passages does not significantly change downstream performance.With sparse relevance annotations, the authors favor a more realistic oracle that adds known relevant passages to retrieved candidates over one containing only known relevant passages; it shows roughly half the headroom.Source: §4.1 · p. 6
4.2 SOURCE POINTER ROBUSTNESS
- Because FiD-LightSP under-produces three passages on TriviaQA and FEVER, promoting selected passages in the ranking yields statistically significant gains across configurations.Its output-count distribution closely matches training, which excludes relevant passages that were not retrieved; this does not establish that selected passages are correct. At reduced input counts, FiDSP loses effectiveness while FiD-LightSP nearly maintains it.Source: §4.2 · pp. 6–7
4.3 EFFICIENCY - EFFECTIVENESS TRADEOFF
- Across KILT tasks, FiD-LightSP yields statistically significant efficiency and effectiveness improvements and can cut query latency by more than 2× while improving effectiveness with a larger backbone.With T5-Base, open-domain QA shows an effectiveness drop versus the strongest baseline; FEVER and slot-filling tasks show smaller reductions when encoded vectors are reduced.Source: §4.3 · pp. 8–9
4.4 COMPARISON TO RELATED WORK
- T5-Large and T5-XL set new KILT-score state of the art on six of seven tasks; HotpotQA improves by 11.1 KILT-EM points (61%).The submitted systems used a single checkpoint across tasks.Source: Table 3 · p. 9, §4.4 · p. 9
5 CONCLUSION
FiD-Light compresses information passed to the decoder to reduce inference latency, with only a modest effectiveness reduction that larger T5 backbones can alleviate. FiD-LightSP achieves new state-of-the-art combined retrieval and generation results on six KILT tasks.
5 CONCLUSION
- FiD-Light compresses decoder input to reduce inference latency, with larger T5 backbones alleviating the modest effectiveness reduction and producing Pareto-optimal results on six KILT tasks.The multi-task system achieved new state-of-the-art combined retrieval and generation results on six KILT tasks.Source: §5 · p. 9
- The results suggest researchers need not always scale up to achieve the highest effectiveness.Source: §5 · p. 9
- The reported leaderboard comparison is against previous methods on the public leaderboard.Source: §5 · p. 9
A EXPERIMENT DESIGN
The experiments use T5-based FiD(-Light) models and seven KILT tasks, evaluating text generation together with provenance retrieval. KILT-scores count a text metric only when the query’s R-Precision is 1.
Implementation
- Models use T5 v1.1 backbones, up to 40 retrieved passages for training, and beam search with a size-4 beam for decoding.Inputs are capped at 384 combined query-and-passage tokens, outputs at 64 tokens; T5-XL is trained for 30K steps, while other models are trained for 50K.Source: A EXPERIMENT DESIGN · p. 13
Datasets
- Experiments cover seven KILT tasks, using filtered training and passage sets and the original evaluation sets.The tasks are HotpotQA, TriviaQA, Natural Questions, T-REx, Zero Shot RE, FEVER, and Wizard of Wikipedia.Source: A EXPERIMENT DESIGN · p. 13
Evaluation
- KILT-scores combine a text metric with R-Precision, counting the text score only when R-Precision equals 1.The reported dev results use means and 95% confidence intervals across the last 10 checkpoints; leaderboard comparisons use one checkpoint, and significance tests against other methods were unavailable.Source: A EXPERIMENT DESIGN · p. 13
- Table 5 reports passage-level Recall@40 for the GTR dense retrieval model on KILT dev sets, alongside ΔT, the relative difference from the respective training-set Recall@40; lower ΔT is better.Source: Table 5 · p. 14
B DENSE RETRIEVAL TUNING RESULTS
Fine-tuned dense retrieval improves Recall@40 over zero-shot retrieval across all tasks, while learning-rate choices affect train–test distribution shift. The authors proceed with a 0.005 learning rate for its overall results and low shifts.
Retrieval training
- The retriever is fine-tuned on query, relevant-passage, and sampled-negative triples, but overfitting could shift generator training data away from expected retrieval performance.The negative passages are sampled from the top 100 zero-shot GTR rankings; the retriever is trained for 100K steps.Source: B DENSE RETRIEVAL TUNING RESULTS · p. 14
Retrieval results
- R@40 gains over the zero-shot baseline ranged from 0.13 to 0.20 absolute across all tasks.The comparison is on the dev set; high learning rates, especially 0.1 and 0.05, show high train–test differences.Source: B DENSE RETRIEVAL TUNING RESULTS · p. 14
Learning-rate study
- Open-domain QA reached optimal dev results with higher learning rates, while other tasks favored lower rates; 0.005 gave the best overall results with low distribution shifts.The authors did not further tune retrieval-training composition, describing it as a secondary contribution with small differences relative to the zero-shot margin.Source: B DENSE RETRIEVAL TUNING RESULTS · p. 14
C DETAILED RELATED WORK COMPARISONS
FiD-LightSP combines competitive text generation with strong passage ranking, producing strong combined KILT scores. Its ranking results improve on prior state of the art, while text generation remains competitive with larger, slower systems.
Comparison context
- The comparison reports text-generation scores and R-Precision separately; prior high-performing FiD systems use 50 or 100 passages and submit no supporting passages.Those systems focus on text generation and use T5-XL or T5-XXL.Source: C DETAILED RELATED WORK COMPARISONS · p. 15
Leaderboard comparison
- FiD-LightSP sets no new text-generation state of the art but remains competitive with the leaderboard’s largest and slowest entries.Its models span multiple T5 and encoded-vector sizes; leaderboard R-Precision is computed at document level after converting passage ranks to document ranks.Source: C DETAILED RELATED WORK COMPARISONS · p. 15
- FiD-LightSP’s competitive text generation plus state-of-the-art ranking results explain its strong combined KILT scores.The ranking results are reported for models beginning with T5-Base, even though they re-rank only 40 passages.Source: C DETAILED RELATED WORK COMPARISONS · p. 15
D FAILURE ANALYSIS
The failure analysis separates text-answer and provenance outcomes across KILT tasks. Failure patterns vary by task, and noisy relevance and answer annotations limit how confidently failures can be interpreted.
Failure patterns
- Figure 6 separates raw outputs into four joint text/provenance outcomes; QA tasks more often fail on both, and HotpotQA has the highest double-failure rate.The analysis compares FiD-LightSP with T5-Base and T5-XL backbones.Source: D FAILURE ANALYSIS · p. 16
Interpretation limits
- Failure categories are difficult to interpret because most KILT tasks mark only 1–2 relevant passages and provide few answer-text variations.The authors excluded Wizard of Wikipedia because it has no exact text matches; they say confirming whether double failures indicate true model failures would require thorough annotation.Source: D FAILURE ANALYSIS · p. 16
Qualitative inspection
- An interactive explorer presents 10 random results per category and task, allowing targeted qualitative inspection without hand-picked examples.Each refresh generates a new random sample.Source: D FAILURE ANALYSIS · p. 16