Source-linked AI summary
Condenser: a Pre-training Architecture for Dense Retrieval
Luyu Gao, Jamie Callan
TL;DR
Dense bi-encoders are efficient but difficult to train because standard language-model attention is not prepared to aggregate text into a single vector, especially with limited data. Condenser pre-trains toward this structure by conditioning language-model prediction on dense representations, improving performance across similarity and retrieval tasks while retaining standard fine-tuning capacity.
Problem
Bi-encoders require sophisticated training and lose performance in low-data settings because standard language models lack attention structure for aggregating information into dense representations.
Method
Condenser is a Transformer pre-training architecture that conditions language-model prediction on dense representations and removes its auxiliary head during fine-tuning.
Results
Condenser improves over standard language models across sentence similarity, question-answering retrieval, and web-search retrieval, with comparable low-data performance to task-specific pre-trained models.
Takeaways & Limitations
Structural readiness is a fundamental property of easy-to-train bi-encoders, and general language-model pre-training may provide a lightweight alternative to complicated training techniques.
Takeaways & Limitations
The paper focuses on passage retrieval and defers discussion of long-document retrieval.
Abstract
from arXiv · showhide
Pre-trained Transformer language models (LM) have become go-to text representation encoders. Prior research fine-tunes deep LMs to encode text sequences such as sentences and passages into single dense vector representations for efficient text comparison and retrieval. However, dense encoders require a lot of data and sophisticated techniques to effectively train and suffer in low data situations. This paper finds a key reason is that standard LMs' internal attention structure is not ready-to-use for dense encoders, which needs to aggregate text information into the dense representation. We propose to pre-train towards dense encoder with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. Our experiments show Condenser improves over standard LM by large margins on various text retrieval and similarity tasks.
1 Introduction
Dense bi-encoders offer efficient single-vector text representations but are harder to train than cross-encoders, especially with limited data. Condenser addresses this difficulty by pre-training language models toward dense-encoder structure and improves retrieval performance.
- Dense encoders represent text sequences as single vectors for efficient comparison and retrieval.
- Bi-encoders require sophisticated training methods and suffer substantial performance losses in low-data settings.
- Standard language models lack attention structure trained to aggregate sophisticated information into one dense representation.
- Condenser pre-trains language models toward bi-encoder structure by conditioning language-model prediction on dense representations.
- Under low-data setups, Condenser improves over standard language models and achieves performance comparable to strong task-specific pre-trained models.
- With large training data, Condenser retrievers outperform previous models trained with complicated techniques using one round of negative mining.
2 Related Work
Related work establishes dense retrieval as an efficient alternative to cross-encoder retrieval and develops pre-training and fine-tuning strategies for stronger bi-encoders. Condenser differs by targeting bi-encoder structure through general language-model pre-training.
- Transformer bi-encoders enable efficient sentence similarity and dense retrieval through vector representations.
- Dense retrieval compares encoded query and document vectors with inner products and supports millisecond-level online retrieval from pre-encoded indexes.
- Inverse Cloze Task and related pre-training methods improve bi-encoders, while models without pre-training fail under low-data retrieval settings.
- Condenser provides an architecture-based explanation for bi-encoder learning difficulty using general language-model pre-training.
- Later dense-retrieval work improves end performance through hard-negative mining and other costly fine-tuning techniques.
- This paper focuses on task-specific fine-tuning of the entire model rather than universal sentence-embedding features.
3 Method
Condenser modifies Transformer language-model pre-training so dense representations actively aggregate information across layers. Its auxiliary head guides this structure during pre-training, then is removed so fine-tuning uses the standard-capacity encoder backbone.
- 3.1 Preliminaries: Transformer encoders refine each token representation through stacked self-attentive blocks using sequence-wide context.
- 3.1 Preliminaries: Masked language-model pre-training predicts masked tokens from their final representations, typically alongside a prepended CLS token.
- 3.2 Issues with Transformer Encoder: In standard language models, CLS remains relatively inactive through middle layers and gains broad attention only near the final layer.
- 3.3 Condenser: Condenser conditions language-model pre-training on dense representations to establish structural readiness for bi-encoder fine-tuning.
- 3.3 Condenser: The architecture separates early and late backbone layers from Condenser-head layers and connects early outputs directly to the head.
- 3.3 Condenser: The late CLS must aggregate newly generated backbone information, while skip connections let it focus on global input meaning.
- 3.3 Condenser: The Condenser head is used only during pre-training and is dropped during fine-tuning, leaving an encoder with identical capacity to a comparable Transformer.
- 3.4 Condenser from Transformer Encoder: When initialized from a pre-trained Transformer, Condenser adds a backbone MLM constraint to prevent the randomly initialized head from corrupting backbone weights.
4 Experiments
Experiments evaluate Condenser on sentence similarity, open-domain question answering, and web search under low- and full-data settings. Across these settings, Condenser remains competitive with or outperforms standard and task-specific alternatives, while requiring less elaborate training.
- Experimental setup: Experiments cover STS-b, Wikipedia Section, Natural Question, TriviaQA, and MS-MARCO, using both low-data and full-training settings.The evaluation includes sentence-level similarity, open QA passage retrieval, and web-search retrieval.
- Sentence similarity: With only 500 STS-b training pairs, NLI-pre-trained BERT and Condenser outperform the best Universal Sentence Encoder baseline.NLI-pre-trained BERT and Condenser consistently outperform standard BERT, with larger margins as training data decreases.
- Sentence similarity: On Wiki Section, BERT and Condenser achieve almost identical results and retain effectiveness without NSP pre-training.Using 1K training examples causes only about a 10% accuracy drop relative to training with all data.
- Open QA retrieval: At 1K QA training examples, ICT and Condenser outperform vanilla BERT, dropping less than 10% on Top-20 Hit and less than 5% on Top-100 versus full-data training.ICT and Condenser show comparable performance; ICT is slightly better on NQ, while Condenser is better on TQA.
- Open QA retrieval: On full-data NQ, Condenser performs best for Top-20 and comes within 0.1 of RocketQA for Top-100 without its sophisticated, costly training pipeline.On TQA, Condenser outperforms all other lexical and dense systems, including systems using query expansion.
- Web search retrieval: On MS-MARCO, Condenser outperforms prior systems by large margins and slightly exceeds RocketQA without complex training techniques or a heavier model or retrieval process.Against RocketQA variants, Condenser with batch size 64 and BM25 negatives performs better than RocketQA with batch size 8192, and is outperformed only by the external-data variant.
5 Attention Analysis
Attention analysis compares CLS attention entropy across BERT, ICT, and Condenser before and after fine-tuning. BERT changes substantially, whereas ICT and Condenser retain attention structures that require only small adjustments.
- CLS attention entropy is used to characterize attention patterns across pre-trained and fine-tuned BERT, ICT, and Condenser.The analysis measures how CLS attends across tokens to compare internal attention behavior.
- BERT shows a drastic change in CLS attention between pre-training and fine-tuning, indicating substantial internal restructuring for bi-encoder use.This supports the claim that typical Transformer encoder LMs are not structurally ready for bi-encoder fine-tuning.
- ICT and Condenser show only small attention changes between pre-training and fine-tuning, retaining structures suited to effective bi-encoder use.Their attention structures require smaller modifications than BERT's during fine-tuning.
6 Conclusion
The conclusion argues that standard pre-trained LMs lack attention structures ready for bi-encoders, and introduces Condenser to establish this readiness during pre-training. Across several tasks and data regimes, Condenser offers comparable low-data performance and a lightweight alternative to more elaborate training techniques.
- Typical pre-trained LMs lack internal attention structures ready to condense information into a single bi-encoder vector.
- Condenser establishes bi-encoder structural readiness during LM pre-training and performs effectively on sentence similarity, question answering retrieval, and web search retrieval.
- With low data, Condenser achieves performance comparable to task-specific pre-trained models.
- Structural readiness is identified as a fundamental property of easy-to-train bi-encoders, and task-specific objectives may not be necessary.
- Practitioners can use Condenser as an initializer for an immediate performance boost, while combining it with hard negatives and other techniques for higher performance.
A.1 Hyper Parameters Settings
The appendix specifies optimization settings for STS-b, Wikipedia Section Distinction, Open QA, and Web Search, including task-specific learning rates, epochs, batch sizes, and negative-sampling choices.
- STS-b: STS-b uses Adam, a 2e-5 learning rate with linear scheduling, and 4 epochs; low-data BERT selects between 4 and 8 epochs.
- Wikipedia Section Distinction: Wikipedia Section Distinction uses Adam, a 2e-5 learning rate with linear scheduling, and 1 epoch; low-data BERT selects among 1, 4, and 8 epochs.
- Open QA: Open QA uses batch size 128, one BM25 negative, in-batch negatives, 40 epochs, and a 1e-5 learning rate with warmup and linear scheduling.
- Web Search: Web Search uses Adam, a 5e-6 learning rate, 3 epochs, and total batch size 64; low-data BERT selects among 5, 10, and 40 epochs.
A.2 Model Size
Condenser matches BERT base in parameter count during fine-tuning, while its pre-training head increases the total model size.
- Condenser has about 100 M parameters during fine-tuning, matching BERT base, and roughly 120 M parameters when its pre-training head is included.
A.3 ICT Model
ICT-based models are difficult to reproduce because they depend on very large batches and batch-wide contrastive negatives, whereas Condenser uses an instance-wise MLM loss compatible with gradient accumulation.
- Body First Selection and Wiki Link Prediction depend heavily on Wikipedia structure and therefore do not apply in general situations.
- ICT reproduction requires large batches, with Lee et al. using 4096 and Chang et al. using 8192 examples.
- ICT’s contrastive loss uses the entire batch as its negative pool, so gradient accumulation reduces the pool and can harm pre-training.
- Condenser uses an instance-wise MLM loss and can naively use gradient accumulation.
A.4 Document Retrieval
Document-retrieval experiments compare Condenser with BERT and other retrievers on MS-MARCO Document, including hard-negative variants and DL2019 evaluation.
- The document-retrieval evaluation uses the MS-MARCO Document dataset and reports unavailable results as ‘n.a.’ in Table 8.
- Condenser improves over BERT by a large margin on NDCG@10 for DL2019.
- Adding hard negatives boosts Condenser, and Condenser + HN performs best on the Dev set.
- ANCE performs best on DL2019, which the authors conjecture may reflect BM25-negative effects and labels favoring lexical retrievers.
- The Condenser implementation is based on BERT in Hugging Face Transformers and modifies the model architecture while reusing the existing pre-training pipeline.