Condenser: a Pre-training Architecture for Dense Retrieval

Luyu Gao; Jamie Callan

Source-linked AI summary

Condenser: a Pre-training Architecture for Dense Retrieval

Luyu Gao, Jamie Callan

arXiv:2104.08253v2Apr 16, 2021cs.CLcs.IR

TL;DR

Dense bi-encoders are efficient but difficult to train because standard language-model attention is not prepared to aggregate text into a single vector, especially with limited data. Condenser pre-trains toward this structure by conditioning language-model prediction on dense representations, improving performance across similarity and retrieval tasks while retaining standard fine-tuning capacity.

Problem
Bi-encoders require sophisticated training and lose performance in low-data settings because standard language models lack attention structure for aggregating information into dense representations.
Source: §1 · p. 1
Method
Condenser is a Transformer pre-training architecture that conditions language-model prediction on dense representations and removes its auxiliary head during fine-tuning.
Source: §1 · p. 1, §3.3 · p. 4
Results
Condenser improves over standard language models across sentence similarity, question-answering retrieval, and web-search retrieval, with comparable low-data performance to task-specific pre-trained models.
Source: §1 · pp. 1–2, §6 · p. 9
Takeaways & Limitations
Structural readiness is a fundamental property of easy-to-train bi-encoders, and general language-model pre-training may provide a lightweight alternative to complicated training techniques.
Source: §6 · p. 9
The paper focuses on passage retrieval and defers discussion of long-document retrieval.
Source: §4.4 · p. 7

Abstract

from arXiv · show

Pre-trained Transformer language models (LM) have become go-to text representation encoders. Prior research fine-tunes deep LMs to encode text sequences such as sentences and passages into single dense vector representations for efficient text comparison and retrieval. However, dense encoders require a lot of data and sophisticated techniques to effectively train and suffer in low data situations. This paper finds a key reason is that standard LMs' internal attention structure is not ready-to-use for dense encoders, which needs to aggregate text information into the dense representation. We propose to pre-train towards dense encoder with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. Our experiments show Condenser improves over standard LM by large margins on various text retrieval and similarity tasks.

1 Introduction

Dense bi-encoders offer efficient single-vector text representations but are harder to train than cross-encoders, especially with limited data. Condenser addresses this difficulty by pre-training language models toward dense-encoder structure and improves retrieval performance.

Dense encoders represent text sequences as single vectors for efficient comparison and retrieval.Source: §1 · p. 1
Bi-encoders require sophisticated training methods and suffer substantial performance losses in low-data settings.Source: §1 · p. 1
Standard language models lack attention structure trained to aggregate sophisticated information into one dense representation.Source: §1 · p. 1
Condenser pre-trains language models toward bi-encoder structure by conditioning language-model prediction on dense representations.Source: §1 · p. 1
Under low-data setups, Condenser improves over standard language models and achieves performance comparable to strong task-specific pre-trained models.Source: §1 · pp. 1–2
With large training data, Condenser retrievers outperform previous models trained with complicated techniques using one round of negative mining.Source: §1 · p. 2

2 Related Work

Related work establishes dense retrieval as an efficient alternative to cross-encoder retrieval and develops pre-training and fine-tuning strategies for stronger bi-encoders. Condenser differs by targeting bi-encoder structure through general language-model pre-training.

Transformer bi-encoders enable efficient sentence similarity and dense retrieval through vector representations.Source: §2 · p. 2
Dense retrieval compares encoded query and document vectors with inner products and supports millisecond-level online retrieval from pre-encoded indexes.Source: §2 · p. 2
Inverse Cloze Task and related pre-training methods improve bi-encoders, while models without pre-training fail under low-data retrieval settings.Source: §2 · p. 2
Condenser provides an architecture-based explanation for bi-encoder learning difficulty using general language-model pre-training.Source: §2 · p. 2
Later dense-retrieval work improves end performance through hard-negative mining and other costly fine-tuning techniques.Source: §2 · p. 2
This paper focuses on task-specific fine-tuning of the entire model rather than universal sentence-embedding features.Source: §2 · p. 2

3 Method

Condenser modifies Transformer language-model pre-training so dense representations actively aggregate information across layers. Its auxiliary head guides this structure during pre-training, then is removed so fine-tuning uses the standard-capacity encoder backbone.

3.1 Preliminaries

Transformer encoders refine each token representation through stacked self-attentive blocks using sequence-wide context.Source: §3.1 · p. 3
Masked language-model pre-training predicts masked tokens from their final representations, typically alongside a prepended CLS token.Source: §3.1 · p. 3

3.2 Issues with Transformer Encoder

In standard language models, CLS remains relatively inactive through middle layers and gains broad attention only near the final layer.Source: §3.2 · p. 3

3.3 Condenser

Condenser conditions language-model pre-training on dense representations to establish structural readiness for bi-encoder fine-tuning.Source: §3.3 · p. 3
The architecture separates early and late backbone layers from Condenser-head layers and connects early outputs directly to the head.Source: §3.3 · p. 3, Figure 1 · p. 3
The late CLS must aggregate newly generated backbone information, while skip connections let it focus on global input meaning.Source: §3.3 · p. 4
The Condenser head is used only during pre-training and is dropped during fine-tuning, leaving an encoder with identical capacity to a comparable Transformer.Source: §3.3 · p. 4

3.4 Condenser from Transformer Encoder

When initialized from a pre-trained Transformer, Condenser adds a backbone MLM constraint to prevent the randomly initialized head from corrupting backbone weights.Source: §3.4 · p. 4

4 Experiments

Experiments evaluate Condenser on sentence similarity, open-domain question answering, and web search under low- and full-data settings. Across these settings, Condenser remains competitive with or outperforms standard and task-specific alternatives, while requiring less elaborate training.

Experimental setup

Experiments cover STS-b, Wikipedia Section, Natural Question, TriviaQA, and MS-MARCO, using both low-data and full-training settings.The evaluation includes sentence-level similarity, open QA passage retrieval, and web-search retrieval.Source: §4 · p. 4, §4.2 · p. 5, §4.3 · p. 5

Sentence similarity

With only 500 STS-b training pairs, NLI-pre-trained BERT and Condenser outperform the best Universal Sentence Encoder baseline.NLI-pre-trained BERT and Condenser consistently outperform standard BERT, with larger margins as training data decreases.Source: §4.2 · p. 5
On Wiki Section, BERT and Condenser achieve almost identical results and retain effectiveness without NSP pre-training.Using 1K training examples causes only about a 10% accuracy drop relative to training with all data.Source: §4.2 · p. 5

Open QA retrieval

At 1K QA training examples, ICT and Condenser outperform vanilla BERT, dropping less than 10% on Top-20 Hit and less than 5% on Top-100 versus full-data training.ICT and Condenser show comparable performance; ICT is slightly better on NQ, while Condenser is better on TQA.Source: §4.3 · p. 6
On full-data NQ, Condenser performs best for Top-20 and comes within 0.1 of RocketQA for Top-100 without its sophisticated, costly training pipeline.On TQA, Condenser outperforms all other lexical and dense systems, including systems using query expansion.Source: §4.3 · p. 6

Web search retrieval

On MS-MARCO, Condenser outperforms prior systems by large margins and slightly exceeds RocketQA without complex training techniques or a heavier model or retrieval process.Against RocketQA variants, Condenser with batch size 64 and BM25 negatives performs better than RocketQA with batch size 8192, and is outperformed only by the external-data variant.Source: §4.4 · pp. 7–8

5 Attention Analysis

Attention analysis compares CLS attention entropy across BERT, ICT, and Condenser before and after fine-tuning. BERT changes substantially, whereas ICT and Condenser retain attention structures that require only small adjustments.

CLS attention entropy is used to characterize attention patterns across pre-trained and fine-tuned BERT, ICT, and Condenser.The analysis measures how CLS attends across tokens to compare internal attention behavior.Source: §5 · p. 8
BERT shows a drastic change in CLS attention between pre-training and fine-tuning, indicating substantial internal restructuring for bi-encoder use.This supports the claim that typical Transformer encoder LMs are not structurally ready for bi-encoder fine-tuning.Source: §5 · p. 8
ICT and Condenser show only small attention changes between pre-training and fine-tuning, retaining structures suited to effective bi-encoder use.Their attention structures require smaller modifications than BERT's during fine-tuning.Source: §5 · pp. 8–9

6 Conclusion

The conclusion argues that standard pre-trained LMs lack attention structures ready for bi-encoders, and introduces Condenser to establish this readiness during pre-training. Across several tasks and data regimes, Condenser offers comparable low-data performance and a lightweight alternative to more elaborate training techniques.

Typical pre-trained LMs lack internal attention structures ready to condense information into a single bi-encoder vector.Source: §6 · p. 9
Condenser establishes bi-encoder structural readiness during LM pre-training and performs effectively on sentence similarity, question answering retrieval, and web search retrieval.Source: §6 · p. 9
With low data, Condenser achieves performance comparable to task-specific pre-trained models.Source: §6 · p. 9
Structural readiness is identified as a fundamental property of easy-to-train bi-encoders, and task-specific objectives may not be necessary.Source: §6 · p. 9
Practitioners can use Condenser as an initializer for an immediate performance boost, while combining it with hard negatives and other techniques for higher performance.Source: §6 · p. 9

A.1 Hyper Parameters Settings

The appendix specifies optimization settings for STS-b, Wikipedia Section Distinction, Open QA, and Web Search, including task-specific learning rates, epochs, batch sizes, and negative-sampling choices.

STS-b

STS-b uses Adam, a 2e-5 learning rate with linear scheduling, and 4 epochs; low-data BERT selects between 4 and 8 epochs.Source: §A.1 · p. 12

Wikipedia Section Distinction

Wikipedia Section Distinction uses Adam, a 2e-5 learning rate with linear scheduling, and 1 epoch; low-data BERT selects among 1, 4, and 8 epochs.Source: §A.1 · p. 12

Open QA

Open QA uses batch size 128, one BM25 negative, in-batch negatives, 40 epochs, and a 1e-5 learning rate with warmup and linear scheduling.Source: §A.1 · p. 12

Web Search

Web Search uses Adam, a 5e-6 learning rate, 3 epochs, and total batch size 64; low-data BERT selects among 5, 10, and 40 epochs.Source: §A.1 · p. 12

A.2 Model Size

Condenser matches BERT base in parameter count during fine-tuning, while its pre-training head increases the total model size.

Condenser has about 100 M parameters during fine-tuning, matching BERT base, and roughly 120 M parameters when its pre-training head is included.Source: §A.2 · p. 12

A.3 ICT Model

ICT-based models are difficult to reproduce because they depend on very large batches and batch-wide contrastive negatives, whereas Condenser uses an instance-wise MLM loss compatible with gradient accumulation.

Body First Selection and Wiki Link Prediction depend heavily on Wikipedia structure and therefore do not apply in general situations.Source: §A.3 · p. 12
ICT reproduction requires large batches, with Lee et al. using 4096 and Chang et al. using 8192 examples.Source: §A.3 · p. 12
ICT’s contrastive loss uses the entire batch as its negative pool, so gradient accumulation reduces the pool and can harm pre-training.Source: §A.3 · p. 12
Condenser uses an instance-wise MLM loss and can naively use gradient accumulation.Source: §A.3 · p. 12

A.4 Document Retrieval

Document-retrieval experiments compare Condenser with BERT and other retrievers on MS-MARCO Document, including hard-negative variants and DL2019 evaluation.

The document-retrieval evaluation uses the MS-MARCO Document dataset and reports unavailable results as ‘n.a.’ in Table 8.Source: §A.4 · p. 12
Condenser improves over BERT by a large margin on NDCG@10 for DL2019.Source: §A.4 · p. 12
Adding hard negatives boosts Condenser, and Condenser + HN performs best on the Dev set.Source: §A.4 · p. 13
ANCE performs best on DL2019, which the authors conjecture may reflect BM25-negative effects and labels favoring lexical retrievers.Source: §A.4 · p. 13
The Condenser implementation is based on BERT in Hugging Face Transformers and modifies the model architecture while reusing the existing pre-training pipeline.Source: §A.5 · p. 13