Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction

1 INTRODUCTION

• Dense retrieval encodes queries and documents into low-dimensional representations and compares them with dot product or cosine similarity, making representation quality foundational to effective search.Source: §1 · p. 1
• Autoencoder-based pre-training may reduce discriminative power by reconstructing common words, while even a weak decoder can bypass the encoder through access to previous tokens.The bypass risk is especially noted for long texts.Source: §1 · pp. 1–2
• COSTA drops the decoder and trains the encoder to bring each text representation closer to its own random spans and farther from other groups.The approach also uses the masked language modeling task during pre-training.Source: §1 · p. 2
• Group-wise supervision encourages representations within a semantic group to align and different groups to separate, while encoder-only pre-training avoids decoder bypass.Source: §1 · p. 2
• COSTA achieves significant improvements over state-of-the-art baselines across four representative downstream dense retrieval datasets.The experiments include MS MARCO passage and document ranking and two TREC 2019 Deep Learning tracks.Source: §1 · p. 2

2.1 Dense Retrieval

• Dense retrieval commonly uses separate query and document encoders with similarity scoring; techniques such as hard-negative mining and late interaction can improve performance but may require extra computation or storage.Source: §2.1 · p. 2

2.2 Pre-training for IR

• Prior IR pre-training includes ICT, BFS, and WLP, while other approaches target reranking, structural readiness, or document representations.The passage reports marginal improvements for ICT, BFS, and WLP on typical ad-hoc retrieval benchmarks, and limited improvements for sampled-word preference prediction when used for retrieval.Source: §2.2 · pp. 2–3
• SEED-Encoder is the closest related work described: it uses a three-layer weak Transformer decoder and restricts span attention to 2.The passage points to two issues with that method discussed in the introduction.Source: §2.2 · p. 3

3.1 Motivation

• COSTA removes the decoder because it can bypass the encoder and decoding all text may reduce representation discriminability by favoring common words.The encoder itself enforces the information bottleneck through contrastive span prediction.Source: §3.1 · p. 3

3.2 Contrastive Span Prediction Task

• COSTA samples word-, phrase-, sentence-, and paragraph-level spans, whose fine-to-coarse granularities capture different properties of the input text.It samples T spans at each level and filters stopwords from word-level spans.Source: §3.2 · pp. 3–4
• A group-wise contrastive loss pulls each text representation toward its own sampled spans and away from representations from other groups in the mini-batch.The text and spans are encoded with a Transformer; span representations use average pooling over their output word representations.Source: §3.2 · pp. 3–4

3.3 Masked Language Modeling

• COSTA jointly trains with masked language modeling, which predicts randomly masked tokens from the remaining tokens to support contextual text representations.The contrastive span prediction and MLM losses are optimized together.Source: §3.3 · pp. 4–5

4.1 Research Questions

• The research questions compare COSTA with models using the same or more complicated fine-tuning, and examine its components, discriminative ability, and low-resource performance.Source: §4.1 · p. 5

4.2 Pre-training Corpus and Downstream Tasks

• COSTA uses English Wikipedia for pre-training; MS MARCO Passage and Document Ranking have about 0.5 million and 0.4 million training queries, and TREC 2019 supplies novel, more comprehensively labeled test sets.Source: §4.2 · p. 5

4.3.1 Traditional Sparse Retrieval Models.

• Traditional sparse baselines include BM25 and DeepCT, alongside representative results reported in the TREC overview.Source: §4.3.1 · p. 5

4.3.2 Other Pre-training Models.

• Pre-training baselines include BERT, ICT, PROP, B-PROP, and SEED, which use different language-modeling, retrieval, word-preference, or reconstruction objectives.ICT is designed for passage retrieval in question answering; SEED combines reconstruction and masked language modeling on English Wikipedia.Source: §4.3.2 · p. 5

4.3.3 Advanced Dense Retrieval Models that Fine-tunes with Compli-

• Advanced dense-retrieval comparators include ANCE, TCT-ColBERT, TAS-B, ADORE+STAR, and RocketQA, which use more elaborate training strategies.Source: §4.3.3 · p. 5

4.4 Evaluation Metrics

• Evaluation uses MRR and recall for MS MARCO, and NDCG@10 plus recall at the specified cutoff for the two TREC tasks.MS MARCO passage uses MRR@10 and R@1000; document uses MRR@100 and R@100. TREC passage uses NDCG@10 and R@100, while TREC document uses NDCG@10 and R@1000.Source: §4.4 · pp. 5–6

4.5.1 Pre-training Procedures.

• Pre-training uses BERT initialization, learns the projector from scratch, and runs for six epochs on Wikipedia with multi-granularity spans.Phrase, sentence, and paragraph spans use length ranges 4–16, 16–64, and 64–128; five spans are sampled per granularity, and long inputs are chunked to 512 tokens.Source: §4.5.1 · p. 6

4.5.2 Fine-tuning Procedures.

• Fine-tuning uses a two-stage strategy for passage ranking and a separate static-hard-negative procedure for document ranking.Passage models warm up on BM25 negatives, then train on hard negatives mined once; document models start from the passage model and train for one epoch in each of two static-negative iterations. BERT and SEED use open-sourced models, while ICT follows COSTA’s pre-training procedure and PROP and B-PROP use a bi-encoder.Source: §4.5.2 · p. 6, §4.5.1 · p. 6

• The section analyzes experimental results for the proposed COSTA method.Source: §5 · p. 6
• Its stated purpose is to demonstrate COSTA’s effectiveness.Source: §5 · p. 6
• The section is devoted to experimental results rather than method description.Source: §5 · p. 6

5.1 Baseline Comparison with the Same Fine-tuning Strategy

• Dense retrieval models outperform traditional sparse models by a large margin, while SEED significantly outperforms BERT, PROP, B-PROP, and ICT.The passage-ranking results also report that COSTA improves over SEED and that its group-wise contrastive task combines learning from the input with distinguishing other representations.Source: §5.1 · p. 7
• Document-ranking trends match passage-ranking trends except on TREC2019 Doc, where dense retrieval significantly underperforms BM25 on R@1000; COSTA improves SEED’s MRR by 7% on both datasets.The authors suggest unlabeled relevant documents treated as negatives may bias training and hurt performance.Source: §5.1 · p. 7
• COSTA improves SEED’s top-ranking performance by 7% on both MS MARCO passage-ranking and document-ranking datasets in MRR, indicating better ability to rank positive documents higher.For document ranking, Table 2 reports COSTA’s improvements over baselines as statistically significant (p ≤ 0.05).Source: §5.1 · p. 7, Table 2 · p. 7
• COSTA’s gains over SEED are greater on ranking metrics such as MRR and NDCG than on recall metrics.The paper relates this pattern to COSTA’s ability to rank positive documents higher.Source: §5.1 · p. 7

5.2 Baseline Comparison with Different Fine-tuning Strategies

• With simple fine-tuning, COSTA performs better than advanced dense-retrieval models trained with complicated strategies.Table 3 compares COSTA with advanced models on MARCO Dev Passage.Source: §5.2 · p. 7, Table 4 · p. 8
• The authors attribute COSTA’s improvement mainly to its pre-training stage.Some compared dense-retrieval models use more powerful pre-trained models, including RoBERTa and ERNIE.Source: §5.2 · p. 7
• The paper leaves applying the advanced fine-tuning methods to COSTA for further study.Source: §5.2 · p. 7

5.3.1 Impact of the Span Granularity.

• Removing paragraph-level spans causes a significant MRR@10 decrease, while removing word-level spans causes only a slight decrease.All span granularities contribute; phrase- and sentence-level spans also help capture fine-grained information.Source: §5.3.1 · p. 8

5.3.2 Impact of the Span Number.

• T=5 achieves the best performance across tested span counts; increasing the count does not improve performance consistently.At T=20, the objective aligns whole-text representations with 80 span representations, making early training harder; curriculum learning is suggested for future study.Source: §5.3.2 · p. 8

5.3.3 Impact of the Projector Architecture.

• A nonlinear projector performs better than a linear projector during pre-training and much better than no projector, while fine-tuning results differ little with or without one.The authors suggest nonlinear projection helps keep whole-text representations distinct from span representations and align with multiple spans simultaneously.Source: §5.3.3 · p. 8, Table 6 · p. 8

5.3.4 Impact of the Temperature in Loss Function.

• Performance is sensitive to temperature: values that are too small or too large perform badly, so COSTA uses 0.1 for pre-training.The authors suggest small values may make convergence too difficult, while large values may lead to a trivial solution.Source: Table 7 · p. 8, §5.3.4 · p. 9

5.4 Visual Analysis

• The analysis uses t-SNE to visualize query and text representations on the MARCO Dev Passage dataset.The comparison fine-tunes COSTA and SEED with 1,000 queries and plots a sampled query with its top-100 candidate documents.Source: §5.4 · p. 9
• COSTA maps the relevant document closer in semantic space and farther from other candidates, while SEED’s relevant-document distribution is relatively random.The visualization uses one sampled query and its top-100 candidate documents.Source: §5.4 · p. 9
• The authors attribute SEED’s relatively random distribution to reconstructing input texts while treating all tokens equally, which they say does not learn discriminative representations..Source: §5.4 · p. 9

5.5 Low-Resource Setting

• COSTA significantly outperforms SEED on both datasets across all metrics in the low-resource setting.The experiments sample 500, 1,000, 2,000, or 5,000 queries, using three splits per setting and reporting average performance.Source: §5.5 · p. 9
• With 2,000 queries, COSTA outperforms BM25 on MARCO Dev Passage in both MRR@10 and R@1000.The authors describe 2,000 queries as less than 0.5% of the queries.Source: §5.5 · p. 9
• COSTA performs worse on TREC2019 Passage, which the authors attribute to false negatives in MARCO fine-tuning..Source: §5.5 · p. 9

6 CONCLUSION

• COSTA pre-trains a discriminative text encoder for dense retrieval through contrastive span prediction..Source: §6 · p. 9
• The task brings together a bottleneck principle, where text representations represent the input through reconstruction, and a contrastive strategy that distinguishes representations across texts.It brings representations of original texts and their own spans closer while pushing away representations of other groups.Source: §6 · p. 9
• Across four benchmark dense-retrieval datasets, COSTA outperforms several strong baselines; future work will apply it to open-domain question answering and conversational systems..Source: §6 · p. 9