PaperlayerView exact revision on arXiv

Source-linked AI summary

REDSI: Addressing the Reproducibility and Evaluation Consistency of Differentiable Search Indexing for Document Retrieval

Vivien Nicolas, Hicham Randrianarivo, Pascale Sébillot, Caio Corro

arXiv:2609.08860v1cs.IR

TL;DR

DSI is difficult to reproduce because public implementations do not cover all original identifier types, baseline results vary, and NQ320K construction is underspecified. ReDSI pairs a common implementation with a parameterizable dataset pipeline; experiments find atomic identifiers strongest and robust under downscaling, while dataset choices also affect effectiveness.

  • Problem

    DSI reproduction is hindered by the lack of a public implementation covering all three identifier types, varying reported results, and underspecified NQ320K preprocessing.

    Source: §1 · p. 1

  • Method

    ReDSI implements all three original DSI identifier types in a common pipeline and provides a parameterizable NQ320K construction process.

    Source: §1 · p. 1, §3 · p. 4

  • Results

    Atomic identifiers achieve the strongest effectiveness, degrade least under T5 downscaling, and train substantially faster.

    Source: §7 · pp. 8–9

  • Takeaways & Limitations

    Separating document deduplication from serialization shows that both choices affect retrieval effectiveness.

    Source: §7 · p. 8

    Experiments are limited to English NQ-derived collections and T5-family backbones from T5-Efficient-Tiny to T5-Base; scalability to millions of documents is not established.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The differentiable search index (DSI) framework (Tay et al., 2022) has become the de facto baseline for generative retrieval. However, DSI is hard to reproduce: no public implementation covers all three original document identifier types (atomic, naive, semantic), reported results vary widely, and the ubiquitous NQ320K dataset is built from Natural Questions through diverse and underspecified preprocessing. We introduce ReDSI, the first open-source DSI implementation supporting all three identifier types, together with a parameterizable and well-documented NQ320K construction pipeline. Experimentally, we achieve results that are competitive with or stronger than previous DSI baselines. Moreover, we conduct extensive experiments under model downscaling, covering retrieval effectiveness, parameter efficiency, training methods and decoding strategies, opening novel directions for future research.

1 Introduction

DSI (Tay et al., 2022) is difficult to reproduce because identifier coverage, reported baselines, and NQ320K construction vary. ReDSI provides a common implementation and dataset pipeline, and its experiments find atomic identifiers especially robust under model downscaling.

1 Introduction

  • DSI lacks a public implementation covering all three original identifier types, and inconsistent NQ320K preprocessing makes results difficult to compare.The reported results also vary across partial reimplementations.Source: §1 · p. 1
  • ReDSI is the first open-source implementation supporting atomic, naive, and semantic identifiers in a common training and evaluation pipeline.Its parameterizable NQ320K construction pipeline enables controlled analysis of dataset and modeling choices.Source: §1 · pp. 1–2
  • Across evaluated scales, atomic identifiers outperform naive and semantic identifiers; a 44M-parameter atomic model exceeds 220M-parameter models using either alternative.Atomic identifiers also train faster.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The study examines downscaled models across retrieval effectiveness, parameter efficiency, training methods, and decoding strategies.The authors focus on robustness to downscaling the T5 backbone.Source: §1 · pp. 1–2

2 Background and Related Work

This section introduces DSI and reviews later extensions alongside the challenges of comparing independently reconstructed baselines. It motivates a common implementation and flexible dataset construction for examining the original identifier types.

2 Background and Related Work

  • The section situates REDSI within background on DSI and subsequent generative-retrieval work that motivates the reproduction study.The review covers later extensions and independently implemented baselines.Source: §2 · p. 2

2.1 Differentiable Search Index

  • DSI retrieves documents by assigning each one an identifier and ranking documents by the model’s conditional probability of that identifier given a query.Atomic identifiers use one dedicated output token per document; naive identifiers use arbitrary numerical sequences, while semantic identifiers encode paths through a document hierarchy.Source: §2.1 · p. 2

2.2 Subsequent DSI Extensions

  • Later work explores content-derived and learned multi-token identifiers, while REDSI compares all three original types in a common pipeline and studies downscaling.The authors note that atomic identifiers have received comparatively little dedicated attention in static-corpus settings.Source: §2.2 · p. 3

2.3 Prior DSI Baselines are not Comparable

  • Published DSI baselines differ in datasets, training, backbones, and retrieval decoding, so their reported results are not comparable.Most works also ignore atomic identifiers, while NCI propagates one dataset reconstruction and evaluates semantic identifiers exclusively.Source: §2.3 · p. 4

3 Reconstructing NQ320K

Constructing NQ320K requires choices about which Wikipedia snapshots count as the same document and how HTML snapshots become model input. ReDSI examines both choices and provides configurable approaches to each.

3 Reconstructing NQ320K

  • NQ320K construction is underspecified because snapshots need deduplication and HTML pages need serialization into textual model input.DSI-style models use only the first few dozen serialized tokens, so serialization can affect the indexing term.Source: §3 · p. 4

3.1 Document Deduplication

  • PAGEID deduplication aligns snapshots across revisions and renames, while title, URL, and text-based rules have different matching limitations.The doc=text4k rule is difficult to reproduce: its source representation and character handling are unspecified, and examined interpretations did not reproduce the reported 228K documents.Source: §3.1 · p. 5

3.2 Document Serialization

  • HTMLNORM reconstructs text from original HTML through a configurable pipeline, whereas NCINORM inherits preprocessing artifacts that affect tokenization.Within the first 32 T5 tokens, HTMLNORM removes many unknown tokens; NCINORM can inherit incorrect spacing that consumes input positions.Source: §3.2 · pp. 5–6, Table 3 · p. 6

4 Experimental Setup

The experiments evaluate NQ development-query retrieval across identifier types, using a common training and decoding setup. They test the indexing-to-retrieval sampling ratio and compare REDSI with prior DSI baselines.

4 Experimental Setup

  • Evaluation uses NQ development queries with Hits@1, Hits@10, and MRR@10; multi-token identifiers use width-40 unconstrained beam search followed by invalid-identifier filtering.Models are optimized with Adafactor for 1M updates, and checkpoints are selected by development MRR@10.Source: §4 · p. 6
  • The natural indexing-to-retrieval ratio of 1:2.8 works best on average, so REDSI uses it instead of DSI’s reported 32:1 ratio.The natural ratio comes from sampling each document and labeled query once during a full iteration.Source: §4 · p. 6
  • REDSI results are comparable to Pradeep et al. (2023), and PAGEID deduplication yields higher Hits@1 than TITLE deduplication for both serialization methods.The two serialization methods show small differences in this comparison.Source: §4 · p. 6

5 Identifier Type Robustness under Downscaling

Atomic identifiers retain retrieval effectiveness better than naive and semantic identifiers as the T5 backbone is downscaled, while also reducing training time. Semantic identifiers produce fewer invalid hypotheses, and constrained decoding offers no consistent gain over post-filtering.

5 Identifier Type Robustness under Downscaling

  • Atomic MRR@10 falls by 6.6 points from T5-Base to T5-Efficient-Tiny, versus 13.6 for naive and 12.2 for semantic identifiers.The smaller scales reuse identifier-specific hyperparameters selected for T5-Base without retuning.Source: §5 · p. 7
  • 62.2 MRR@10 from the 44M-parameter atomic model exceeds 60.1 for 220M-parameter naive and 59.7 for 220M-parameter semantic models.Atomic identifiers introduce document-specific output embeddings and require one decoder step rather than autoregressive generation.Source: §5 · p. 7
  • For T5-Base, atomic identifiers reach peak MRR@10 after 6.8 hours, while naive and semantic identifiers continue improving beyond 74.5 hours.The reported checkpoints correspond to approximately 65K updates for atomic and 900K updates for naive and semantic identifiers.Source: §5 · p. 7
  • Semantic identifiers produce fewer invalid hypotheses than naive identifiers at every evaluated scale; on the smallest model, their MRR@10 is 47.5 versus 46.5.For naive identifiers, the invalid proportion increases with the smallest backbone, whereas it decreases sharply for semantic identifiers.Source: §5 · p. 7
  • Constrained decoding provides no consistent improvement over simple post-filtering, including when the model is downscaled.In this setting, smaller T5 backbones retain retrieval effectiveness without complex decoding.Source: §5 · p. 7

6 Qualitative Analysis of NQ320K

The NQ320K construction analysis considers both document deduplication and document-text construction. It reports effects from PAGEID matching and differences between serialization methods.

Document deduplication.

  • PAGEID resolution creates cross-page collisions involving 93 PageIDs, affecting 1.62% of training queries and 2.54% of development queries.PAGEID deduplication associates 1,054 additional training queries with development-target documents, increasing supervision coverage from 75.3% to 76.4%.Source: Document deduplication · p. 8

6 Qualitative Analysis of NQ320K

  • Within the first 32 T5 tokens, NCINORM and HTMLNORM differ for 92.6% of documents, while documents containing at least one <unk> fall from 18.7% to 1.6%.Source: Document deduplication · p. 8

7 Conclusion

ReDSI supports the original DSI identifier types alongside a parameterizable NQ320K construction pipeline. Experiments find atomic identifiers strongest and most robust under downscaling, and the released variants provide a basis for reproducible evaluation.

7 Conclusion

  • ReDSI is an open-source DSI implementation supporting all original identifier types.The conclusion presents the implementation together with a parameterizable NQ320K construction pipeline.Source: §7 · p. 8
  • Separating deduplication from serialization reveals that both choices affect retrieval effectiveness; PAGEID yields larger, more consistent gains than TITLE, and HTMLNORM modestly improves over NCINORM.The pipeline is parameterizable and supports NQ320K variants.Source: §7 · p. 8
  • Atomic identifiers achieve the strongest effectiveness, degrade least under downscaling, and train substantially faster.The conclusion summarizes the experimental findings across effectiveness, downscaling robustness, and training speed.Source: §7 · pp. 8–9
  • The released HTMLNORM variant is intended to support future work that relies more on document textual content.REDSI and the named NQ320K variants provide a reproducible basis for evaluating future DSI-style systems.Source: §7 · p. 9

Limitations

The experiments are bounded to English NQ-derived collections and T5-family models, while the appendices document semantic identifiers, comparison operators, and PageID resolution. These details clarify the tested setting and how key dataset-construction choices were made.

Limitations

  • The controlled experiments cover English NQ-derived collections and T5-family backbones from T5-Efficient-Tiny to T5-Base; scalability to millions of documents is not established.Atomic identifiers add one output vector per document, and smaller backbones reuse T5-Base hyperparameters rather than scale-specific tuning.Source: Limitations · p. 9

A Semantic Identifiers

  • Semantic identifiers use recursively clustered document embeddings: ten clusters per internal node, with local indices assigned once clusters contain at most 100 documents.The root-to-leaf cluster labels form a shared decimal prefix, followed by a unique local index.Source: A Semantic Identifiers · p. 12

B.1 Comparison Operators and Collection Sizes

  • The four principal deduplication keys are the first 4,000 lowercased text code points, revision-specific URL, normalized title, and recovered PageID with a title fallback.TITLE uses a split-dependent source: the first HTML h1 for training and document_title for development.Source: §B.1 · p. 12

B.2 PageID Resolution

  • The Wikipedia metahistory dump resolves 231,589 of 231,695 NQ revision identifiers to PageIDs, a 99.95% resolution rate.The remaining 106 identifiers cover 156 NQ records; unresolved records use a URL-title fallback key and are marked in the grouping audit.Source: §B.2 · p. 13

B.3 Revision-Specific Content Variation under a Shared Title

Title-based grouping can erase revision-specific distinctions, and a fixed revision URL does not guarantee one unique rendered text. The appendices describe HTMLNORM reconstruction and serialization choices, alongside relevant properties and boundaries of the NQ formats.

B.3 Revision-Specific Content Variation under a Shared Title

  • Title equality does not guarantee equivalent revision text; whether revisions form one retrieval document depends on whether the intended unit is a persistent page or a textual snapshot.The examples include records whose text distinguishes the 2014 John Wick film from the broader franchise and the original South Korean program from its international franchise.Source: §B.3 · p. 13, Table 9 · p. 14

B.4 Rendered-Text Variation under a Fixed Revision

  • NQ contains 241,247 distinct flattened document texts for 231,695 distinct revision URLs; template rendering can make a revision URL map to multiple texts.A revision URL is a stable source-revision key, not a one-to-one mapping to the flattened NQ representation.Source: §B.4 · p. 14

C HTMLNORM Reconstruction and Serialization; C.1 Byte-Aligned Span Recovery; C.5 Document Serialization Examples

  • HTMLNORM recovers token surfaces from byte-aligned HTML spans, restores spacing and block boundaries, and serializes structured content before Wikipedia-specific cleanup.Its reconstructed text preserves readable structure, unlike the simplified representation that space-joins stored token strings.Source: C HTMLNORM Reconstruction and… · p. 14, §C.1 · p. 14, Figure 6 · p. 17

C.2.1 List Formatting; C.2.2 Table Formatting

  • HTMLNORM keeps list items line-oriented and selects table serialization by layout, favoring locally interpretable facts over flat cell concatenation.List formatting preserves item structure after markup removal.Source: §C.2.1 · p. 15, §C.2.2 · p. 15

C.3 Wikipedia-Specific Cleanup

  • Wikipedia cleanup removes interface artifacts, introductory infoboxes, and footer-like sections to keep non-article content from consuming the fixed indexing prefix.Other lists and tables in article bodies are retained using the serialization rules.Source: §C.3 · pp. 15–16

C.4 Pronunciation Cleanup

  • The pronunciation heuristic checks only the first parenthetical span within 256 characters and removes it if T5 tokenization contains an unknown token; it can remove non-pronunciation content.Later parentheticals are not removed by this procedure.Source: §C.4 · p. 16

D Natural Questions; D.1 NQ Formats

  • Full NQ records include the original HTML and tokenized article content, but the preprocessing pipeline that produced document_tokens is unavailable.The stored token strings may contain substitutions that alter subsequent T5 tokenization.Source: §D.1 · pp. 16–17

D.2 Character Substitutions in the NQ Preprocessing Pipeline

NQ token strings can differ from their byte-aligned HTML spans, and space-joining tokens can introduce artifacts that affect T5 tokenization. The analysis measures substitutions and describes how HTMLNORM avoids added whitespace tokens.

D.2 Character Substitutions in the NQ Preprocessing Pipeline

  • 1.88M of 54.13M compared tokens differ from their original spans (3.47%), across 22,333 replacement pairs; ten frequent pairs account for 94.6% of mismatches.The count covers 7,830 development records.Source: §D.2 · pp. 17–18, Table 12 · p. 19
  • Most mismatches are systematic quote, dash, and bracket normalization, which can change T5 token sequences and unknown-token coverage.SIMPLENORM inherits these transformations, while HTMLNORM recovers the corresponding byte spans from document_html.Source: §D.2 · p. 18

D.3 Whitespace Artifacts and T5 Tokenization

  • HTMLNORM avoids extra tokens introduced by whitespace artifacts, which would otherwise consume positions in the fixed-length input budget.The NQ pipeline’s space insertion can alter subsequent T5 tokenization.Source: §D.3 · pp. 18–19

E Indexing-to-Retrieval Ratio

The appendix examines how the indexing-to-retrieval sample ratio, implementation choices, and decoding procedures relate to retrieval and invalidity. It also documents configurations and comparisons used in the experiments.

E Indexing-to-Retrieval Ratio

  • The natural NQ320K-TITLE/NCINORM ratio is approximately 1:2.8, compared with DSI’s reported 32:1 ratio (Tay et al., 2022).The ratio counts indexing-example samples against retrieval-example samples; indexing-heavy settings repeat document examples, while 0:1 removes indexing supervision.Source: E Indexing-to-Retrieval Ratio · p. 19
  • Removing indexing supervision substantially reduces retrieval effectiveness, while indexing-heavy ratios generally degrade it; the natural ratio remains close to best across identifier types.A 1:1 ratio yields the highest MRR@10 for atomic and naive identifiers, but only marginally exceeds the natural ratio; the natural ratio is best for semantic identifiers and avoids identifier-specific tuning.Source: E Indexing-to-Retrieval Ratio · p. 19

F Hyperparameter Search Results

  • Atomic identifiers retain ℓ=64, a 5 × 10−4 learning rate, and 10K warmup steps; naive and semantic identifiers use ℓ=32, 10−3, and no warmup.These settings were selected from the batch-128 hyperparameter sweep and reused throughout the experiments.Source: F Hyperparameter Search Results · p. 19, Table 14 · p. 21

G.1 Software and Hardware

  • The implementation uses Transformers 4.44.1, T5 1.0 checkpoints, FP16 on 16 GB NVIDIA V100 GPUs, and Adafactor; atomic identifiers add document-specific output tokens.Naive and semantic identifiers retain the original T5 vocabulary and are generated autoregressively.Source: §G.1 · p. 20

G.2 Framework Differences

  • Unpacked batches count one padded indexing or retrieval example per element, but T5X packing is unreported for Pradeep et al. (2023), so nominal batch sizes may not be comparable.Other undocumented differences include token-loss normalization, auxiliary losses, identifier construction, beam search, and checkpoint selection, preventing attribution of remaining effectiveness differences to one factor.Source: §G.2 · p. 20

H Run-to-Run Variability of NQ320K Construction Effects

  • The construction-effect variability analysis repeats each T5-Efficient-Tiny configuration across seeds 42–46, for 60 runs, selecting checkpoints by development MRR@10.The runs reuse the main-experiment hyperparameters; Tables 16 and 17 report variability and paired construction effects.Source: H Run-to-Run Variability of NQ320K… · p. 20, Table 16 · p. 21

I Invalidity across Construction Variants and Model Scales

J.1 Decoding Procedures

  • Raw decoding searches the full T5 vocabulary, post-filtering removes completed invalid identifiers without changing retained scores or order, and trie constraints limit generation to valid-identifier prefixes.Post-filtering can return fewer than 40 documents and cannot recover valid hypotheses outside the original beam; trie constraints guarantee validity but not relevance.Source: §J.1 · pp. 20–21

J.2 Identifier-Space Budget Sensitivity

The identifier-space budget sensitivity analysis finds that invalid hypotheses become more common at larger budgets, while retrieval effectiveness varies non-monotonically. Filtering becomes more useful as invalid hypotheses occupy more of the beam, but trie constraints show mixed effects at larger budgets.

J.2 Identifier-Space Budget Sensitivity

  • Trie-constrained decoding guarantees valid outputs but provides no consistent effectiveness advantage over post-filtering in the evaluated settings.Larger budgets make assignments sparser, but also alter identifier characteristics.Source: §J.2 · pp. 21–22
  • The budget sweep is a sensitivity analysis, not an isolated test of invalidity, because larger budgets also change identifier tokenization, length, and prefix sharing.The identifier-assignment seed and other settings remain fixed.Source: §J.2 · pp. 21–22
  • Invalid@40 rises from 8.8% at 1× to 53.8% at 32×, while trie-constrained decoding changes MRR@10 by −0.97 to +0.23 percentage points at larger budgets.Figure 8 compares raw, post-filtered, and trie-constrained decoding for naive identifiers, alongside Invalid@40; constrained decoding closely matches post-filtering through 4×.Source: §J.2 · p. 22, Figure 8 · p. 22
Loading 2609.08860v1…