Source-linked AI summary
Results of the Big ANN: NeurIPS'23 competition
Harsha Vardhan Simhadri, Martin Aumüller, Amir Ingber, Matthijs Douze, George Williams, Magdalen Dobson Manohar, Dmitry Baranchuk, Edo Liberty, Frank Liu, Ben Landrum, Mazin Karjikar, Laxman Dhulipala, Meng Chen, Yue Chen, Rui Ma, Kai Zhang, Yuzheng Cai, Jiayang Shi, Yizhuo Chen, Weiguo Zheng, Zihao Wan, Jie Yin, Ben Huang
TL;DR
The paper addresses ANN search for practical workloads that extend beyond standard dense-vector indexing. It describes a competition spanning four search variants, its datasets and evaluation framework, and participant approaches. The competition reported notable improvements in search accuracy and efficiency, while identifying boundaries in its rules and unresolved questions about combining streaming methods.
Problem
The challenge targeted practical ANN variants—filtered, out-of-distribution, sparse, and streaming search—that differ from standard dense-vector indexing.
Source: §1 · p. 1Method
The competition defined four tracks and their datasets, evaluated submissions under constrained resources, and summarized participant approaches and the evaluation framework.
Source: §1 · pp. 1–2, §2 · p. 2Results
The competition reported notable improvements in search accuracy and efficiency through approaches from academic and industrial participants.
Source: §6 · p. 8Takeaways & Limitations
The competition established task definitions, datasets, workloads, and an experimental framework that were later used in other research, including Bruch et al. [5].
Source: §1 · p. 2The discussion leaves open whether combining GrassRMA’s memory access patterns with PtANNs’ reduced precision and caching would perform better.
Source: §4.4 · p. 7
Abstract
from arXiv · showhide
The 2023 Big ANN Challenge, held at NeurIPS 2023, focused on advancing the state-of-the-art in indexing data structures and search algorithms for practical variants of Approximate Nearest Neighbor (ANN) search that reflect the growing complexity and diversity of workloads. Unlike prior challenges that emphasized scaling up classical ANN search ~\cite{DBLP:conf/nips/SimhadriWADBBCH21}, this competition addressed filtered search, out-of-distribution data, sparse and streaming variants of ANNS. Participants developed and submitted innovative solutions that were evaluated on new standard datasets with constrained computational resources. The results showcased significant improvements in search accuracy and efficiency over industry-standard baselines, with notable contributions from both academic and industrial teams. This paper summarizes the competition tracks, datasets, evaluation metrics, and the innovative approaches of the top-performing submissions, providing insights into the current advancements and future directions in the field of approximate nearest neighbor search.
1 Introduction
The paper frames ANN search as increasingly important for large, high-dimensional workloads and introduces a competition focused on practical variants beyond standard dense-vector indexing. It presents the tracks, datasets, metrics, and participant approaches, while noting that competition metrics cannot capture every nuance of robust vector search.
1 Introduction
- ANN search supports applications including computer vision, NLP, information retrieval, and retrieval-augmented generation, where billions of high-dimensional vectors make efficient and accurate search essential.The paper notes that LLMs use ANN search to add knowledge after training [15].Source: §1 · p. 1
- The challenge targeted filtered, out-of-distribution, sparse, and streaming ANN search as practical variants beyond standard dense-vector indexing.It aimed to encourage algorithms and optimized implementations for diverse datasets under constrained computational resources.Source: §1 · p. 1
- The paper surveys the competition’s tracks, datasets, metrics, and notable participant approaches, and reports that its task definitions, workloads, and framework were later used in other research, including Bruch et al. [5].Small datasets, accessible hardware, and development grants were intended to broaden participation.Source: §1 · pp. 1–2
- The authors state that the competition’s metrics and tracks cannot capture every nuance of a robust vector-search algorithm.They also state that the tracks cover diverse scenarios and use widely accepted community metrics.Source: §1 · p. 2
2 Tracks and datasets
The competition defined four independent tracks with datasets and workloads tailored to filtered, out-of-distribution, sparse, and streaming search. It used fixed database vectors and fresh final queries for evaluation, with a different runbook in the streaming track.
2 Tracks and datasets
- Each of four tracks indexed a dataset for its query variant, with public datasets, fixed database vectors, fresh final queries, and independent leaderboards.The final streaming evaluation also used a runbook different from the initial release.Source: §2 · p. 2, §2.4 · p. 3
2.1 Filtered Search Track
- Filtered search retrieved CLIP-embedded YFCC images whose tags contained each query’s one or two specified tags.The dataset used 10 million images and a vocabulary of 200,386 tags.Source: §2.1 · p. 2, Figure 1 · p. 3
2.2 Out-Of-Distribution Track
- The out-of-distribution track paired 10 million Yandex image embeddings with text-query embeddings from a DSSM variant, modeling different query and database distributions in shared space.The image vectors were 200-dimensional and produced by Se-ResNext-101.Source: §2.2 · p. 2
2.3 Sparse Track
- The sparse track used 8,841,823 MSMARCO passages represented by SPLADE vectors and evaluated top-k retrieval by maximal inner product.Base vectors averaged approximately 120 nonzero elements; query vectors averaged approximately 49.Source: §2.3 · p. 3
2.4 Streaming Track
- The streaming track tested insertions, deletions, and searches in batches on a 10-million-vector MS Turing slice, under an 8GB memory limit.The final runbook had 1,280 batches and a one-hour time bound.Source: §2.4 · p. 3, Figure 2 · p. 4
3 Evaluation
Evaluation standardized the machine, query workloads, and scoring across four independent leaderboards. Most tracks ranked throughput subject to a recall threshold, while streaming ranking required completing the runbook within an hour.
3 Evaluation
- Entries ran on Azure D8lds_v5 machines with 8 vCPUs and 16GB RAM, except streaming, which had an 8GB memory limit.Each task had an independent leaderboard and participants could submit separate entries.Source: §3 · p. 4, §3.1 · p. 4
3.1 Metrics
- For filtered, out-of-distribution, and sparse tracks, the score was the highest query throughput achieving at least 90% 10-recall@10.Throughput was measured in queries per second; 10-recall@10 averaged recall over queries.Source: §3.1 · p. 4
3.2 Evaluation protocol
- Streaming entries were ranked by maximum recall across searches, averaged over checkpoints, only if all runbook operations finished within one hour.The framework automated dataset preparation, execution, and result evaluation; non-streaming index builds had a 12-hour limit.Source: §3.1 · p. 4, §3.2 · p. 4
4 Competition results: baselines and notable approaches
The competition received 26 entries across filtered, out-of-distribution (OOD), sparse, and streaming search tracks. Results highlighted filtering structures, graph-based methods, and quantization, with streaming results later corrected after a recall-calculation error.
4.1 Filtered Search Track
- The top filtered-search result was more than 11x faster than baseline, with the organizers attributing leading performance mainly to better filtering structures.ParlayANN used tag-primary indexing, with different strategies for common and less common tags and for single- versus two-tag queries.Source: §4.1 · p. 5, Table 2 · p. 6
4.2 Out-Of-Distribution (OOD) Track
- MysteryANN (later RoarANN) and PyANNS were joint OOD winners after extremely close performance.MysteryANN used query-distribution-informed graph construction; PyANNS used a Vamana graph and credited its performance to quantization, VNNI, and adaptive prefetching.Source: §4.2 · pp. 5–6, Table 3 · p. 6, Figure 4 · p. 6
4.3 Sparse Track
- PyANNS and GrassRMA won the sparse track with graph-structured indices, using quantization and refinement or memory-local coordinates and dot-product bounds.PyANNS used HNSW with quantization and a refinement step; GrassRMA co-located sparse coordinates and values and used value bounds to terminate dot-product calculations sooner.Source: Figure 5 · p. 7, §4.3 · p. 7
- On the harder private sparse dataset, PyANNS, shnsw, and sustech-whu were around 25% slower, while NLE was around 45% slower.All submissions built indexes within the 12-hour limit except sustech-whu, which took 14 hours.Source: §4.3 · p. 7
4.4 Streaming Search Track
- Under corrected scoring, PyANNS won streaming with out-of-the-box DiskANN and 8-bit scalar quantization, which allowed deeper graph search.The earlier recall calculation reflected the first runbook snapshot rather than average recall over the runbook.Source: §4.4 · p. 8
5 Discussion
The organizers attribute the wider gap between submissions and the baseline partly to accessible hardware, smaller datasets, and less baseline optimization, among other factors. The discussion also notes limits in task scope, dataset construction, and competition organization.
General remarks
- The submission–baseline performance gap was wider than in 2021; organizers attribute it to five factors, including accessible hardware, 10-million-vector datasets, and less baseline optimization.They interpret the gap as evidence of nontrivial improvement opportunities on several tracks.Source: §5 · p. 8
Discussion
- The filtered track deliberately limited predicates to one or two words, narrowing the competition’s scope while encouraging specialized data structures.Source: §5 · p. 8
- The OOD track encouraged using query samples when constructing the index, as intended, while filtered-track specialization may be less useful in more general settings.Source: §5 · p. 8
Organization glitches
- Reused frozen datasets prevented private query sets for every track; filtered-data construction also missed near-duplicate vectors, making ground-truth ordering arbitrary and introducing recall jitter below 0.00015.Source: §5 · p. 8
- Matching registrations to pull requests was difficult, making some participant affiliations hard to verify; the organizers recommend non-anonymous accounts linked to CMT entries.Source: §5 · p. 8
- The organizers had no published rule defining who counted as an organizer or could compete, prompting ad-hoc decisions for distantly affiliated participants.The paper says clearer rules could have avoided the resulting tensions.Source: §5 · p. 8
6 Conclusion
The conclusion presents the challenge as addressing complex ANN search scenarios and reports improvements in accuracy and efficiency. It highlights graph, quantization, hybrid-indexing, and memory-access advances, alongside resource-efficient and open-source participation.
Conclusion
- The challenge addressed filtered, out-of-distribution, sparse, and streaming searches, with notable accuracy and efficiency improvements from academic and industrial participants.Source: §6 · p. 8
- Key advances included graph-based indexing, quantization, hybrid vector–metadata structures, and efficient memory-access strategies.Source: §6 · p. 8
- The competition fostered broad participation by emphasizing resource-efficient solutions and open-source contributions.Source: §6 · p. 9
- The challenge catalyzed ongoing research, with later work improving its top results, including [6],, and.Source: §6 · p. 9