Reveal Hidden Pitfalls and Navigate Next Generation of Vector Similarity Search from Task-Centric Views

1 Introduction

• On Glint360K-ViT for face recognition, NSG attains 99% synthetic recall@100 faster than RaBitQ [31], yet achieves only 90% label recall and does so more slowly than RaBitQ.On ImageNet, under inner-product distance, synthetic recall@100 reaches 99.9% while label recall falls below 1%.Source: §1 · p. 2
• The Information Loss Funnel attributes task-level degradation to embedding loss, metric misuse, and distribution sensitivity.Distribution-sensitive indexes can lose throughput and end-task accuracy when they are not robust to the vector distribution in use.Source: §1 · p. 2
• The benchmark suite is designed to validate the diagnostic model and assess VSS within downstream tasks rather than synthetic metrics alone.Its contributions connect funnel analysis, task-specific benchmarks, method selection, and future directions.Source: §1 · p. 2
• Iceberg benchmarks eight datasets across four application domains and evaluates 13 VSS methods with synthetic and downstream-utility metrics.The suite includes datasets from image classification, face recognition, text retrieval, and large-scale recommendation, with scales from 1M to 100M vectors.Source: §1 · pp. 2–3
• Iceberg’s decision tree maps dataset characteristics to suitable VSS methods using features such as clustering tightness, vector norms, angular separation, and distance contrast.The authors report that these easy-to-compute features correlate strongly with end-task performance.Source: §1 · p. 3
• Iceberg identifies task-aware, metric-aware, and distribution-aware VSS as directions for bridging benchmarks and applications.The proposed directions include aligning retrieval with downstream objectives, adapting similarity metrics, and adjusting indexes to heterogeneous distributions.Source: §1 · p. 3

2.1 The Background of Vector Similarity Search

• VSS covers ANNS and MIPS, with tree-, hash-, clustering-, and graph-based methods trading search speed, storage, or recall in different ways.Synthetic recall commonly measures overlap between retrieved top-k vectors and ground-truth top-k neighbors.Source: §2.1 · p. 3

2.2 The Current Popular Benchmarks

• Existing benchmarks largely emphasize synthetic recall under fixed metrics, with limited funnel coverage, downstream utility analysis, and explainable method-selection guidance.Iceberg spans ANNS and MIPS, modern embeddings, end-to-end task utility, full-funnel probes, and an explainable decision tree.Source: §2.2 · p. 4

2.3 Mainstream Task-Centric Metrics for VSS

• Task-centric measures differ by application: classification and face recognition assess class or identity matches, text retrieval assesses semantic relevance, and recommendation assesses user-item relevance.Practitioners also consider the trade-off between synthetic recall and query throughput alongside task utility.Source: §2.3 · p. 4

3 Resource Collection for Funnel Analysis

• Iceberg organizes its resource collection around image classification, face recognition, text retrieval, and recommendation, with data processing, method evaluation, and reproducibility described as parts of the workflow.Figure 3 illustrates the workflow; Table 2 summarizes dataset domains, scales, embedding models, and downstream metrics.Source: §3 · p. 4

3.1 Datasets for Important Application

• ImageNet provides 1,281,167 training and 50,000 validation images; category labels support Label Recall@K, which measures matching labels among retrieved results.The benchmark also evaluates DINOv2, EVA02, ConvNeXt, and AlexNet embeddings.Source: §3.1 · pp. 4–5
• Glint360K supplies over 17 million face images across 360,232 identities, with one sampled face per identity and the first 20,000 identities used for testing.Identity IDs serve as task labels, and face recognition uses Label Recall@K.Source: §3.1 · p. 5
• The text benchmark uses 9,250,529 BookCorpus paragraph chunks and 10,000 sampled queries, evaluating whether a relevant paragraph appears in the top K with Hit@K.Stella generates 1024-dimensional text embeddings.Source: §3.1 · p. 5
• Commerce contains 99,085,171 grocery-item records and 64,111 user queries; Matching Score@K evaluates retrieval of preferred items using their popularity scores.The benchmark uses 48-dimensional embeddings generated with ResFlow.Source: §3.1 · p. 6

3.2 Methods Selection for Evaluation

• Iceberg evaluates 13 methods across ANNS and MIPS, including five partition-based methods and eight graph-based methods.The selected partition-based methods are Fargo, ScaNN, DB-LSH, RaBitQ, and IVFPQ; the graph-based set includes HNSW, NSG, Vamana, Mobius, ip-NSW, ip-NSW+, NAPG, and MAG.Source: §3.2 · p. 6

3.3 Reproducibility and Maintenance

• The reproducibility framework provides code, datasets, embedding models, and documentation, while planned maintenance includes updates to models, tasks, and retrieval methods.The authors also plan containerized environments and outline a pipeline for extending the framework to new tasks.Source: §3.3 · p. 7

4.1 Experiment Setup

• Synthetic recall measures overlap with distance-metric ground truth, while Iceberg also evaluates task-centric metrics to assess downstream utility.The evaluation uses both task-centric metrics defined for each task and synthetic recall@K to assess the recall-versus-query-speed trade-off under a given similarity metric.Source: §4.1 · p. 7

4.2 Impact of the Information Loss Funnel

• 71% label recall@100 for ImageNet DINOv2 versus 85% for EVA02 at 99% synthetic recall@100 shows that embedding choice limits downstream utility.DINOv2 label recall@100 also saturates at 71% despite vector recall@100 reaching 99.9%; AlexNet-based embeddings achieve about 21%. On Commerce, matching score@100 first rises and then declines as synthetic recall improves.Source: §4.2 · pp. 7–8, Table 5 · p. 10
• On ImageNet-ConvNeXt, Euclidean ANNS scores roughly 4% higher label recall@100 than MIPS at about 99.9% synthetic recall@100.On BookCorpus, Euclidean ANNS reaches 100% task-centric hit recall while MIPS stays below 50%; on Glint360K-IR101, MIPS reaches 98% label recall@100 versus HNSW’s 92%. Metric affinity to the embedding geometry explains these reversals: Euclidean distance suits ConvNeXt’s class clusters, while inner product suits ArcFace’s angular structure.Source: §4.2 · pp. 8–9, Figure 6 · p. 9
• ScaNN is 1.7× and 3× faster than the leading graph-based method on ImageNet-DINOv2 and Glint360K-IR101, while graph-based methods are faster on BookCorpus and Commerce.On Commerce and BookCorpus, top graph-based indexes reach up to 3× and 4× ScaNN’s speed, respectively; in ANNS, HNSW is 2.7× and 3.2× faster than RaBitQ. The funnel analysis identifies embedding loss as the task-accuracy ceiling, metric misuse as decisive for retrieval accuracy, and distribution sensitivity as primarily governing efficiency, with a non-trivial task-metric effect.Source: §4.2 · pp. 9–10

4.3 Decision Tree For Method Selection

• The decision tree selects inner product when DBI_E ≥ DBI_C and CV ≤ 0.1; otherwise it selects Euclidean distance, then uses RA and RC to choose the VSS method.It favors partition-based methods when RA ≥ 60 or RC ≤ 1.5, and otherwise favors graph-based methods. On Glint360K-IR101, inner product achieves 4% higher label recall@100 than ANNS; on Face-IR101, ScaNN is 3× lower latency than ip-NSW at comparable synthetic recall.Source: §4.3 · p. 11

4.4 Iceberg Leaderboard 1.0

• The Iceberg 1.0 leaderboard evaluates all eight benchmark datasets, comparing query speed at matched synthetic recall and at matched task-centric metric levels.This part introduces the leaderboard evaluation but does not provide its overall ranking results.Source: §4.4 · p. 12, Table 7 · p. 12
• ip-NSW+ is the overall winner over MAG on the task-centric leaderboard, despite MAG’s higher synthetic recall at lower latency.Iceberg integrates downstream utility into its evaluation; users can select methods using synthetic recall–QPS and task-centric recall–QPS curves.Source: §4.4 · p. 12

4.5 Insights & Future Directions

• On ImageNet-ConvNeXt, Euclidean nearest neighbors do not always share the query’s label, and many queries show poorly aligned distance-label relationships.Figure 8 examines label recall when synthetic metric recall@100=99%.Source: §4.5 · p. 12, Figure 8 · p. 13
• Task-aware VSS could improve downstream alignment through label-recall-based early stopping or task-aware proximity ranges, with joint end-to-end optimization as a further direction.Co-optimizing representation learning, VSS components, and downstream tasks could reduce information loss, particularly in complex, large-scale AI applications.Source: §4.5 · pp. 12–13
• Metric-aware VSS remains limited by single-metric designs; MAG [17] supports IP and NN in one index but still requires manual tuning.Future methods should adapt across inner-product, Euclidean, and non-Euclidean spaces; MAG’s manual tuning and limited complex-metric support leave a performance gap with metric-specific methods such as ScaNN on certain datasets.Source: §4.5 · p. 13
• Glint360K’s extreme angular separation and high intra-class distance variability hinder graph-based indices, whereas BookCorpus’s multiple norm directions favor graph pruning over partition-based boundary identification.In BookCorpus, average vector angles remain within 60 degrees; partition-based methods struggle to identify border points and achieve high synthetic metric recall efficiently.Source: §4.5 · p. 13, Figure 9 · p. 13
• Distribution-aware VSS should adapt to dataset geometry because existing methods overfit particular distributions and metrics, while current hybrid approaches lack adaptive mechanisms and concrete guidance.Potential directions include automatic index-parameter tuning, adaptive method stitching, and dynamic distribution-sensitive methods.Source: §4.5 · p. 13

Lessons Learned

• Iceberg’s lessons call for end-to-end evaluation, better alignment of embeddings, metrics, and retrieval strategies, and dynamic optimization tailored to real-world scenarios.The discussion says synthetic-only optimization is insufficient because information is progressively lost across the VSS pipeline and method improvements often fail to yield task-centric gains.Source: §5 · p. 13

Limitations

• The study focuses on Euclidean and inner-product metrics, algorithmic characteristics, and current AI-generated vectors; it leaves niche metrics and commercial-system optimizations outside scope.The authors say the decision tree may need ongoing updates as model landscapes evolve.Source: §5 · p. 14
• The study acknowledges upstream machine-learning information loss but proposes no solution, leaving it as future work.Source: §5 · p. 14

• Iceberg integrates eight large-scale datasets across four domains, eight representation models, thirteen VSS methods, four downstream task metrics, and an explainable decision tree.The suite is designed to shift evaluation from synthetic measures toward downstream task performance and includes variations for empirical analysis.Source: §6 · p. 14