PaperlayerView exact revision on arXiv

Source-linked AI summary

HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditions

Yuichi Tateno

arXiv:2606.22778v1cs.IRcs.CL

TL;DR

Existing retrieval benchmarks are difficult to rerun for consistent comparisons of models and efficiency settings. HAKARI-Bench provides lightweight, unified Nano-sets, whose rankings reproduce official evaluations with Spearman correlations above 0.97.

  • Problem

    Existing benchmarks are large and rarely compare models and efficiency settings consistently under identical conditions.

  • Method

    HAKARI-Bench reconstructs retrieval suites as unified Nano-sets spanning multilingual, multidomain tasks and retrieval architectures under shared conditions.

  • Results

    Spearman correlations exceed 0.97 in all three comparisons with official MTEB retrieval v2, MMTEB v2 retrieval, and English BEIR full.

  • Takeaways & Limitations

    HAKARI-Bench supports lightweight model-ranking comparison while retaining the need for full evaluation.

  • Takeaways & Limitations

    The benchmark does not measure each model’s inference speed directly, using active and total parameters as rough proxies for inference cost.

Abstract

from arXiv · show

With the rapid spread of retrieval-augmented generation and semantic search, choosing the right embedding and retrieval configuration is increasingly hard. Large retrieval benchmarks are comprehensive but too heavy to rerun during development, and there is little infrastructure for comparing production settings--dimensionality reduction, quantization, reranking--across many models under identical conditions. We present HAKARI-Bench, a lightweight benchmark that reconstructs existing retrieval suites into small datasets (Nano-sets): 35 benchmarks and 551 tasks across 43 languages in a unified format, enabling same-condition, model-agnostic comparison of five retrieval families (BM25, dense, sparse, late interaction, rerankers) and their efficiency variants. Across 55 models, its overall ranking reproduces the official MTEB retrieval v2, MMTEB v2 retrieval, and English BEIR (full) at Spearman >0.97. HAKARI-Bench does not replace full evaluation; it enables rapid model selection, regression detection, and reading the quality-efficiency Pareto frontier. Code, data, and leaderboard are released under the MIT license.

1 Introduction

HAKARI-Bench reconstructs large retrieval suites into lightweight Nano-sets and a unified infrastructure for comparing retrieval architectures, models, and efficiency settings under identical conditions. Its results reproduce established benchmark rankings while supporting repeated, sliced evaluation without replacing full evaluation.

  • Practical use: Nano-sets enable repeated measurement at realistic speed and slicing by language, domain, and query length for use-appropriate model selection.The benchmark is intended to reveal domains a model has not learned rather than maximize only its overall score.
  • Ranking reproducibility: 0.983, 0.975, and 0.973 are the Spearman correlations with MTEB retrieval v2, MMTEB v2 retrieval, and English BEIR (full), respectively.The corresponding Pearson correlations are 0.981, 0.969, and 0.974, and Borda-score correlations are also high.
  • Benchmark infrastructure: HAKARI-Bench reconstructs 35 benchmarks and 551 retrieval tasks as Nano-sets in a unified format for comparing BM25, dense, sparse, late-interaction, and reranker families.Each task contains a corpus, queries, relevance labels, and a top candidate set, enabling candidate generation and reranking evaluation with role-appropriate metrics.
  • Ranking reproducibility: Spearman > 0.97 holds in all three independent comparisons with official MTEB retrieval v2, MMTEB v2 retrieval, and BEIR (full) rankings.The comparisons use common models and intersecting tasks, while HAKARI-Bench remains complementary to full evaluation.
  • Efficiency evaluation: Dimensionality reduction, int8/binary quantization, sparse pruning, and rescoring are evaluated across models under the same conditions to expose quality-efficiency tradeoffs.These settings serve as reproducible proxies for storage and retrieval cost; inference speed itself is not measured because fair measurement is difficult.

2 Related Work

Existing retrieval benchmarks broadened coverage and reduced evaluation cost, but comprehensive, same-condition comparison of retrieval architectures, reranking, and efficiency settings remains limited. HAKARI-Bench builds on lightweight ranking proxies while extending comparisons across languages, expert domains, and efficiency configurations.

  • Retrieval benchmarks: MTEB standardized embedding evaluation, while MMTEB expanded it beyond 250 languages and 500 tasks with quality review and correlation-based downsampling.BEIR established an 18-dataset zero-shot information-retrieval suite; MIRACL, MS MARCO, CoIR, LongEmbed, FollowIR, and IFIR target additional languages, query types, domains, or tasks.
  • Retrieval benchmarks: RTEB adds multilingual retrieval evaluation across legal, finance, code, and medical production domains using public and private datasets.Its use of closed datasets is intended to improve robustness to training-data contamination and leaderboard overfitting, while remaining focused on embedding-model evaluation.
  • Retrieval architectures: Retrieval systems generally separate candidate generation from reranking, covering BM25, dense bi-encoders, learned sparse models, late interaction, and rerankers.The benchmark scope is restricted to retrieval and reranking to support iterative comparison of candidate generation, reranking, and efficiency settings.
  • Lightweight evaluation: NanoBEIR reduces each BEIR dataset to about 50 queries × up to 10K documents, using BM25 and dense-model sampling to reduce evaluation cost.MMTEB also demonstrates low-cost task reduction through correlation-based downsampling while seeking conclusions close to full evaluation.
  • HAKARI-Bench’s position: HAKARI-Bench extends small-collection ranking proxies to non-English languages, expert domains, fixed candidate sets, and dimensionality-reduction and quantization comparisons.Existing infrastructure rarely supports same-condition, cross-model measurement of efficiency effects; HAKARI-Bench evaluates rerankers on one fixed hybrid candidate set across 551 retrieval tasks.

3 Design of HAKARI-Bench

HAKARI-Bench is a five-stage evaluation infrastructure that standardizes task specification, candidate sets, evaluation, result storage, and leaderboard aggregation. Its Nano-set task design enables low-cost comparisons across retrieval families and efficiency variants while fixing candidate coverage for reranking evaluation.

  • Pipeline: HAKARI-Bench uses a five-stage pipeline spanning declarative task specification, a common task format, evaluation, unified result records, and leaderboard aggregation.Results are stored in a single schema and aggregated in DuckDB with macro/micro averages and multi-axis filters.
  • Common task format: Each task aligns a corpus, queries, qrels, and a fixed candidate set, normally the hybrid top 100 formed by fusing BM25 and dense retrieval with RRF.This common interface supports candidate-generation and reranking comparisons under the same task structure.
  • Evaluation: The benchmark runs BM25, dense, sparse, late-interaction, and reranker families alongside dimensionality reduction, int8, binary, rescore, and sparse-pruning variants on identical tasks.Candidate-generation methods retrieve from the whole corpus, whereas rerankers reorder the shared candidate set.
  • Nano-set task design: Nano-sets shrink source evaluations to about 50–200 queries and about 1K–10K documents, lowering the cost of repeated evaluation.The benchmark contains 35 benchmarks and 551 retrieval tasks, covering multilingual, expert, long-document, reasoning, and code-oriented families.
  • Reranking candidate sets: 100% query coverage is enforced by appending one positive at rank 101 only when the top 100 candidate set contains no positive.This safeguard isolates reranker evaluation to ranking accuracy over candidates rather than failures caused by missing relevant documents.

4 Evaluation Methodology

HAKARI-Bench evaluates five retrieval roles under a shared task specification and result schema, using stored top-100 rankings to support multiple metrics and robust aggregation. It also derives efficiency variants from base embeddings to compare quality–efficiency tradeoffs, while treating post-hoc quantization as a robustness measure rather than an advanced production ANN method.

  • The evaluation modes follow BEIR’s five retrieval architectures, and every mode consumes the same task specification and produces the same result schema.
  • BM25 evaluates stored top-100 lexical results, while dense retrieval searches the full corpus and reports the better task nDCG@10 from cosine or inner-product similarity.
  • Rerankers directly score query–document pairs to reorder candidate sets, and dense, sparse, and late-interaction models can be evaluated in the same reranking role.
  • Efficiency variants are derived after one base-embedding inference through truncation, per-dimension int8 or binary quantization, and floating-point rescoring of the top 100 quantized results.Truncation preserves leading dimensions; int8 uses 256 scalar levels calibrated only on corpus embeddings, while binary retains each dimension’s sign.
  • The leaderboard exposes a quality–efficiency Pareto frontier, with each variant stored separately and its “delta vs. base” showing quality degradation.The frontier identifies settings that cannot improve quality or efficiency without worsening the other axis.
  • nDCG@10 is the default metric, while stored per-query top-100 rankings permit recomputation of nDCG, recall, accuracy, MRR, and MAP; macro averages prevent MNanoBEIR task counts from dominating.MNanoBEIR is hierarchically averaged over languages and datasets before contributing to the primary per-benchmark macro average.

5 Results

HAKARI-Bench closely reproduces official retrieval rankings while exposing quality-efficiency tradeoffs and architecture choices under consistent conditions. Its results support rapid model selection and regression detection, but not replacement of full evaluation or fine-grained absolute-score comparison.

  • Dense-model results: 64.93 is the highest dense-model macro average, achieved by jinaai/jina-embeddings-v5-text-small, followed by v5-text-nano at 63.80.The next models are microsoft/harrier-oss-v1-0.6b at 63.68, perplexity-ai/pplx-embed-v1-0.6b at 63.64, and google/embeddinggemma-300m at 62.58.
  • Efficiency variants: −6.50 points is the mean nDCG@10 quality change for binary quantization, compared with −1.95 for int8, −0.93 for binary_rescore, and −0.09 for int8_rescore.Rescoring therefore makes int8 nearly lossless and substantially reduces binary’s quality drop.
  • Reranker results: 68.03 is the only reranker score exceeding the best dense score of 65.51 across all 551 tasks, achieved by Qwen/Qwen3-Reranker-0.6B.This result uses reranking over a fixed hybrid candidate set with a safeguard and is not an end-to-end production retrieval comparison.
  • Ranking fidelity: Spearman rank correlation exceeds 0.97 for Nano-set rankings against MMTEB, BEIR-en, and MTEB-v2 official evaluations.Rank differences average about 1, with maximum differences of 4 for MMTEB, 3 for BEIR-en, and 2 for MTEB-v2.
  • Practical implications: Nano-sets function as low-cost proxies for model selection, separating top and middle groups, and pre-release regression detection, but do not replace official full retrieval evaluation.They do not guarantee absolute-score agreement or reliable fine distinctions between nearby models.
  • Conditional adoption: Target scope changes the best model and architecture: on English BEIR, late interaction ranks first and learned sparse enters the top quartile.Scope-dependent comparisons include language, document length, latency budget, and index size.

6 Discussion

Nano-sets preserve benchmark rankings closely across independent retrieval suites, supporting their use as lightweight ranking proxies. HAKARI-Bench also supports domain-specific model selection and joint quality-efficiency analysis, while production interpretation requires attention to task and system constraints.

  • Ranking validity: 0.975 / 0.983 Spearman correlations and maximum rank differences of 4 / 2 support Nano-sets as ranking proxies on MMTEB v2 retrieval and MTEB retrieval v2.Borda Pearson correlations were 0.969 / 0.981 for the same comparisons.
  • Model selection: 50-plus-point differences across benchmarks show why overall averages should expose domains a model has not learned rather than determine production selection.The discussion attributes these differences to domain variation rather than benchmark difficulty differences.
  • Model selection: NanoMIRACL ranked overall-rank-10 BAAI/bge-m3 first among 38 first-stage systems, while Japanese NanoJMTEB-v2 ranked overall-rank-28 cl-nagoya/ruri-v3-310m first.These results illustrate that models ranked highly for a specific language or domain may differ from the overall leader.
  • Model selection: Filtering by language tag or category supports iterative selection of models strong on tasks close to the intended retrieval situation.The filtering capability is provided specifically to support use-appropriate model selection.
  • Efficiency analysis: Quality, dimensionality, quantization precision, and reranking can be read together as a Pareto frontier rather than a single leaderboard score.The benchmark stores dimensionality-reduction, quantization, rescore, and candidate-set reranker variants side by side.
  • Limitations: Production interpretation must account for task closeness, candidate sets, model size, latency, and memory; scalar/binary post-hoc quantization omits advanced ANN methods.Reranker values assume 100% query coverage and exclude degradation when candidate generation misses positives, which should be assessed separately.

7 Limitations

HAKARI-Bench’s Nano-sets can make retrieval easier than full-corpus evaluation and produce noisy fine-grained comparisons. Its scope is further limited by simplified candidate sets, model coverage, possible contamination, prompt handling, and omitted inference-speed measurements.

  • Nano-set difficulty: Nano-sets shrink corpora to about 10K documents, easing incidental matches; fever_hard_negatives scores shift from official 27.5–92.9 to Nano 74.1–99.1.Hard-negative-dependent tasks therefore may tilt toward the easy side relative to full benchmarks.
  • Nano-set noise: At 50–200 queries, averaged macro 95% bootstrap confidence-interval half-width is ±2.1 points, reaching ±2.3, so nearby-model distinctions are not guaranteed.The estimate uses 2,000 resamples over tasks within each benchmark for 33 dense models.
  • Candidate-set limitations: Candidate safeguards provide 100% query coverage but only about 87% relevant-document coverage on dense retrieval, unlike production two-stage retrieval.This simplification preserves reranker evaluation meaning but makes results dependent on candidate-set construction.
  • Model coverage: The benchmark mainly covers open-published models of about 1B parameters or fewer and excludes paid commercial APIs, limiting direct comparisons with very large or commercial models.Current targets include 33 dense, 4 learned sparse, 6 late-interaction, and 11 reranker models.
  • Contamination: Training-data overlap and near-web-scale pretraining may indirectly advantage models on evaluated benchmarks, while contamination status is not systematically verified.Examples include Multilingual E5 and BGE-M3 using MS MARCO and MIRACL data; this is not necessarily strict memorization of evaluation positives.
  • Prompt and efficiency limitations: The benchmark applies at most one query and one document prompt per model and omits inference speed, whose measurement depends on hardware, batching, sequence length, and parallelism.It evaluates reproducible efficiency proxies such as embedding dimension, quantization precision, and non-zero sparse dimensions instead.

8 Conclusion · A Nano-set construction and dataset list

HAKARI-Bench is presented as lightweight infrastructure for evaluating multilingual, multi-domain retrieval models, rerankers, and efficiency settings under unified conditions. The conclusion identifies model expansion, full-corpus verification, and improved Nano-set down-sampling as future work.

  • 8 Conclusion: HAKARI-Bench evaluates multilingual, multi-domain retrieval models in a lightweight benchmark.
  • 8 Conclusion: It unifies MTEB, MMTEB, BEIR, MIRACL, and NanoBEIR contexts in one infrastructure.
  • 8 Conclusion: Candidate-generation and reranking methods are evaluated according to their respective roles under the same conditions.
  • 8 Conclusion: Dimensionality reduction, quantization, and sparse pruning are compared side by side in the same table.
  • 8 Conclusion: Future work expands evaluation to public models over 1B parameters and commercial APIs.
  • 8 Conclusion: Future work extends real-data model-adoption use cases to full-corpus verification.
  • A Nano-set construction and dataset list: Future work improves Nano-set down-sampling, whose current shrinking fills task corpora without hard negatives with filler documents.
  • 8 Conclusion: The benchmark compares multilingual, multi-domain retrieval models, rerankers, and efficiency settings under the same conditions.

A.1 Benchmark/task list … B Evaluation protocol details

The benchmark organizes 35 benchmarks and 551 retrieval tasks across five retrieval families, with broad language and code coverage. Its versioned Nano-set protocol improves reproducibility but can alter score comparability and increase evaluation uncertainty relative to original benchmarks.

  • A.1 Benchmark/task list: 35 benchmarks contain 551 retrieval tasks classified into five retrieval families.Table 4 summarizes each benchmark’s natural-language tasks, code tasks, languages, and source benchmark.
  • A.1 Benchmark/task list: 526 tasks use natural language, while 25 use code across five benchmarks.NanoCoIR and NanoCodeRAG are code-only; NanoBRIGHT, NanoRTEB, and NanoRARb mix code with natural-language tasks.
  • A.1 Benchmark/task list: Natural-language tasks span 43 languages in total.Language coverage is reported across the natural-language portion of the 551-task collection.
  • A.2 Dataset versions and sources: Dataset commits can be specified explicitly, and the resolved SHA is recorded in result files.This allows results from different dataset contents to be distinguished over time.
  • A.2 Dataset versions and sources: Nano-sets reference published Nano-family datasets on Hugging Face by name and version instead of re-implementing shrinking procedures.Each shrinking procedure follows the original paper or distributor’s description.
  • A.3 Known differences from Nano-set construction: Sampling and retrieval-space reconstruction mean Nano-set absolute scores and variance do not necessarily match the original benchmark.Combining multiple subsets into a 10K-document corpus can compress the score range.
  • A.3 Known differences from Nano-set construction: Limited query counts produce larger standard errors for Nano-set evaluation values than for the original benchmark.This is a stated consequence of Nano-set construction.

B.1 Metric definitions … D.2 Per-model ranking tables

HAKARI-Bench standardizes metric computation, aggregation, model coverage, prompts, and execution conditions, then validates Nano rankings against official benchmarks under complete, comparable model-task sets. Per-model tables show close rank preservation, while Nano scores can shift because corpus/task compression changes the retrieval space.

  • B.1 Metric definitions: nDCG@10 and accuracy@100 are stored with top-100 rankings, enabling downstream recomputation of nDCG@100, recall@{10, 100}, accuracy@{1, 10, 100}, MRR@10, and MAP@100.The benchmark restricts k to {1, 10, 100} and uses binary relevance labels.
  • B.2 Aggregation method: Macro averages are the paper’s primary reporting basis, while the leaderboard defaults to micro averages and permits switching to macro.Per-benchmark results are simple task averages, and cross-benchmark aggregation co-reports equal-weight micro and macro averages.
  • B.3 Handling missing tasks: Rankings include only model–variant pairs with the complete expected task set, excluding models with unevaluable tasks.This complete-model rule keeps comparisons aligned within the selected display range.
  • C Models, prompts, and execution environment: The evaluation targets 55 models spanning dense, learned sparse, late-interaction, BM25, cross-encoder reranker, and LLM-style reranker families.The fixed DuckDB snapshot contains 57 models, but two unreleased dense models are excluded from analyses, aggregations, and figures.
  • C Models, prompts, and execution environment: Official query/document prompts are applied uniformly across all 551 tasks, but this can disadvantage models designed for fine-grained per-task instruction switching.The comparison therefore reflects performance under uniform application of each model’s official base prompt.
  • D.1 Common model set: Spearman 0.973, Borda Pearson 0.974, max rank difference 3 characterize NanoBEIR-en versus full BEIR across 19 common models and 13 tasks.Comparisons use models whose corresponding tasks occur in a single official revision, ranking only over tasks common to both sides.
  • D.2 Per-model ranking tables: For MMTEB, 18 of 24 models differ by at most one rank, while the largest difference is −4; the top model remains rank 1 in both rankings.Nano means are about 7 points lower on average, mainly because NanoMMTEB-v2 compresses tasks into 10K-document combined corpora.
  • D.2 Per-model ranking tables: For MTEB-v2, 16 of 18 models differ by at most one rank, all by at most two, and the top model remains rank 1 in both rankings.Unlike MMTEB, Nano means are about 7 points higher on average because hard-negative pools and corpus caps compress extreme low-score regions.

D.3 Per-task mean/variance differences … E.3 Candidate-set construction

HAKARI-Bench is intended as a relative-ranking proxy rather than an absolute-score substitute: per-task scales and variances shift with corpus construction, sampling, and evaluation policy, while overall rankings remain highly stable. Its efficiency and reranking evaluations standardize variant definitions, sparse-pruning controls, and fixed hybrid candidate sets for fair comparison.

  • D.3 Per-task mean/variance differences: Per-task means and standard deviations shift because query/document counts, qrels density, candidate-pool difficulty, and subset mixing alter score scales, despite preserved model ordering.Nano-sets should therefore be interpreted as rank proxies rather than absolute-score substitutes.
  • D.3 Per-task mean/variance differences: Official means of 64.2 / 65.5 for belebele / mlqa drop to Nano means of 17.1 / 13.2 after multiple language pairs or subsets are combined into one 10K-document corpus.Variance can also move in opposite directions: mlqa compresses to 27% of official standard deviation, whereas belebele widens to 1.11×.
  • D.3 Per-task mean/variance differences: 27–51% standard-deviation compression occurs for several hard-negative or capped-corpus tasks, while fever_hard_negatives’ score range shifts from official 27.5–92.9 to Nano 74.1–99.1.The passage attributes this to candidate-pool and qrels design rather than an intrinsically easier task.
  • D.3 Per-task mean/variance differences: Task-specific instability requires caution: lembpasskey has task Spearman 0.662 and maximum rank difference 20, while domain-query standard-deviation ratios reach 1.05–1.18 without changing overall Spearman 0.983.For saturated or low-score tasks, bootstrap confidence intervals or query-level success-rate distributions are more appropriate than standard-deviation ratios.
  • D.4 Bootstrap confidence intervals of macro ranking: 95% task-bootstrap confidence-interval half-width averages ±2.1 points and reaches ±2.3, with neighboring intervals overlapping substantially; model pairs separated by about 1 point or more virtually never swap ranks.The bootstrap recomputes macro averages 2,000 times by resampling tasks with replacement within each benchmark.
  • E.1 Variant list: Dense efficiency variants comprise base, truncate, int8, binary, rescore, and their cross product, with standard runs auto-deriving int8, binary, and rescore unless explicit variants are specified.Explicit enumeration is required for Matryoshka comparisons across standard dimensions, reduction, and quantization.
  • E.2 Sparse pruning settings: Sparse pruning independently controls query-side and document-side maximum active dimensions, linking query settings to search latency and document settings to latency plus index and embedding-matrix footprint.The provided implementation passage also states that calibration uses only the corpus-side distribution, with query values clipped to the same range and no separate calibration samples or training.
  • E.3 Candidate-set construction: Rerankers use a fixed hybrid candidate set formed by fusing BM25 and dense top-500 lists with RRF using rrf_k=100, then taking the RRF top 100.The set is stored once per task and reused across models; queries lacking a positive receive one appended at rank 101, yielding 100% query coverage.

E.4 Candidate coverage and reranker / dense comparison · F Real-data use cases (details)

HAKARI-Bench links retrieval performance to candidate coverage, query/document characteristics, and reranker type, revealing when dense models or rerankers are preferable. Its same-harness, multi-model and multi-task measurements also support practitioner adoption decisions beyond a single overall leaderboard.

  • E.4 Candidate coverage and reranker / dense comparison: 100.0% query coverage and 86.6% relevant-document coverage were observed on average across 33 dense models.Query coverage was guaranteed by the safeguard, while relevant-document coverage measured relevant documents appearing among top candidates.
  • E.4 Candidate coverage and reranker / dense comparison: On multilingual tasks, BAAI/bge-reranker-v2-m3 (+1.21) exceeded jinaai/jina-embeddings-v5-text-small (+1.09), while English tasks favored Qwen3-Reranker-0.6B (+1.32) over the best dense (+1.11).The multilingual comparison places the best dense immediately behind the leading cross-encoder; the English comparison places the LLM-style reranker first.
  • E.4 Candidate coverage and reranker / dense comparison: bge-reranker-v2-m3 favored short tasks (+1.10) over long tasks (+0.06), whereas Qwen3-Reranker-0.6B reached +1.62 on long tasks.Cross-encoders were described as short-favored, while the LLM-style reranker remained strong on long text and reasoning-oriented scopes.
  • E.4 Candidate coverage and reranker / dense comparison: At query lengths of 300+ characters, bge-reranker-v2-m3 had mean −12.5 points and a 24% win rate, while Qwen3-Reranker-0.6B retained a 57% win rate.Below 300 characters, bge-reranker-v2-m3 was consistently positive with 50–67% win rates; Qwen3-Reranker-0.6B had no length penalty.
  • E.4 Candidate coverage and reranker / dense comparison: Short factual queries favored multilingual cross-encoders, while long reasoning or instruction queries favored dense or LLM-style rerankers.Examples included short factual NanoMIRACL, Japanese, and Korean queries, versus paragraph-length NanoBRIGHT psychology and long NanoR2MED clinical queries.
  • E.4 Candidate coverage and reranker / dense comparison: Long-document multilingual retrieval favored cross-encoders and LLM rerankers, whereas similarity-type scidocs tasks favored dense models.Dense models compress documents into one vector, while rerankers jointly read query and document subject to input-window truncation.
  • F Real-data use cases (details): Six use cases frame model-adoption guidance as three practitioner questions.The examples use measured values from the benchmark results to expose decision material unavailable from a single overall leaderboard.
  • F Real-data use cases (details): Same-harness measurement across many models, tasks, architectures, and efficiency settings makes the use-case results reproducible from the shared benchmark outputs.Queries and derivation scripts are provided in the repository, with aggregation unified to per-benchmark macro averages.

F.1 Retrieval: the best model and architecture depend on the scope · F.2 English NanoBEIR: late interaction and learned sparse become first-class choices · F.3 Reranking: the reranker advantage concentrates in the semantic-search scope

HAKARI-Bench shows that retrieval winners depend on evaluation scope: multilingual semantic search favors tuned multilingual dense models, while long documents favor BM25. English-only evaluation elevates late interaction, and reranking provides its clearest advantage in semantic-search comparisons.

  • F.1 Retrieval: the best model and architecture depend on the scope: No single retrieval system dominates every scope, and winners span multiple architectures.The benchmark ranks 38 first-stage retrieval systems overall and by representative benchmarks.
  • F.1 Retrieval: the best model and architecture depend on the scope: BAAI/bge-m3 ranks first on multilingual NanoMIRACL, followed by intfloat/multilingual-e5-large at rank 3.NanoMIRACL covers 18 languages with short queries and passages fitting a 512-token window; their overall ranks are 10 and 15, respectively.
  • F.1 Retrieval: the best model and architecture depend on the scope: BM25 ranks first on both long-document series despite an overall rank of 24.NanoMLDR documents are approximately 5K–28K characters and NanoLongEmbed documents approximately 28K–326K characters.
  • F.1 Retrieval: the best model and architecture depend on the scope: Dense models can truncate long documents, whereas BM25 matches the whole document lexically regardless of length.Truncation may leave relevant passages outside the maximum sequence length.
  • F.2 English NanoBEIR: late interaction and learned sparse become first-class choices: On the 13 English NanoBEIR tasks, lightonai/ColBERT-Zero leads with 67.97 and lightonai/GTE-ModernColBERT-v1 follows with 67.47.The leaderboard ranks 44 systems by micro average, including 33 dense, 4 learned sparse, 6 late-interaction, and 1 BM25 system.
  • F.2 English NanoBEIR: late interaction and learned sparse become first-class choices: English-centric ColBERT-family late interaction and SPLADE-family learned sparse models perform poorly on multilingual macro but become first-class choices in NanoBEIR-en.Restricting evaluation to English exposes their performance rather than hiding it in aggregate results.
  • F.3 Reranking: the reranker advantage concentrates in the semantic-search scope: Qwen/Qwen3-Reranker-0.6B is the only reranker exceeding the dense top overall, scoring 68.03.Scores use a fixed hybrid candidate set, allowing direct comparison between embedding models and rerankers; dense models occupy ranks 2–6.
  • F.3 Reranking: the reranker advantage concentrates in the semantic-search scope: Overall reranking is led by a modern general reranker, while dense embedding models fill the next five ranks.The comparison covers 54 models excluding BM25 and uses macro averaging with a safeguard.

F.4 Dimensionality reduction and quantization: mild, uniform, and model-specific costs

Dimensionality reduction causes mild, largely uniform quality loss when measured by native-dimension ratio, while int8 quantization imposes a small, consistent cost. Binary quantization is substantially more model-specific, with severe degradation for multilingual-E5 models but limited losses for models trained for quantization robustness.

  • Dimensionality reduction: About 99% of base macro is retained at 50% native dimension, falling to about 95% at 25%.Retention should be compared by native-dimension ratio rather than absolute dimensions.
  • Dimensionality reduction: 96% retention is achieved by jinaai/jina-embeddings-v3 at 12.5% native dimension, using 128 dimensions from a native 1024.
  • int8 quantization: −1.90 is the 33-model average int8 cost, with a worst-case drop of −3.25.int8 changes each dimension from a 4-byte float to 1 byte, reducing storage to about 1/4.
  • Binary quantization: −6.87 is the average binary-quantization degradation, spanning −2.01 to −35.79 across models.The wide range indicates a model-specific rather than uniform cost.
  • Binary quantization: −35.8, −20.7, and −17.9 are the binary drops for multilingual-E5 small, base, and large, while an E5-derived bilingual model drops −16.2.The five E5/E5-derived models average about −19, far below the other group’s roughly −2 to −10.
  • Binary quantization: Within 2–4 points is the binary degradation for quantization-robust families including jina-embeddings-v5, embeddinggemma-300m, snowflake-arctic-embed-l-v2.0, and Qwen3-Embedding-0.6B.Binary degradation is not explained by model size or embedding dimension; the correlation with dimension is weak at +0.32.

F.5 float rescore: an operation that preserves cross-model comparison, and its exception · F.6 learned sparse pruning: the document side is a cheap knob, the query side an expensive knob · G Availability and licensing

Float rescoring largely preserves cross-model rankings after quantization, except that binary quantization can severely collapse some models without rescoring. Learned sparse pruning is asymmetric: document dimensions can shrink aggressively, while query caps are more compute-sensitive; the project releases code and Nano-sets under stated licensing conditions.

  • F.5 float rescore: an operation that preserves cross-model comparison, and its exception: Rescoring the top 100 quantized-embedding results with pre-quantization float embeddings restores cross-model comparisons toward the float ranking.The operation is evaluated for model selection rather than per-model quality recovery.
  • F.5 float rescore: an operation that preserves cross-model comparison, and its exception: 0.995 Spearman correlation for int8 alone and 1.000 for int8 + rescore show that int8 nearly preserves the float ranking across 33 dense models.The float leaderboard can therefore be used as-is for int8 model selection.
  • F.5 float rescore: an operation that preserves cross-model comparison, and its exception: 0.937 Spearman correlation for binary alone rises to 0.988 with binary + rescore, showing that rescoring largely repairs binary-induced ranking disruption.Binary alone substantially disturbs cross-model superiority, whereas binary + rescore returns it near the float ordering.
  • F.5 float rescore: an operation that preserves cross-model comparison, and its exception: 58.18 →40.27 for mE5-large and 53.60 →17.82 for mE5-small show severe binary collapse, unlike jina-v5-small’s 64.93 →62.92 decline.Binary + rescore recovers robust examples near float, including jina-v5-small at 64.85 and bge-m3 at 59.62.
  • F.5 float rescore: an operation that preserves cross-model comparison, and its exception: 88–98% of base is retained by 11 MRL-capable models at 256 dimensions + binary + rescore, using 32 bytes/vector versus float 1024 dimensions.Under this setting, the float-best model remains a valid binary-operation choice, and jinaai/jina-embeddings-v3 is the most robust model.
  • F.6 learned sparse pruning: the document side is a cheap knob, the query side an expensive knob: 100.6% at 512 →256 and 99.1% at 512 →128 show that document-side pruning is aggressive, while query-side pruning reaches 99.5% at q=32 →24 but drops about 3% at q=16.The ≥99% operating envelope is q≥24 and d≥128; d=64 causes about a 5.5% drop and q=8 about 10%.
  • F.6 learned sparse pruning: the document side is a cheap knob, the query side an expensive knob: 99.4% of base is retained at d=256 and q=24, limiting index size and search-time compute with an effectively 0.6% quality drop.Document-side d controls storage/index size, whereas query-side q controls posting-list processing and latency.
  • G Availability and licensing: The evaluation and visualization code is released on GitHub under the MIT license, while Nano-sets are released on Hugging Face Datasets under their original dataset licenses.Users must comply with each original source’s license terms; NanoBEIR-{lang} reuses already-published Nano-sets for consistency.
Loading 2606.22778v1…