PaperlayerView exact revision on arXiv

Source-linked AI summary

The Commercial Tax: Rent-vs-Own Blind Spots in Multi-Hop Retrieval Benchmarks

Luis M. Sanchez, Kosrow Dehnad

arXiv:2608.16096v1cs.IRcs.CL

TL;DR

Multi-hop retrieval benchmarks omit commercial deployability and build cost, leaving buyers without conditions needed to interpret published numbers. This paper audits licensing, measures embedders on a common MuSiQue harness, and models costs, finding that a commercial self-hostable embedder now matches the research anchor while graph construction dominates costs.

  • Problem

    Multi-hop retrieval benchmarks omit whether retrieval backbones are commercially deployable and what indexing costs, despite those conditions determining whether published numbers answer buyers’ questions.

  • Method

    The paper audits licensing and costs across five systems, then evaluates thirteen embedders with a fixed dense-retrieval MuSiQue harness and bootstrap confidence intervals.

  • Results

    A commercial, self-hostable embedder matches the non-commercial anchor, while embedding and annual answering costs remain 7.5x–900x and 350x or more below graph-construction costs.

  • Takeaways & Limitations

    Commercial deployment no longer requires a measurable retrieval-quality penalty on this corpus, but buyers should distinguish free self-hosting from recurring API charges and undisclosed graph-construction costs.

  • Takeaways & Limitations

    The cost model excludes human labor, while the benchmark evidence comes primarily from one MuSiQue corpus, limiting broader cost and performance generalization.

Abstract

from arXiv · show

Enterprises connect language models to their own data through retrieval. The benchmarks that rank multi-hop retrieval systems leave out two facts a buyer needs before a published number can be used: whether the retrieval backbone may be deployed commercially, and what it costs to build. On licensing: the field's dense-retrieval anchor, NV-Embed-v2, is licensed cc-by-nc-4.0. Of the four leading MuSiQue systems we audit (HippoRAG-2, PropRAG, SAG, KET-RAG), three depend on it for their best numbers and none says so. On performance: we measure thirteen embedders from eight makers on one identical MuSiQue harness with bootstrap confidence intervals throughout. Until mid-2026 there was a real commercial tax: the best commercially-licensed embedder trailed the anchor by 2.31 Recall@5 points (95% CI [0.91, 3.71], p=0.001). NVIDIA's Nemotron-3-Embed-8B, released 2026-07-16, has closed it: +0.24 at Recall@5 (95% CI [-0.94, +1.43], p=0.69), -0.58 at Recall@10 (p=0.28). It matches the anchor, does not beat it, and is the only entrant that is commercially licensed, free to self-host, and indistinguishable from the anchor; every other entrant meeting the first two conditions sits 5.2 to 14.6 points below. The durable finding is the paid-versus-free divide: API embedders charge per token on every re-index, self-hosted ones charge nothing. On cost: three of five audited systems (adding Microsoft's GraphRAG) do not disclose indexing cost, and the only published GraphRAG dollar figures span 11x inside one third-party paper (USD 2.30 vs USD 24.94 to index a 5.64 MB corpus once); extrapolated to 1 TB that undisclosed choice separates roughly USD 428K from $4.6M. Our cost model keeps one-time embedding apart from recurring answering: at 1 TB, embedding sits 7.5x-900x below graph construction, and a year of answering at 10,000 queries/day sits 350x or more below it.

1 Introduction

Enterprise AI relies heavily on retrieval, but multi-hop benchmark results omit two buyer-critical conditions: whether components may be deployed commercially and what system construction costs. This paper audits those licensing and cost blind spots without claiming the underlying research or protocols are invalid.

  • Motivation: 86% of organizations use augmented LLMs versus 14% generic models, while vector-database usage grew 377% year over year on Databricks.These figures support the paper’s motivation that retrieval is central to enterprise AI adoption, though the paper relies on their direction rather than their digits.
  • Benchmark blind spots: Multi-hop benchmarks can report transferable-looking numbers while omitting whether retrieval components are commercially deployable and what they cost to build.The paper centers this issue on the MuSiQue closed-corpus protocol standardized by HippoRAG-2.
  • Licensing: NV-Embed-v2 is the standard dense-retrieval anchor, yet its cc-by-nc-4.0 license permits non-commercial use only.HippoRAG-2, PropRAG, and SAG are among the leading systems audited for reliance on this backbone.
  • Cost disclosure: PropRAG discloses about $4 for the shared corpus and KET-RAG $1.89, while SAG reports no figure and GraphRAG’s $33K figure is a third-party extrapolation.The paper argues that published text therefore cannot answer what deployment would cost an organization.
  • Research questions: The paper asks whether benchmark embedding models are licensed for commercial use, how commercial alternatives compare in retrieval quality, and whether systems disclose license and cost.It treats these as measurable gaps rather than evidence that NV-Embed-v2 research, evaluation protocols, or papers are invalid.

2 Background

This section motivates MuSiQue as a response to weaknesses in HotpotQA and establishes the evaluation corpus and protocol. It also identifies NV-Embed-v2 as the dense-retrieval floor while distinguishing commercial licenses from research-only licensing.

  • Benchmark background: MuSiQue was built to address a known weakness of HotpotQA, an earlier multi-hop benchmark with 113,000 crowdwritten questions over Wikipedia article pairs.HotpotQA labels the sentences humans used to answer each question, but the supplied passage does not specify the weakness beyond noting that some questions lacked genuine multi-hop reasoning.
  • Evaluation corpus: 11,656 passages totaling 5.64MB of raw text, with 79.8 average words per passage, anchor the paper’s cost extrapolations.The corpus measurement is explicitly identified as load-bearing for §3.4.
  • Evaluation protocol: 1,000 released MuSiQue questions are used unchanged, making the 69.55 result directly comparable to the release table’s 69.7.The protocol preserves the distributed question order and aligns released per-question vectors index-by-index.
  • Retrieval licensing: NV-Embed-v2 is the protocol’s dense-retrieval floor, but its CC-BY-NC-4.0 license prohibits commercial deployment.Apache 2.0 and MIT permit commercial deployment; NVIDIA directs commercial users toward separately licensed, paid NeMo Retriever Microservices.
  • Disclosure gap: Over 70% of 1,800+ audited AI datasets had missing or incorrect license disclosure, motivating model-dependency and cost-disclosure audits here.The paper applies this disclosure discipline one layer down to model dependencies and separately to cost disclosure.

3 Methodology

The study remeasures thirteen embedders on one dense-retrieval MuSiQue harness, audits commercial licensing and backbone disclosure, and models embedding costs separately from production retrieval costs. It uses exact cosine retrieval as an index-free ceiling while accounting for asymmetric API embedding and variant-selection effects.

  • Licensing audit: License verification used model-card metadata, card text, and current provider documentation; passing required commercial deployment without a separate paid license or non-commercial clause.The audit also searched four leading MuSiQue papers and linked repositories for licensing disclosures and found that the relevant license strings were absent from all four.
  • Retrieval evaluation: 1,000 questions and 11,656 passages were embedded without decomposition, rewriting, or reranking, then scored by cosine-similarity Recall@k.The corpus was embedded once, queries online, and top-k passages compared with gold supporting passages.
  • Retrieval evaluation: Thirteen embedders from eight makers were evaluated on one identical harness, including NV-Embed-v2.The panel included provider APIs and self-hosted open-weight models, using each API provider’s documented query/document mode.
  • Variant control: Each entrant’s number of tested variants was reported because selecting the best variant can flatter models beyond the confidence interval’s coverage.NV-Embed-v2 was swept over four query instructions, while Nemotron-3-Embed-8B’s +0.24 comparison used one configuration against the anchor’s best variant.
  • Cost model: Embedding cost was modeled as a one-time, corpus-scaled expense, with self-hostable open-weight models costing $0 per token in provider charges.The model applies published API rates to measured token counts and extrapolates them to larger corpora.
  • Cost model: Exact brute-force cosine retrieval over all 11,656 passage vectors omits approximate-nearest-neighbour indexing, making reported Recall@k an index-free ceiling.The paper treats vector-store costs and deployment-time ANN recall loss as outside this measurement.

4 Results

Nemotron-3-Embed-8B closes the prior commercial quality gap with the research anchor without establishing a new winner, while self-hosting preserves a paid-versus-free cost distinction. The audit also finds undisclosed anchor licensing and substantial variation in published indexing costs.

  • Commercial quality gap: 0.24 Recall@5 points above NV-Embed-v2 and −0.58 Recall@10 points below it, with neither difference significant.Nemotron-3-Embed-8B scores 69.79/77.54 versus NV-Embed-v2’s 69.55/78.12 on the same harness.
  • Statistical interpretation: Only NV-Embed-v2 is indistinguishable from the top entrant among the panel’s twelve anchor-referenced comparisons after Holm–Bonferroni correction.The comparison family excludes the pre-specified primary comparison and corrects the remaining eleven tests.
  • Commercial quality gap: 2.31 Recall@5 points was Gemini embedding-001’s significant deficit versus NV-Embed-v2 before Nemotron-3-Embed-8B’s 2026-07-16 release.The earlier gap had 95% CI [0.91, 3.71] and p = 0.001; the paper concludes the gap has closed, not that Nemotron is a new winner.
  • Licensing disclosure: 0 of 3 systems relying on NV-Embed-v2 disclose its non-commercial license in their papers, limitations, or linked repositories.The license is stated on the model card, so the finding concerns reporting rather than concealment.

5 Discussion

Published retrieval benchmarks can be unusable for buyers when they omit commercial deployability, reproducibility, and indexing cost. The commercial performance tax has closed only at the frontier, while structured-retrieval costs remain substantial and often undisclosed.

  • Commercial performance tax: 2.31 Recall@5 points separated Gemini embedding-001 from NV-Embed-v2 in June 2026, with a 95% confidence interval of [0.91, 3.71] excluding zero.The comparison covers the thirteen measured entrants, and the commercial reference surfaced less gold than the anchor on 184 of 1,000 questions.
  • Commercial performance tax: One commercially licensed embedder reached the anchor one month later, making the tax statistically indistinguishable rather than demonstrably gone.Nemotron-3-Embed-8B was released on 2026-07-16, eight days before the paper’s initial draft was finalized.
  • Commercial performance tax: 9.7–14.6 Recall@5 points separate Qwen3-VL, mxbai, and BGE-M3 from NV-Embed-v2 at the raw-embedder level.This gap exceeds the 1–3-point differences suggested by system-level comparisons that add reranking or graph structure.
  • Methodological implications: Reporting point estimates without confidence intervals, harnesses, or per-item outputs can conceal benchmark defects that materially change conclusions.A one-line index defect understated the anchor by 2.5 points; correction changed the headline from “exceeds” to “matches.”
  • Cost and reporting: Graph-augmented and SQL-structured retrieval costs are frequently undisclosed and can plausibly reach millions of dollars at realistic scale.The paper does not claim Recall or cost figures for any specific commercial structured-retrieval product.
  • Buyer diligence: Three audited systems depend on a cc-by-nc-4.0 embedding model without disclosing that dependency, making their quality figures difficult to transfer legally into production.Deployability concerns apply to the embedding model inside the system, not merely to the system’s overall license.

6 Limitations

The evidence is limited by one benchmark corpus, unresolved GraphRAG cost configurations, changing provider rates, and a machine-time-only cost model. Additional embedding-format checks support the reported comparisons, but do not remove these scope limitations.

  • Measurement scope: Three NVIDIA embedding endpoints were measured, while three others were unavailable under the study’s credentials at measurement time.The panel therefore does not cover all NVIDIA embedding NIM endpoints.
  • Cost ambiguity: Microsoft’s GraphRAG paper names neither cost configuration, leaving readers to choose which figure “the cost of GraphRAG” represents.The study reports both figures because the spread is attributed to chunk-size choice but remains unresolved.
  • Benchmark scope: 2Wiki reduces the commercial reference gap to 0.12 points (95% CI [−1.07,+0.82], p = 0.82), weakening the magnitude claim beyond MuSiQue.The authors narrow the conclusion to this corpus: the tax was real and is now closed, but its original size is corpus-dependent.
  • Cost scope: 2026-07-21 rates are subject to provider change, although the relative structure of self-hosted free, API recurring, and indexing-dominant costs should hold.This limitation concerns the durability of published cost figures rather than the model’s qualitative cost ordering.
  • Cost scope: The cost model prices machine time, including LLM inference and embedding, but excludes human time.This omission limits interpretation of total project cost.
  • Validation: Cosine similarity was 1.0000 for title\ntext matches across all 120 re-embedded samples and three models, while text-only medians were 0.963, 0.989, and 0.987.The comparison confirms the embedding format rather than inferring it.

7 Conclusion

The conclusion argues that multi-hop retrieval benchmarks omit commercial licensing and build-cost information buyers need, while a commercial, self-hostable embedder now matches the non-commercial anchor on the measured corpus. It qualifies that result with limited generalization, reproducibility safeguards, and disclosed commercial interests.

  • Commercial blind spots: Multi-hop benchmarks omit the commercial-licensing information buyers need, including undisclosed non-commercial backbones in three audited systems.The paper traces the restriction to a specific training dataset.
  • Commercial parity: 2.31 Recall@5 points was the commercial-license penalty as recently as June 2026; it is now not measurable.Nemotron-3-Embed-8B matches the anchor rather than beating it and is the only one of twelve commercially deployable entrants to do so.
  • Commercial parity: The durable economic divide is paid API use versus free self-hosting after embedding.API providers impose a permanent per-token toll, while self-hostable open-weight models incur no provider charge per token.
  • Limits and next steps: The measured result does not establish that self-hostable specialists generalize to private documents or domains.The authors are building a released multimodal benchmark and preregister the prediction that a frontier model matching a domain-adapted specialist on unseen data would be the result reported.
  • Limits and next steps: Every entrant used the same fixed harness and released per-question vectors, but the authors disclose commercial interests in system selection and benchmark construction.The ranking is therefore recomputable without re-embedding, while the benchmark’s neutrality cannot be guaranteed by method alone.

Code and Data Availability

The paper releases its measurement code and result artifacts in a persistent Zenodo record, including the full retrieval, evaluation, cost-model, and bootstrap infrastructure. It also preserves the defective indexing script while documenting upstream retrieval for the non-redistributed MuSiQue corpus.

  • Artifacts: All measurement code and result artifacts for Tables 1, 5, and 9 are released in a repository and archived with persistent DataCite DOI 10.5281/zenodo.21972866.The DOI is identified as the citable record; it was deposited on 2026-08-17.
  • Artifacts: The release includes the thirteen-embedder retrieval harness, exact answering-LLM prompts and decoding parameters, standardized cost model, bootstrap code, and computed comparison artifacts.The artifacts include panel_multiplicity.json and panel_anchor_family.json.
  • Reproducibility: The defective indexing script is published unchanged alongside the corrected version, while the MuSiQue corpus is not redistributed and fetch instructions point to the upstream HippoRAG-2 project.This preserves both the defect and the correction for inspection without repackaging the corpus.

A Panel Composition

The panel excludes Anthropic because it does not ship a text-embedding model. Anthropic documentation recommends third-party providers, including Voyage AI, whose voyage-3.5 model is already included.

  • Panel composition: Anthropic is absent because it does not ship a text-embedding model.The paper notes that no Anthropic/Claude embedder exists.
  • Panel composition: The panel records each entrant’s access channel and per-model notes, with maker and license information provided separately.Table 6 covers access channel and model notes; Table 1 contains maker and license details.
  • Panel composition: Anthropic documentation recommends third-party providers, principally Voyage AI, represented here by voyage-3.5.Voyage AI is already part of the panel.

B Worked Metric Example and Format Sensitivity

Recall@5 counts how many gold supporting passages appear among the top five retrieved results, while query-formatting sensitivity varies sharply by model. Qwen3-VL-Embedding-8B changed by 11.6 points from wording alone, whereas BGE-M3 barely moved.

  • Worked metric example: 100% Recall@5 results when all three gold passages land in the top five; two of three yields approximately 66.7%.The Scanderbeg example has three gold passages: “Scanderbeg (opera),” “Orlando furioso (Vivaldi, 1714),” and “Rialto Bridge.”
  • Format sensitivity: 11.6 points: Qwen3-VL-Embedding-8B swung from 48.3 to 59.9 Recall@5 solely because of query-instruction wording.This demonstrates that formatting can materially affect retrieval results even with the model held constant.
  • Format sensitivity: 0.71 points: BGE-M3 changed from 54.22 bare to 54.93 instruction-prefixed Recall@5, indicating minimal format sensitivity.The paper carries 54.93 throughout; its 54.87 precorrection figure differs by the +0.06 of Table 9.

C Realistic Scale: Institutional Data Volumes

A reproducible measurement of two public synthetic legal-diligence matters grounds institutional-scale retrieval in per-matter volumes rather than round-number assumptions. One matter is roughly 225MB of extracted text, while indexing costs far more than embedding and graph-construction extrapolations may underestimate true costs.

  • Measured institutional scale: 225MB of extracted text per matter is roughly 40x the entire 5.64MB MuSiQue corpus used to derive this paper’s rates.The measurement extracts text from every .docx, .xlsx, .pptx, .eml and .txt file in LAB’s two largest diligence matters.
  • Measured institutional scale: 5GB corresponds to about 23 concurrent matters, while 1TB corresponds to roughly 4,650, volumes a mid-sized transactional practice can reach without being an outlier.These mappings use the measured per-matter extracted-text volume.
  • Cost implications: 8 to several hundred times: indexing one matter costs that much more than embedding it, making embedding-provider comparisons the smaller budget line.The indexing range partly reflects the same system priced two ways; GraphRAG’s two reconstructable estimates differ by 8.5x.
  • Extrapolation limits: 185,900x: the 1TB column requires a far larger extrapolation than the 225MB per-matter figure’s 40x step from the measured corpus.If graph construction scales super-linearly, the distortion affects the 1TB estimate more and underestimates the true cost.
  • Measurement caveats: The benchmark is a text-only, per-file, reproducible measurement of synthetic matters, not a survey statistic or a sample of actual engagements.The Library of Congress’s 21 petabytes across 914 million files cannot anchor a text-corpus extrapolation because it spans all content types and is dominated by scanned page images.

D Panel Re-Measurement and Reproduction

The panel’s headline comparison is reproducible: all thirteen entrants reproduced within 0.6 Recall@5 points, while corpus format materially affected nearly every model. Re-measurement also documents expected hosted-endpoint drift and channel-specific deployment details.

  • Corpus format: Nine of ten models incurred a title-text versus text-only penalty of 1.46 to 4.73 points at 95% confidence.The paired bootstrap used B = 105; text-embedding-3-large’s 0.79-point loss was not distinguishable from zero.
  • Corpus format: 2.54 points was Nemotron-3-Embed-8B’s corpus-format penalty, versus 2.46 for NV-Embed-v2, ranking the latter sixth-largest among ten measured models.The largest penalties were 4.73 for nv-embedqa-e5-v5 and 3.92 for Llama-Nemotron-Embed-1B-v2.
  • Reproduction: All thirteen entrants reproduced within 0.6 points on the matched corpus, confirming correct indexing across twelve original-run comparisons and one dedicated anchor replicate.Replication re-executed the same configuration from scratch; the anchor’s replicate was identical on every per-question score.
  • Re-measurement drift: −0.54 was text-embedding-3-large’s largest run-to-run deviation among thirteen entrants, while four rows were reported at ±0.00.Hosted endpoints are not version-pinned, so some drift is expected; three of the four zero rows were exactly zero before rounding.
  • Deployment details: nemotron-3-embed-8b is self-host-only and unavailable on the hosted NIM endpoint, while NIM uses nemotron-3-embed-1b rather than the HuggingFace identifier.Model identifiers are specific to their distribution channel, which matters when reproducing hosted versus self-hosted measurements.

E Extended Cost Notes

The cost analysis anchors deployment to one owned DGX Spark for the full retrieval-and-answering pipeline and uses a tightly specified answering setup. Several estimates remain directional because of model-conversion imprecision, changing API rates, and unavailable candidate models.

  • Deployment anchor: $4,699 DGX Spark anchors the deployment model: one owned 128GB-unified-memory box hosts the full retrieval-and-answering pipeline.An 8B embedder needs roughly 16GB of VRAM and can run on a consumer GPU under $1,000, but the anchor reflects the planned heavier workloads.
  • Answering configuration: gpt-4o-mini answers each question once with temperature=0.0 and max_tokens=32, using deterministic decoding without self-consistency or reranking.Retrieved passages are serialized with titles and text, and answers are scored by maximum SQuAD token-overlap F1 over gold answers and aliases.
  • Estimate limitations: $19.49–$38.99 is a directional GraphRAG reconstruction because Llama-3.3-70B token counts were converted to gpt-4o-mini rates across different tokenizers and extraction behavior.The estimate is explicitly limited by two compounding model differences: tokenization and the amount of entity/relation content extracted.
  • Estimate limitations: $33K is KET-RAG’s extrapolated cost to index a 5GB legal case, while including its excluded rate would expand the 1TB upper bound from $4.64M to approximately $6.9M.The paper retains the conservative MuSiQue-based range because KET-RAG’s figures use a different dataset and chunking.
  • Pricing caveat: $0.075/1M input and $0.30/1M output were the billed batch rates on 2026-07-21, but the model was absent from OpenAI’s public pricing page and should be re-verified.The billing meter agreed with the rate card within 4%; standard rates were $0.15/$0.60.
  • Model availability: Commercial metadata identified nv-embedqa-mistral-7b-v2 as Community License plus Apache-2.0, but its disappearance from the NIM support matrix led the study to treat it as unavailable.It was not served under the study’s API credentials and had no MS MARCO lineage in the cited catalog metadata.
Loading 2608.16096v1…