The Commercial Tax: Rent-vs-Own Blind Spots in Multi-Hop Retrieval Benchmarks

1 Introduction

• Industry practitioners interviewed in prior work prefer retrieval over fine-tuning for updateable knowledge bases and transparent sourcing [1], motivating the paper’s focus on enterprise deployment.The introduction treats vendor adoption figures as indicative rather than definitive evidence.Source: §1 · p. 1
• MuSiQue’s closed-corpus benchmark tests whether retrieval surfaces all passages needed to connect multiple facts, a structure the paper says also occurs in institutional corpora.The paper uses Recall@k, the percentage of needed source passages appearing among the top k results, to capture this failure mode.Source: §1 · pp. 1–2
• The paper’s central claim is that benchmark numbers omit whether components may legally be deployed and what they cost to build, leaving buyer-relevant conditions unanswered.It proposes measuring headline retrieval quality and deployment frictions on the same harness, while clarifying that it does not claim protocols are unsound or researchers engaged in misconduct.Source: §1 · pp. 2–3
• HippoRAG-2 [7], PropRAG [21], and SAG [24] rely on non-commercial NV-Embed-v2 for their leading results without disclosing its restriction; KET-RAG [28] instead uses a commercial embedder.NV-Embed-v2 is licensed cc-by-nc-4.0, while KET-RAG uses OpenAI’s text-embedding-3-small.Source: §1 · p. 2
• PropRAG reports about $4 and KET-RAG $1.89 for indexing the shared corpus, while the $33K figure associated with GraphRAG is a third-party extrapolation, not Microsoft’s reported cost.HippoRAG-2 reports token counts from which cost must be derived; SAG and Microsoft’s GraphRAG paper [29] report no dollar figure.Source: §1 · pp. 2–3
• The study asks about exact embedding-model licenses, the retrieval gap to commercial alternatives, disclosure of licenses and costs, and indexing and answering costs at institutional scale.These questions organize the paper’s license verification, benchmark comparison, literature audit, and standardized cost model.Source: §1 · p. 3

2.1 MuSiQue and the HippoRAG-2 Protocol

• Recall@5 is 69.7% for NV-Embed-v2, 74.7% for HippoRAG-2 [7], 78.3% for PropRAG [21], and 80.0% for SAG [24] in the protocol comparison.SAG reports 81.7% (81.71% in its released repository) when its embedder is swapped to NV-Embed-v2, whose license is noncommercial; the table’s stated purpose is to show SAG does not depend on that embedder.Source: §2.1 · p. 3
• MuSiQue [22] composes questions from single-hop chains and filters shortcut-answerable cases, addressing a weakness in HotpotQA [26] and making it, as of early 2026, the hardest widely-used closed-corpus multi-hop benchmark.HotpotQA [26] contains 113,000 crowdwritten questions over pairs of Wikipedia articles, but many two-hop questions could be answered from one passage or surface cues.Source: §2.1 · p. 3
• The protocol uses 11,656 Wikipedia passages and a released sample of 1,000 MuSiQue dev questions; the paper uses that file and order for directly comparable results.The measured corpus is 5.64 MB of raw text, averaging 79.8 words per passage, which supports the paper’s cost extrapolations.Source: §2.1 · p. 3

2.2 Why NV-Embed-v2 Specifically Is NonCommercial

• CC-BY-NC-4.0 prohibits commercial deployment of NV-Embed-v2 [9], while its model card directs commercial users to separately licensed, paid NVIDIA NIMs.Apache 2.0 and MIT permit commercial deployment; NIMs provide hosted inference for the same restricted model rather than a separate license.Source: §2.2 · p. 4
• NV-Embed-v2’s noncommercial restriction is linked to its training data: NVIDIA’s representative attributed it to noncommercial datasets, and the paper identifies MS MARCO from the disclosed training mix.Using NIM changes who operates the GPUs, not the model’s license; Longpre et al. [12] found over 70% of 1,800+ audited AI datasets had missing or incorrect license disclosure, motivating this paper’s model-dependency and cost-disclosure audit.Source: §2.2 · p. 4

3 Methodology

• Recall@k measures the fraction of a question’s gold supporting passages retrieved in the top k after corpus and query embeddings are ranked by cosine similarity.Source: §3 · p. 4

3.1 Task A: License Verification

• Licenses were checked against model-card YAML and text or providers’ current documentation; a model passed only if commercial use required no separate paid license and had no noncommercial clause.Source: §3.1 · p. 4

3.2 Task B: Commercial-Embedder Retrieval Floor

• The retrieval-floor test embeds 1,000 questions and 11,656 passages without rewriting, decomposition, or reranking, then retrieves by cosine similarity and scores Recall@k.The same harness was used for all thirteen embedders, including NV-Embed-v2.Source: §3.2 · p. 4
• Best-of-variant selection was uneven: NV-Embed-v2 had four query instructions, while Nemotron-3-Embed-8B had one configuration, and confidence intervals omit selection uncertainty.The panel spans thirteen embedders from eight makers, but five are NVIDIA models; Nemotron’s single configuration is compared with the anchor’s best of four.Source: §3.2 · pp. 4–5

3.3 Tasks C and D: Literature Disclosure Audits

• The disclosure audit searched four MuSiQue papers for license information and five systems’ papers for indexing costs, including full texts, tables, and appendices.The cost search included Microsoft’s GraphRAG; license and cost searches used explicit terms because both audits report absences.Source: §3.3 · p. 5
• The audited set is a relevancy-seeded snowball sample, so its counts describe only those papers and do not estimate field-wide prevalence.Microsoft’s GraphRAG was added to the cost audit only.Source: §3.3 · p. 5

3.4 Standardized Cost Model: Two Axes, Kept Separate

• Embedding cost is modeled as a one-time, corpus-scaled charge, while answering cost recurs with query volume and uses measured API usage for a fixed LLM.Self-hosted open-weight models have $0 provider charge per token; the model anchors hardware cost to a DGX Spark priced at $4,699 in July 2026.Source: §3.4 · p. 6
• Exact brute-force cosine search avoids ANN-related recall loss as a confound, so reported Recall@k is an index-free ceiling rather than a production figure.The model uses self-hosted vectors on local NVMe; a managed vector service would add recurring storage costs absent from this model.Source: §3.4 · p. 6

4.1 License Verification

• NV-Embed-v2 is licensed cc-by-nc-4.0, while both Nemotron-3-Embed tiers, Llama-Nemotron-Embed-1B-v2, and nv-embedqa-e5-v5 are commercial.mxbai-embed-large-v1 and Qwen3-VL-Embedding-8B are Apache-2.0, BGE-M3 is MIT, and the OpenAI, Cohere, Google, and Voyage APIs are commercial products under their terms of service.Source: §4.1 · p. 6

4.2 The Commercial-Embedder Retrieval Floor

• Under the same harness across the 13-embedder panel, Nemotron-3-Embed-8B scored 69.79/77.54 Recall@5/@10 versus NV-Embed-v2’s 69.55/78.12; differences were +0.24 and −0.58, respectively.Source: §4.2 · p. 6, Table 1 · p. 7
• Nemotron-3-Embed-8B is the only commercially licensed, self-hostable entrant indistinguishable from the anchor; Apache-2.0 and MIT entrants trail by 9.7 to 14.6 Recall@5 points.Recall@10 exceeded Recall@5 for all thirteen embedders, with gains from +7.62 to +10.67 points; BGE-M3 ranked second-to-last at 54.93 Recall@5 / 62.89 Recall@10.Source: §4.2 · p. 8

4.3 Inference and Multiplicity

• Holm-corrected paired comparisons leave NV-Embed-v2 as the sole entrant not separable from the top model, while paired intervals describe question-sampling sensitivity, not cross-corpus uncertainty.The analysis resamples the fixed 1,000 MuSiQue questions as representatives of a MuSiQue-like population; it does not quantify uncertainty across corpora, domains, or workloads.Source: §4.3 · pp. 8–9

4.4 Literature Audit: License and Cost Disclosure

• None of the three audited systems relying on NV-Embed-v2 disclosed its non-commercial license, and only two of five audited systems published dollar indexing figures.KET-RAG reports GraphRAG indexing costs of $2.30 and $24.94 for two configurations on the same corpus, an 11x spread; Microsoft’s GraphRAG paper gives no dollar figure.Source: §4.4 · pp. 9–10

4.6 Embedding Cost: One-Time, at Scale

• At 10GB, API providers charge $46–$346 per full re-index at standard tiers, while self-hosted models have no per-token charge.Batch pricing reduces the cheapest cited re-index to approximately $23; self-hosting still entails a one-time hardware cost anchored at $4,699, and NV-Embed-v2 is not commercially deployable.Source: §4.6 · p. 10

4.7 Answering Cost: Recurring, per Query

• At 1TB, API embedding costs $5.2K–$39.3K against indexing costs of $296K–$4.64M, a factor of roughly 7.5 to 900.At 10,000 queries/day, answering costs $361–$836/year at k=10; k=5-to-k=10 answering costs average 1.89x, with a 1.86–1.93x range.Source: §4.7 · pp. 10–11

4.5 Indexing Cost at Realistic Scale

• At 1TB, GraphRAG’s two configurations extrapolate to $428K and $4.6M, an 11x build-cost spread hidden by Microsoft’s undisclosed configuration choice.The rates are extrapolated linearly, which may underestimate graph-construction costs if entity extraction and community detection scale super-linearly.Source: §4.5 · p. 10, Table 3 · p. 11, Figure 3 · p. 12

5.1 The Commercial Tax Was Real, Has Just Closed, and Was Never Academic

• The June 2026 commercial leader, Gemini embedding-001, scored 67.24 Recall@5 versus 69.55 for the non-commercial anchor, a 2.31-point gap (95% CI [0.91, 3.71]).One month later, Nemotron-3-Embed-8B was statistically indistinguishable from the anchor; the evidence supports saying the penalty is no longer measurable at this sample size, not that it is gone. On released per-question vectors, the commercial reference surfaced less gold than the anchor on 184 questions, more on 113, and tied on 703; downstream decision effects were not measured.Source: §5.1 · p. 12
• Exactly one commercially licensed embedder reaches the anchor, so the commercial tax closed at the frontier, not across the category.The other eleven entrants remained 2.55 to 15.71 points behind the top entrant, all significantly so. Free open-weight Qwen3-VL, mxbai, and BGE-M3 trailed NV-Embed-v2 by 9.7–14.6 Recall@5 points, while system-level comparisons suggest only 1–3 points after reranking or graph structure; benchmark-specific performance matters more than general-purpose reputation.Source: §5.1 · pp. 12–13
• Correcting the anchor’s body-text-only indexing raised it by 2.5 points and changed the headline from “exceeds” to “matches.”The authors argue that reporting intervals and releasing the harness and per-item outputs make such errors checkable; benchmark figures without commercial licensing and reproduction-cost information are not usable for buyers.Source: §5.1 · pp. 13–14

5.2 Why Expensive-But-Governed Might Be Correct

• The case for expensive structured retrieval is conditional: if its premium resembles measured GraphRAG-category costs, it may buy auditability and governance, but the paper cannot determine whether that premium is worthwhile.The authors report no Recall or cost figures for any specific commercial structured-retrieval product; whether the measured cost shape applies is a buyer requirements question.Source: §5.2 · p. 14

5.3 What a Buyer Should Ask

• Buyers should ask about the embedding model’s commercial license, corpus-scale indexing costs, per-token re-indexing charges, and whether quality comparisons use the same corpus format, k, and protocol.The passage reports that corpus format shifts Recall@5 by up to 4.73 points and query formatting shifts one entrant by 11.6; embedding is free per token when self-hosted, while API re-indexing incurs the toll again.Source: §5.3 · p. 14

5.4 What This Paper Leaves for Future Work

• Future work must measure how query-time system machinery recovers raw embedder gaps and what accuracy higher-tier LLMs buy per dollar.The paper’s fixed-LLM cost comparison does not establish the cost-benefit of a better LLM; that question is reserved for a planned follow-on paper.Source: §5.4 · pp. 14–15

6 Limitations

• Only three of NVIDIA’s 12+ embedding NIM endpoints were served and measured; BGE-M3 was tested in dense-vector mode only, not its hybrid mode.Six NVIDIA endpoints were attempted, with three unavailable under the measurement credentials; hybrid BGE-M3 could plausibly narrow its gap, but that was not measured.Source: §6 · p. 15
• GraphRAG’s cost spread remains unresolved because Microsoft’s paper names neither configuration, and published rates reflect 2026-07-21 and may change.The relative structure—self-hosted free, API recurring, and indexing dominant—is expected to hold regardless of provider-rate changes.Source: §6 · p. 15
• On 2WikiMultihopQA, Nemotron-3-Embed-8B versus the anchor is −0.30 Recall@5 (95% CI [−1.23,+0.62], p = 0.54), while the commercial reference trails by 0.12 (95% CI [−1.07,+0.82], p = 0.82).The pilot uses three models and the same harness and question-file convention; it weakens the tax-magnitude claim and shows that its size varies by corpus.Source: §6 · p. 15
• The cost model excludes human setup time, and it measures the authors’ fixed LLM answering cost rather than audited systems’ query-time retrieval-LLM costs.The disclosure audit searched for indexing currency markers, not unreported cost categories; none of the four systems discloses the omitted query-time figures.Source: §6 · p. 15
• Every Recall@k is an index-free ceiling, not a production recall figure, because approximate-nearest-neighbor indexing can reduce recall by an amount not estimated here.The comparison among embedders is unaffected because none of the thirteen was indexed.Source: §6 · p. 15
• The primary comparison was not externally preregistered; the recorded question predates Nemotron-3-Embed-8B’s panel entry but postdates its release.The per-question vectors let readers recompute the prespecified and alternative comparisons, so the choice can be audited even though timing cannot be guaranteed.Source: §6 · p. 15
• For Qwen3-VL-Embedding-8B, mxbai-embed-large-v1, and BGE-M3, text-only format-penalty cells are absent because those corpora were not rebuilt in text-only form.Their cached title-and-text format was confirmed against 120 passages per model, but no format penalty is claimed for them.Source: §6 · pp. 15–16
• Eight of thirteen entrants used hosted endpoints subject to silent server-side change, and text-embedding-3-large shifted −0.54 Recall@5 between measurement rounds.Hosted results are point-in-time; the five API-only models include one stable non-preview Gemini release, alongside three NIM-served NVIDIA tiers.Source: §6 · p. 16

7 Conclusion

• Across five audited systems, two publish no dollar indexing figure, one reports token counts without dollars, and third-party figures for one silent system span 11x.The configuration behind those figures is undisclosed, creating multi-million-dollar uncertainty at realistic scale.Source: §7 · p. 16
• A 2.31-point Recall@5 penalty for requiring a commercial license in June 2026 is now not measurable: Nemotron-3-Embed-8B matches, but does not beat, the anchor.Its reported difference is +0.24, with 95% CI [−0.94,+1.43] and p = 0.69.Source: §7 · pp. 16–17
• Embedding at matched 1TB scale costs roughly 7.5x–900x less than graph construction, while a year of answering at 10,000 queries/day costs 350x or more less.Embedding is a one-time build; answering is an annual recurring sum. API providers charge per token, whereas self-hostable open-weight models incur no provider per-token charge once embedded.Source: §7 · p. 16
• The one-corpus benchmark cannot establish whether self-hostable specialists hold up on buyers’ private documents, and the authors expect model ordering to depend on the corpus.They predict compact self-hostable embedders may match or trail frontier models on public corpora, but fare better on private corpora; the 2Wiki pilot supports corpus-dependent gap magnitude.Source: §7 · p. 16
• The authors are building a releasable multimodal private-corpus benchmark and register their prediction before measurement as a hypothesis, not a finding.They say that if a frontier model matches a domain-adapted specialist on unseen data, they will report that result.Source: §7 · pp. 16–17
• The authors disclose their commercial interest and say fixed, identical evaluation settings plus released per-question vectors make panel rankings recomputable; the harness is open to testing omitted embedders.They acknowledge that audit-topic selection and the planned benchmark could favor their interests, and identify release of the benchmark and harness as protections.Source: §7 · p. 17

Code and Data Availability

• All measurement code and artifacts behind Tables 1, 5, and 9 are archived at Zenodo under DOI 10.5281/zenodo.21972866, deposited 2026-08-17.The paper identifies the DOI, rather than the repository URL, as the citable record.Source: Code and Data Availability · p. 17
• The release includes the thirteen-embedder retrieval harness, answering-LLM evaluation script, cost model, bootstrap code, comparison artifacts, simultaneous intervals, and per-question recall vectors.These materials support recomputing confidence intervals and paired tests, and running comparisons without re-embedding the corpus.Source: Code and Data Availability · p. 17
• Large embedding matrices are mirrored in a revision-pinned HuggingFace dataset; the MuSiQue corpus is not redistributed, and both defective and corrected indexing scripts are released.The release explains how to fetch MuSiQue from the upstream HippoRAG-2 project.Source: Code and Data Availability · p. 17

• Anthropic is absent from the panel because it does not ship a text-embedding model.Source: A Panel Composition · p. 17
• Anthropic’s documentation recommends third-party providers, principally Voyage AI, whose voyage-3.5 is already in the panel.Source: A Panel Composition · pp. 17–18
• Table 6 reports panel access channels and per-model notes, while maker and license information appears in Table 1.Source: Table 6 · p. 18

Worked Metric Example

• For the Scanderbeg example, three of three gold passages in the top five gives Recall@5 of 100%, while two of three gives approximately 66.7%.The three gold passages are “Scanderbeg (opera),” “Orlando furioso (Vivaldi, 1714),” and “Rialto Bridge.”Source: B Worked Metric Example and Format… · p. 18

Format Sensitivity

• Qwen3-VL-Embedding-8B’s Recall@5 swung 11.6 points, from 48.3 to 59.9, solely from query-instruction wording.The passage presents query-formatting choice as a finding whose size varies by model.Source: B Worked Metric Example and Format… · p. 18
• BGE-M3 moved from 54.22 bare to 54.93 with an instruction prefix, while its documentation says no prefix is required.The paper carries 54.93 throughout; its precorrection-run figure of 54.87 differs by +0.06.Source: B Worked Metric Example and Format… · p. 18
• In asymmetric API modes, callers specify whether text is a question or passage, and embedding a question as a document costs Recall.Cohere uses input_type and Gemini uses task_type; providers embed the two roles differently because short questions and long documents are different kinds of objects.Source: B Worked Metric Example and Format… · p. 18

C Realistic Scale: Institutional Data Volumes

• The measurement extracts text from all .docx, .xlsx, .pptx, .eml, and .txt files in the two largest matters in Harvey’s public Legal Agent Benchmark.The paper rejects the Library of Congress’s 21-petabyte figure [10] and a 64% regulated-industry survey statistic as text-corpus anchors because they count mixed content or total data, respectively.Source: C Realistic Scale: Institutional Data… · p. 18
• 225MB of extracted text per matter is 40x the 5.64MB MuSiQue corpus; 5GB represents about 23 matters and 1TB about 4,650.The paper says a mid-sized transactional practice can reach these volumes without being an outlier.Source: Table 7 · p. 19, C Realistic Scale: Institutional Data… · p. 19
• Indexing one matter costs eight to several hundred times as much as embedding it, while GraphRAG’s two estimates differ by 8.5x: $19.49 versus $2.30.That uncertainty is $690 for one matter and $69,000 across 100 matters per year; both costs recur on each re-index.Source: C Realistic Scale: Institutional Data… · p. 19
• The 1TB extrapolation is about 185,900x the measured corpus, versus 40x for one matter; linear O(N) graph-cost estimates may understate costs more at 1TB if scaling is super-linear.The paper describes its per-matter estimates as the least extrapolated and cautions that graph-density effects could raise the bottom-row figures disproportionately.Source: C Realistic Scale: Institutional Data… · pp. 19–20, Table 8 · p. 19
• The measured matters are synthetic, and the two largest are not representative: the median matter is 424x smaller, making 225MB an upper anchor rather than an average.The corpus distribution is bimodal.Source: C Realistic Scale: Institutional Data… · p. 20

D Panel Re-Measurement and Reproduction

• Nemotron-3-Embed-8B had a 2.54-point format penalty, while NV-Embed-v2’s 2.46-point penalty ranked sixth of ten.The two values fall within the panel’s 1.46–4.73-point range; the authors state that a format mismatch would depress any of the thirteen entrants comparably.Source: D Panel Re-Measurement and Reproduction · p. 20
• Nine of ten models incurred significant 1.46–4.73-point title-text versus text-only penalties; text-embedding-3-large’s 0.79-point loss was not distinguishable from zero.The penalty is a corpus-format effect, not run-to-run variation.Source: D Panel Re-Measurement and Reproduction · p. 20, Table 9 · p. 21
• All thirteen entrants reproduced within 0.6 points, and NV-Embed-v2’s dedicated fresh-cache replicate returned 69.55/78.12 with Recall@5 delta +0.00.The replicate matched all 1,000 per-question scores; its 69.55 was the best title-text variant, not a text-only result.Source: D Panel Re-Measurement and Reproduction · p. 20, Table 9 · p. 21
• Text-embedding-3-large had the panel’s largest run-to-run deviation at −0.54, but remained nearer its title-text result than its text-only result.Hosted endpoints are not version-pinned, so some drift is expected.Source: D Panel Re-Measurement and Reproduction · p. 22

DGX Spark anchoring

• The cost model anchors the pipeline to a $4,699 DGX Spark because it hosts heavier LLM workloads, although an 8B embedder needs roughly 16GB VRAM and can run on a sub-$1,000 consumer GPU.The relevant deployment figure is one owned box for retrieval and answering, not an embedding-only appliance.Source: E Extended Cost Notes · p. 22

Answering-LLM configuration

• Table 5 answering costs use one deterministic gpt-4o-mini call per question, with temperature=0.0 and max_tokens=32.The system asks for concise answers and best guesses when context lacks an answer; scoring uses maximum SQuAD token-overlap F1 across the gold answer and aliases, so F1 reflects retrieved-context support rather than calibration.Source: E Extended Cost Notes · p. 22

GraphRAG conversion imprecision

• The $19.49–$38.99 GraphRAG reconstruction is directional because it converts Llama-3.3-70B token counts to gpt-4o-mini rates.Different tokenizers and potentially different entity/relation extraction make the conversion imprecise.Source: E Extended Cost Notes · p. 22

KET-RAG’s $33K trace

• $21 to index a 3.2MB HotpotQA sample is excluded from Table 3; including it raises the 1TB upper estimate from $4.64M to ≈$6.9M.KET-RAG’s often-repeated “$33K” figure for a 5GB legal case is its own linear extrapolation, not an independently sourced estimate.Source: E Extended Cost Notes · p. 22

gpt-4o-mini rate caveat

• The gpt-4o-mini batch rates used were $0.075/1M input and $0.30/1M output, billed on 2026-07-21; readers rerunning costs should re-verify them.The model was no longer listed on OpenAI’s public pricing page on that date; the meter was within 4% of the rate card.Source: E Extended Cost Notes · p. 22

nv-embedqa-mistral-7b-v2

• nv-embedqa-mistral-7b-v2 was treated as unavailable because it no longer appeared in NVIDIA’s NIM support matrix, despite catalog metadata indicating commercial licensing.The catalog listed a Community License and Apache-2.0 and no MS MARCO lineage.Source: E Extended Cost Notes · p. 22