PaperlayerView exact revision on arXiv

Source-linked AI summary

The Embedder's Dilemma: LLMs Are Better, but at What Cost?

Adnan El Assadi, Niklas Muennighoff, Jinhyuk Lee

arXiv:2608.12875v1cs.CL

TL;DR

Practitioners face a choice between embedding pipelines and LLMs for text representation and retrieval. The paper compares both paradigms across a controlled 37-task benchmark with cost and throughput accounting, finding aggregate parity but a task-specific and economically important split: LLMs lead on reasoning-heavy retrieval, while embeddings are generally more efficient elsewhere.

  • Problem

    The paper asks whether practitioners should replace embedding pipelines with LLMs, given uncertainty about their comparative quality across embedding tasks and deployment costs.

  • Method

    The study compares ten LLMs and 26 embedding models on 37 MTEB(LLM) tasks, pairing quality evaluation with API cost accounting and GPU throughput benchmarking.

  • Results

    The paradigms tie in aggregate but differ by task: LLMs lead on reasoning-heavy retrieval, embeddings lead on classification, and the remaining categories are tied.

  • Takeaways & Limitations

    Embedding pipelines remain the cost-efficient default, while reasoning-intensive retrieval is the clearest case for an LLM and supports a hybrid retrieve-then-rerank strategy.

  • Takeaways & Limitations

    The benchmark uses small corpora of 82–415 documents in prompts, so the reported cost gap is a lower bound for production corpora requiring indexed first-stage retrieval.

Abstract

from arXiv · show

Should you replace your text-embedding pipeline with a large language model? We answer this with a controlled, cost-aware comparison of ten LLMs across six families and 26 embedding models (118M to 14B parameters) on 37 tasks spanning classification, semantic textual similarity (STS), clustering, pair classification, and retrieval. In aggregate the two paradigms are effectively tied: the best LLM (Gemini 3.1 Pro, 77.6) and the best embedding model (77.2) differ by 0.4 points. Their strengths differ by task: LLMs lead on reasoning-heavy retrieval, embedding models lead on classification, and the two match on clustering, STS, and pair classification. Reaching that parity is expensive. An LLM costs up to 1,431x more than an embedding model of comparable quality (USD 154 vs. USD 0.11 per benchmark pass), and the open LLMs tested process tokens 2.5 to 736x more slowly on the same GPU. Reasoning tokens account for 28 to 81% of LLM inference cost; lower reasoning budgets preserve or improve retrieval quality for most models in our ablation. The Pareto frontier contains the leading embedding models and one LLM, Gemini 3.1 Pro. These results support a division of labour: use embedding models for similarity, classification, and clustering, and reserve LLMs for reasoning-intensive retrieval. Our code, datasets, and results are publicly available at https://github.com/embeddings-benchmark/embedders-dilemma.

1 Introduction

The paper asks whether LLMs should replace embedding pipelines and finds that they now match embedding quality in aggregate, but with sharply higher cost and lower throughput. The results support embeddings as the default and LLMs for reasoning-heavy retrieval.

  • LLMs without task-specific training match the best text embedding models across a broad suite of standard embedding tasks.
  • The evaluation compares ten frontier LLMs with 26 embedding models on 37 tasks covering classification, STS, clustering, pair classification, and retrieval.Scores are paired with API token accounting for LLMs and GPU throughput benchmarking for embedding models.
  • The paradigms tie in aggregate, but LLMs lead on reasoning-heavy retrieval, embeddings lead on classification, and they are statistically even on clustering, STS, and pair classification.
  • The cost gap is large: Gemini 3.1 Pro costs 1,431× as much as a comparable embedding model, while open LLMs process 2.5–736× fewer tokens per second on identical hardware.Reasoning contributes substantially to the cost asymmetry.
  • Embedding pipelines remain the efficient default for classification, similarity, and clustering, while LLMs are most useful for reasoning-intensive retrieval.A hybrid pipeline uses embeddings for candidate retrieval and LLMs for reasoning over the shortlist.

2 Related Work

Prior work established strong dense retrieval and documented the computational cost of verbose reasoning, but did not provide this paper’s broad cost-aware comparison across embedding tasks. The paper builds on these strands to examine Pareto-efficient deployment choices.

  • MTEB standardized evaluation across eight task categories and tracks hundreds of dense-text models on a public leaderboard.Leading systems achieve strong performance through contrastive training on large curated corpora.
  • Green AI argues that accuracy-focused evaluation neglects computational and environmental costs, motivating cost-aware model comparisons.
  • Prior work on controllable reasoning shows that adaptive test-time compute can outperform a uniform reasoning budget.
  • BEIR established strong zero-shot bi-encoder retrieval, while BRIGHT complements it with reasoning-intensive queries where LLM augmentation can substantially improve retrieval.BRIGHT’s top embedding model scores 18.3 nDCG@10, while LLM-augmented retrieval improves by up to 12.2 points.

3 Experimental Setup

The study evaluates ten LLMs and 26 embedding models using a fixed 37-task benchmark spanning five categories, while measuring deployment cost and throughput under defined protocols. The comparison uses complete zero-shot and embedding-based pipelines rather than isolated model outputs.

  • The LLM set spans ten models across six families, including closed and open-weight dense and mixture-of-experts architectures.The models include reasoning and instruct variants across a wide capability and cost range.
  • The embedding set contains 26 text embedding models ranging from 118M to 14B parameters.
  • MTEB(LLM) contains 37 fixed-seed tasks covering classification, STS, clustering, pair classification, and retrieval.The benchmark uses identical derived datasets so submissions are evaluated on the same data.
  • The pipelines use zero-shot LLM prompts, kNN for classification, cosine similarity for STS and retrieval, and k-means for clustering.A few-shot ablation tests whether limited in-context examples narrow the classification gap.
  • Embedding cost is computed from maximum-throughput H100 processing at sequence length 512, with measured costs ranging from $0.001 to $0.22 per benchmark pass.
  • LLM cost is calculated from provider-reported input, cached, output, and reasoning-token usage, with costs ranging from $3.16 to $154.14.Reasoning tokens are billed at the output rate.
  • Both paradigms are served on the same H100 for throughput comparison, while cost accounting uses API rates for LLMs and GPU-rental rates for embeddings.

4 Results

The results show an aggregate tie that conceals a task-specific division: LLMs lead on retrieval, embeddings lead on classification, and the remaining categories are statistically even. Embedding models occupy most cost–performance frontiers because comparable LLM quality is much more expensive and slower.

  • Aggregate performance: Gemini 3.1 Pro leads with a mean score of 77.6, followed by Octen-8B at 77.2 and Qwen3-E-8B at 77.0.
  • Task-category analysis: Embedding models match the best LLM on 7 of 8 classification tasks and 7 of 10 STS tasks, but on only 1 of 6 retrieval tasks.
  • Task-category analysis: Retrieval favors LLMs: Pro scores 64.5 versus 56.0 for the best embedding and wins five of six retrieval tasks.
  • Task-category analysis: Classification favors embeddings: SFR-2 scores 90.8 versus 85.2 for Pro, with wider gaps on fine-grained tasks.
  • Task-category analysis: Clustering, STS, and pair classification are statistically tied, despite narrow category-leading differences among the top models.The reported category comparisons are SFR-2 66.7 versus Pro 66.6, Qwen3-E-4B 88.8 versus Pro 88.5, and KaLM-12B 87.1 versus Pro 83.2.
  • Cost–performance: Pro extends the retrieval cost–performance frontier, while embedding models define the frontiers for classification, clustering, STS, and pair classification.
  • Cost and throughput: Pro costs 1,431× as much as a comparable embedding model: $154.14 versus $0.11 for a benchmark pass.The marginal aggregate edge is 0.4 points.
  • Reduced thinking: Reasoning tokens account for 28–81% of LLM inference cost, while reduced reasoning preserves or improves retrieval for four of six models.The ablation cuts generated tokens by 54–96%.

5 Discussion

The study finds that LLMs and embedding models have complementary strengths: LLMs help most with reasoning-heavy retrieval, while embeddings remain more efficient for similarity, classification, and clustering. A hybrid retrieve-then-rerank pipeline can add LLM reasoning over a shortlist, but classification comparisons reflect different supervision and corpus-in-context costs are limited at production scale.

  • Task-dependent strengths: LLMs lead on reasoning-heavy retrieval, while embedding models provide balanced or superior coverage across the other task categories.Pro leads on 5/6 retrieval datasets, while embeddings lead on classification and the paradigms are statistically even on clustering, STS, and pair classification.
  • Inference procedures: Embedding pipelines encode documents offline, whereas rerankers repeatedly process query-specific document sets whose size determines cost.Cross-encoders read one document, listwise rerankers read a shortlist of k documents, and corpus-in-context LLMs read N documents jointly.
  • Inference procedures: Reranking is more economical than placing an entire corpus in context when adding reasoning to retrieval.The comparison links cost to repeated query-specific computation, with reranking scaling in k and corpus-in-context processing scaling in N.
  • Scope of comparison: The classification comparison reflects deployment pipelines with different supervision rather than intrinsic representational ceilings.Embedding classifiers use kNN over labelled data, while LLMs receive labels, task descriptions, and up to five examples; no classification-post-trained frontier LLM was available.
  • Reasoning budgets: Lower reasoning budgets preserve retrieval quality for most evaluated models while removing the largest component of LLM cost.The paper attributes this to default reasoning budgets reconsidering straightforward relevance judgments when direct reading comprehension suffices.
  • Deployment implications: A hybrid architecture uses embedding models for high-throughput candidate retrieval and LLMs for reasoning over a shortlist.The retrieve-then-rerank results motivate this division of labour, especially for reasoning-heavy retrieval.
  • Scope of comparison: The corpus-in-context protocol is feasible only at small scale, so its reported cost gap is a lower bound for production corpora.The evaluated corpora contain 82–415 documents, whereas production systems require an indexed first stage.

6 Conclusion

The paper introduces a cost-aware benchmark showing that LLMs and embedding models match in aggregate but differ by task, with LLMs strongest on reasoning-heavy retrieval and embeddings more cost-efficient elsewhere. These findings support hybrid retrieval and reporting Pareto frontiers alongside accuracy.

  • 6 Conclusion: MTEB(LLM) evaluates ten LLMs and 26 embedding models across five MTEB task categories and 37 tasks.The benchmark compares deployment pipelines with cost measured alongside quality.
  • 6 Conclusion: The two paradigms match in aggregate, but LLMs lead on reasoning-heavy retrieval, embeddings lead on classification, and the remaining categories are tied.The conclusion summarizes the cross-task performance pattern rather than a single benchmark result.
  • 6 Conclusion: Comparable LLM quality comes with substantially higher cost and lower throughput.The conclusion frames deployment efficiency as distinct from accuracy-only performance.
  • 6 Conclusion: Embedding models remain the cost-efficient default, while reasoning-intensive retrieval is the clearest case for using an LLM.The recommended hybrid strategy uses embeddings for candidate retrieval and LLMs for reasoning over the shortlist.
  • 6 Conclusion: Pareto frontiers add deployment cost and throughput to accuracy-based rankings, and MTEB(LLM) can expand as new models appear.The released code and datasets are MTEB-compatible.

Reproducibility Statement

The released implementation supports reproducible regeneration of the paper’s evaluation artifacts and extension to future models.

  • Reproducibility Statement: The release includes code, raw results, analysis scripts, datasets, prompt templates, validation logic, and token-usage extraction scripts.Every figure and table regenerates from the released result files.

Ethics Statement

The study uses public benchmark data and publicly available models without human subjects or private data. It also urges practitioners to consider environmental impacts alongside economic deployment costs.

  • Ethics Statement: The evaluation uses publicly hosted datasets previously released for research and involves no human subjects or private data.The statement describes the study’s data and participant scope.
  • Ethics Statement: Practitioners should consider the environmental impact of LLM pipelines, including the carbon footprint of chain-of-thought inference.The paper places environmental considerations alongside the economic factors discussed in the study.

A Statistical Significance Testing

The paired bootstrap finds an overall statistical tie between Gemini 3.1 Pro and the best embedding model, with significant differences confined to classification and retrieval. Classification favors embeddings, while retrieval favors LLMs; STS, clustering, and pair classification show no significant difference.

  • Testing procedure: 10,000 task-level bootstrap resamples compare Gemini 3.1 Pro with the best embedding model using mean-of-means differences across MTEB(LLM) tasks.The procedure uses equal task weighting and accommodates category-specific metrics.
  • Overall comparison: ∆= +0.3, 95% CI = [−2.4, +3.1], p = 0.85: the overall comparison is a statistical tie.The comparison is Pro versus Octen-8B across MTEB(LLM).
  • Category-level results: ∆= −5.6, CI = [−9.2, −2.4], p < 0.01: embeddings significantly outperform LLMs on classification.Classification remains significant under the Bonferroni threshold α = 0.01.
  • Category-level results: ∆= +8.5, CI = [+0.2, +16.8], p < 0.05: LLMs significantly outperform embeddings on retrieval.Retrieval falls above the Bonferroni threshold of α = 0.01.
  • Category-level results: STS (p = 0.74), clustering (p = 0.97), and pair classification (p = 0.49) show no significant paradigm difference.These category-level results are statistically even.
  • Interpretation: Pro ranks first overall largely because of retrieval, but ranks below ten embedding models on classification.Per-task scores vary substantially, with Pro strongest on FQuAD retrieval and SFR-2 strongest on SprintDuplicateQuestions.

B.4 Full Per-Task Scores for All Models

The full score tables organize all evaluated models and tasks by category, while the reduced-thinking experiments show that less reasoning can preserve or improve retrieval without measurable classification gains.

  • Full score tables: Tables 7–11 place models in rows and category tasks in columns, with each caption specifying the category metric.Rows list ten LLMs first and 26 embedding models below, ordered by overall MTEB(LLM) score.
  • Comparative results: Table 4 compares the best LLM with the best embedding model in each category, while Table 6 reports representative per-task scores and the best embedding score per task.Table 6 includes two LLMs and three embedding models across all 37 tasks.
  • Full score tables: The final table column reports each model’s category mean, revealing over- or under-performance relative to overall ranking because rows are not category-sorted.Bold marks the best value in each task or category-mean column.
  • Reduced-thinking ablation: 54–94% reduction occurs on retrieval and 12–30% on classification when Gemini 3 Flash uses low reasoning effort.The ablation evaluates six retrieval tasks and three representative classification tasks.
  • Reduced-thinking ablation: Reduced reasoning preserves or improves all six Flash retrieval scores and four of six cross-family averages.On classification, all three tasks show |∆| < 1.0, indicating no measurable benefit from additional reasoning in this ablation.

C.2 Per-Task Retrieval Analysis

Retrieval is the clearest task-level divide: LLMs generally benefit from joint query–document reasoning, while embeddings remain more precise and efficient in some settings. Across the broader benchmark, category rankings show complementary strengths and substantial deployment trade-offs.

  • Retrieval results: +8.5: Pro leads the aggregate retrieval advantage, winning five of six retrieval tasks while Octen-8B leads AILAStatutes.
  • Retrieval results: 33.4 versus 31.5: Pro’s advantage on keyword-centric TwitterHjerne retrieval is modest because both paradigms handle explicit topical matching similarly.
  • Retrieval results: 23.2 versus 14.5: Octen-8B outperforms Pro on AILAStatutes, where narrower cosine-similarity rankings beat broader associative retrieval.

G Extended Limitations

The paper’s limitations span model coverage, supervision, retrieval scale, reasoning controls, throughput measurement, and aggregate task weighting. These boundaries affect how broadly its cost, quality, and deployment conclusions should be interpreted.

  • LLM coverage: Ten LLMs across six families provide a snapshot of a fast-moving frontier, while uncovered closed APIs may have different cost–capability profiles.The evaluated set includes dense and mixture-of-experts architectures plus reasoning and instruct variants.
  • Supervision asymmetry: Classification gives embeddings labelled-reference kNN and LLMs zero-shot prompting, creating different supervision conditions.Five examples preserve performance on small-label tasks but reduce Banking77 performance.
  • Small-corpus retrieval: The corpus-in-context retrieval protocol uses 82–415 documents and does not vary corpus size, so larger-scale quality changes remain unquantified.At BEIR or production scale, placing the full corpus in context is generally impractical and requires an indexed first stage.
  • Reasoning-control granularity: Reasoning controls differ by provider: Gemini offers a soft low-effort hint, whereas open models support fully disabled reasoning.Reduced reasoning preserves or improves retrieval for four of six models across five families, while token accounting varies by provider.
  • Throughput comparison: Open-weight LLMs and embedding models are compared on the same H100, yielding a 2.5–736× throughput gap consistent with autoregressive decoding versus one encoder pass.The setup removes API rate limits and hardware choice as confounds.
  • Task weighting sensitivity: Equal weighting across MTEB(LLM) tasks may shift under dataset-size, z-score, or task-family weighting, so aggregate results should be read with category-level results.Alternative aggregation methods, including Borda count, are also possible.
  • Token usage: Reasoning-heavy LLMs generate 8–26M reasoning tokens versus 1–3M standard-output tokens across the benchmark.Table 17 reports input, standard-output, cached, reasoning, and total tokens with cost decomposition.
  • Embedding cost measurement: Embedding costs range from $0.00016/MTok for mE5-small to $0.047/MTok for F2LLM-v2-14B, with total benchmark costs spanning 291×.The embedding measurements use a single NVIDIA H100 at sequence length 512 with maximum batch size.

H.3 Cost Sensitivity Analysis

The cost analysis tests how the LLM-to-embedding gap changes under alternative pricing and hardware assumptions. It also translates benchmark costs into deployment economics while documenting the measurement pipelines and controls.

  • Alternative scenarios: Five alternative embedding-inference scenarios all produce an LLM-to-embedding cost ratio above 300×.The baseline H100 spot scenario is 1,431×, while the commercial API scenario is 338×.
  • Sensitivity range: 338–2,424× are the cost ratios reported across alternative hardware and pricing scenarios comparing Octen-8B with Gemini 3.1 Pro.Table 19 evaluates the ratio under fixed API pricing and alternative assumptions.
  • Per-query economics: 29–1,431× remains the per-query cost range because both paradigms answer the same queries and dividing by query count rescales both sides equally.Embedding training is amortised, document embeddings are reused, and corpus-in-context reading repeats per query.
  • Embedding cost pipeline: Embedding cost is computed as (tokens/10^6) × (r_GPU/T) using tokenizer-specific counts, H100 spot pricing, and sustained throughput.The benchmark uses maximum-throughput processing at sequence length 512 with the largest batch fitting in memory.
  • Tokenizer sensitivity: 4.4–5.5M tokens span model tokenizers, and using a GPT-2 proxy would overestimate embedding costs by 22–54%.The range is driven by vocabulary differences, with Gemma-based tokenizers at the low end and Mistral-based tokenizers at the high end.
  • LLM cost pipeline: LLM cost accounting extracts provider usage statistics, with Gemini separating reasoning tokens and OpenRouter including reasoning in completion counts.Total tokens equal input plus output plus reasoning for Gemini, while OpenRouter total equals input plus output.
  • Pricing assumptions: Gemini input/output prices range from $0.25/$1.50 per MTok for Flash-Lite to $2.00/$12.00 for Pro, with cached input billed at 10%.Open models use OpenRouter public rates.
  • Throughput measurement: Holding hardware fixed, throughput is measured on one H100 for open-weight LLMs under vLLM and for batched embedding inference.This removes API rate limits and GPU choice as explanations for the measured gap.
Loading 2608.12875v1…