PaperlayerView exact revision on arXiv

Source-linked AI summary

HUME: Measuring the Human-Model Performance Gap in Text Embedding Tasks

Adnan El Assadi, Isaac Chung, Roman Solomatin, Niklas Muennighoff, Kenneth Enevoldsen

arXiv:2510.10062v3cs.CL

TL;DR

Embedding benchmarks often lack human-performance references, leaving model scores difficult to interpret on ambiguous tasks. HUME measures humans across diverse MTEB tasks and compares their performance with embedding models and LLM annotators. Human performance varies by task quality, and the best LLM annotator falls short of humans.

  • Problem

    Embedding benchmarks provide little human-performance evidence, making model scores difficult to interpret when tasks involve ambiguity and annotator disagreement.

    Source: §1 · p. 2

  • Method

    HUME measures human performance across 16 MTEB datasets and four task categories, compares humans with embedding models, and benchmarks nine LLMs as annotators.

    Source: §1 · p. 2

  • Results

    Human performance varies substantially by task category, and the best LLM, GPT-4.1-mini (76.1%), falls short of human performance (81.2%).

    Source: §6 · p. 10

  • Takeaways & Limitations

    High-agreement tasks provide reliable evaluation targets, while low-agreement tasks often reveal task-formulation issues; LLMs should augment, not replace, human evaluation.

    Source: §6 · p. 10

    Breadth across 16 tasks limited per-task samples to 20–50 instances, so larger samples would better capture human-performance variation and provide more robust estimates.

    Source: §5.4 · p. 10

Abstract

from arXiv · show

Comparing human and model performance offers a valuable perspective for understanding the strengths and limitations of embedding models, highlighting where they succeed and where they fail to capture meaning and nuance. However, such comparisons are rarely made, as human performance on embedding tasks is difficult to measure. To fill this gap, we introduce HUME: Human Evaluation Framework for Text Embeddings. While frameworks like MTEB provide broad model evaluation, they lack reliable estimates of human performance, limiting the interpretability of model scores. We measure human performance across 16 MTEB datasets spanning reranking, classification, clustering, and semantic textual similarity across linguistically diverse high- and low-resource languages. Humans achieve an average performance of 77.6% compared to 80.1% for the best embedding model, though with substantial variation: models reach high performance on some datasets while struggling on notably low-resource languages. Our human annotations also reveal multiple dataset issues. We additionally benchmark nine LLMs as annotators on reranking, classification, and STS tasks, finding that they fall short of human performance (76.1% vs. 81.2%) despite offering scalability advantages. We provide human performance baselines, insights into task difficulty patterns, and an extensible evaluation framework that enables a more meaningful interpretation of results and informs the development of both models and benchmarks. Our code, dataset, and leaderboard are publicly available at https://github.com/embeddings-benchmark/mteb.

1 INTRODUCTION

Embedding benchmarks lack human-performance references, making model scores hard to interpret on tasks where annotators may disagree. HUME measures human performance across embedding tasks and compares it with models, while also testing LLMs as scalable annotators.

1 INTRODUCTION

  • Without human references, benchmark scores are difficult to interpret when tasks involve ambiguity and annotator disagreement.The paper warns that optimizing against noisy labels can reward replication of annotation artifacts rather than semantic progress.Source: §1 · p. 2
  • HUME measures human performance across 16 MTEB datasets covering reranking, classification, clustering, and semantic textual similarity, and compares it with state-of-the-art embedding models.The multi-annotator experiments examine task difficulty and variation across humans.Source: §1 · p. 2
  • The paper contributes an extensible human-evaluation framework, cross-task empirical evidence, and comparisons that surface ambiguity in benchmarks and models.The authors intend these findings to inform future benchmark design.Source: §1 · p. 2
  • The study also evaluates nine LLMs as annotators to assess their viability as scalable proxies for human judgment and identify task-specific limitations.The evaluation tests whether LLM judgments capture the semantic distinctions humans make or exhibit systematic biases.Source: §1 · p. 2

2 RELATED WORK

Prior embedding benchmarks offer broad automated evaluation but generally lack human-performance baselines. Existing human evaluation in NLP provides limited coverage of embedding tasks and rarely yields scores directly comparable with model metrics.

2.1 TEXT EMBEDDING MODELS AND MTEB

  • MTEB consolidates diverse embedding tasks but lacks human-performance baselines, limiting context for model achievements.Embedding models have evolved from static vectors through contextual encoders to models optimized for embedding applications.Source: §2.1 · p. 2

2.2 HUMAN EVALUATION IN NLP

  • Human evaluation is established in generative NLP, whereas embedding-based tasks have relied almost exclusively on automated metrics.This contrast leaves human baselines for embedding tasks sparsely studied.Source: §2.2 · p. 3
  • Existing relevance judgments and human baselines in other benchmarks do not generally provide model-comparable human-performance scores for embedding tasks.STS reports inter-annotator agreement, but not scores converted to model-comparable performance.Source: §2.2 · p. 3

3 METHODOLOGY

HUME aligns human annotation with embedding-model evaluation across diverse tasks and languages. Its methodology combines task-specific instructions and sampling with matched model comparisons, while evaluating LLM annotators on a subset of task types.

3.1 FRAMEWORK DESIGN

  • HUME uses reproducible human-evaluation protocols aligned with model evaluation, including task-specific interfaces, principled sampling, standardized results, and aligned metrics.The design is intended to reveal ambiguity or inconsistency in evaluation practices.Source: §3.1 · p. 3

3.2 TASKS, DATASETS, AND METRICS

  • The dataset selection spans languages, domains, construction methods, established benchmark tasks, and task complexity to cover varied embedding applications.Languages include English, Arabic, Russian, Danish, and Norwegian Bokmål; reranking serves as a tractable proxy for large-scale information retrieval because annotators assess only top-k candidates.Source: §3.2 · p. 3

3.3 INSTRUCTIONS

  • Task-specific instructions match task definitions, but sparse public annotation protocols limit verification of alignment with original dataset procedures.Instructions were based on original dataset descriptions and piloted to resolve ambiguities.Source: §3.3 · pp. 3–4

3.4 ANNOTATION PROCEDURE

  • Annotators used task-specific interfaces and downsampled datasets; multilingual tasks had one language-proficient annotator, while English tasks had two for agreement analysis.Annotators worked independently without access to ground truth or model predictions, and agreement used task-appropriate metrics.Source: §3.4 · p. 4

3.5 MODEL SELECTION AND EVALUATION

  • Thirteen embedding models spanning parameter scale, architecture, instruction tuning, and multilingual capability were evaluated on the same instances and protocols as humans.Primary metrics were MAP for reranking, Accuracy for classification, V-Measure for clustering, and Spearman correlation for STS; confidence intervals used metric-appropriate methods.Source: §3.5 · p. 4

3.6 LLM-AS-ANNOTATOR EVALUATION

  • Nine LLMs received human-matched instructions, instances, metrics, and scoring protocols for reranking, classification, and STS; clustering was excluded because consistent structured assignments were difficult to elicit.The controlled comparison assesses their potential as scalable, lower-cost proxies and their task-specific limitations.Source: §3.6 · pp. 4–5

4 RESULTS AND ANALYSIS

Across 16 tasks, humans occupy a variable middle ground relative to embedding models, with performance changing by task, language, and dataset quality. Results highlight low-agreement benchmark issues and show that LLM annotators fall short of humans in the evaluated tasks.

4 RESULTS AND ANALYSIS

  • Humans rank fourth overall at 0.776, outperforming 10 of 13 embedding models, while their standing varies by task category, language, and dataset quality.The table aggregates results over 26 task-language pairs and reports humans at the top in 5 of 14 aggregated pairs.Source: §4 · p. 5, Table 1 · p. 5

4.1 PERFORMANCE PATTERNS BY TASK CATEGORY

  • Humans fall within the upper portion of model ranges and exceed median model performance on 61.5% of task-language pairs, but match the best models on only 15.4%.Classification is the strongest human category; clustering and reranking show consistent gaps to top models.Source: §4.1 · p. 5, Figure 2 · p. 6
  • Human performance varies by task: humans outperform models on Arabic and Russian sentiment, while models exceed humans in clustering and reranking; STS models are comparable, with human results varying substantially across datasets.Human averages are 70.3 for classification, 67.4 V-measure for clustering, 87.2 MAP for reranking, and 83.2 Spearman correlation for STS. Clustering ranges from 97.6 on WikiCities to 49.2 on ArXiv papers.Source: §4.1 · pp. 5–6

4.2 CROSS-LINGUAL PERFORMANCE ANALYSIS

  • Human wins against the best models are more common on multilingual tasks (29%) than English-only tasks (0%), while English results are more balanced and generally favor or match models.Arabic shows a 67% human win rate against best models; its largest reported gap is semantic similarity, 67.5% versus 40.9%. Russian and Norwegian humans also outperform the best models in sentiment analysis.Source: §4.2 · pp. 6–7

4.3 DATASET QUALITY AND EVALUATION CHALLENGES

  • Low human agreement often accompanies ambiguous task formulation, and apparent “superhuman” model scores may reflect reproduction of systematic label patterns rather than superior understanding.Emotion classification has κ = 0.39 and 52.1% consensus; ArXiv clustering has ARI = -0.001. By contrast, high-agreement reranking and toxicity classification provide more reliable evaluation targets.Source: §4.3 · p. 7

4.4 CAN LLMS REPLACE HUMAN ANNOTATORS?

  • The best LLM annotator, GPT-4.1-mini, scores 76.1% against human performance of 81.2%, and LLM performance is moderately correlated with human performance across tasks.The comparison excludes clustering; task-level results show LLMs near humans in classification but behind in reranking and STS.Source: §4.4 · pp. 7–8, Table 2 · p. 8

5 DISCUSSION

The discussion argues that human agreement is essential for interpreting embedding benchmarks: low-agreement tasks may reward annotation artifacts, while high-agreement tasks offer more reliable comparisons. It also finds limits to LLM annotation and recommends changes to benchmark development, alongside noting study limitations.

5.1 TASK QUALITY AND EMBEDDING EVALUATION RELIABILITY

  • Models achieve their highest relative performance where human agreement is lowest, while humans retain advantages in Arabic semantic similarity and multilingual sentiment.The authors interpret high model scores on low-quality datasets as potentially reflecting consistent annotation artifacts rather than semantic understanding.Source: §5.1 · p. 8

5.2 LLM-BASED EVALUATION

  • On reranking, humans significantly outperform LLMs despite strong annotator agreement; classification shows near-parity alongside lower human agreement.The reported agreement ranges are ρ = 0.64–0.85 for reranking and κ = 0.24–0.55 for classification.Source: §5.2 · p. 9
  • LLMs may introduce systematic biases in large-scale annotation, and their generative architecture prevented evaluation on clustering tasks.The authors recommend using LLMs to augment, rather than replace, human annotation, especially for benchmark development.Source: §5.2 · p. 9

5.3 IMPLICATIONS FOR MODEL DEVELOPMENT AND EVALUATION PRACTICES

  • Reranking provides a dependable development target, with a persistent model-human gap (96.4% vs. 87.2%) representing a relevance-modeling challenge.The authors also identify toxicity classification, with 77.8% human consensus, as a reliable target; they caution that poorly agreed tasks may reward arbitrary labeling patterns.Source: §5.3 · p. 9
  • Arabic semantic similarity shows a human advantage (67.5% vs. 40.9% best model), and multilingual sentiment also favors humans in non-English languages.The reported human results include 95.0% for Arabic and 92.5% for Russian multilingual sentiment; the authors call for approaches beyond simple data scaling.Source: §5.3 · p. 9
  • The authors recommend replacing emotion classification, ArXiv clustering, and STS22-Russian, and reporting human agreement alongside model scores.For agreement below κ < 0.4 or ρ < 0.6, they recommend improving task specifications or removing the dataset; they note that some disagreement is natural judgment variability.Source: §5.3 · pp. 9–10

5.4 LIMITATIONS

  • The study’s breadth across 16 tasks meant only 20–50 instances per task; its multi-task design also limited task-specific training and expert annotation.Sparse original guidelines further made alignment with prior annotation procedures difficult, although the authors say this reflects realistic annotation conditions.Source: §5.4 · p. 10
  • Agreement estimates are limited by two annotations for most tasks and an all-male, 20–35-year-old annotator group; the study also leaves causes of model-human gaps and task-design effects unresolved.Analysis of gap causes is further constrained by unavailable model-training-corpus details.Source: §5.4 · p. 10

6 CONCLUSION

HUME establishes human performance baselines across 16 MTEB datasets and finds that task agreement helps distinguish reliable benchmarks from those with possible design issues. The conclusion also reports that the best tested LLM annotator falls short of human performance, while appendices describe the study’s dataset and result coverage.

6 CONCLUSION

  • HUME measures human performance across 16 datasets spanning reranking, classification, clustering, and STS to provide baselines for interpreting embedding-model achievements.The authors frame the framework as addressing a gap in empirical performance bounds for embedding models.Source: §6 · p. 10
  • Tasks with high human agreement provide reliable benchmarks, while low-agreement tasks often reveal task-formulation issues.Human performance varies substantially by task category.Source: §6 · p. 10
  • 76.1% for the best LLM, GPT-4.1-mini, falls short of human performance at 81.2%, particularly on reranking.The authors conclude that LLMs offer scalability but should augment rather than replace human evaluation.Source: §6 · p. 10

6.1 ETHICAL CONSIDERATIONS

  • The study used co-author annotators who consented; no external crowd workers participated.Source: §6.1 · p. 10

A DATASET SPECIFICATIONS; B DETAILED RESULTS BY TASK CATEGORY

C TASK EXAMPLES

The task examples show that annotation difficulty and human-model performance vary across emotion and sentiment classification, clustering, similarity, and reranking. The appendix also identifies quality issues across multiple datasets as relevant to those comparisons.

C TASK EXAMPLES

  • The examples use annotation interfaces, while the accompanying results illustrate divergent agreement and human-model gaps across tasks.Emotion classification has κ = 0.39 and performance of 45.8% for humans versus 87.1% for the best model; tweet sentiment has κ = 0.48 and 84.4% versus 90.9%.Source: C TASK EXAMPLES · p. 16, Figure 3 · p. 16, Figure 4 · p. 17
  • ArXiv clustering has near-zero agreement (ARI = −0.001) amid competing categorization bases, while Reddit clustering has overlapping themes and fair agreement (ARI = 0.34).SIB200 illustrates multilingual clustering with agreement varying across languages.Source: Figure 5 · p. 17, Figure 6 · p. 17, Figure 7 · p. 17
  • Robust04 reranking shows strong agreement (ρ = 0.72) and 88.5% human performance versus 98.8% for the best model; the multilingual Wikipedia example highlights cross-lingual relevance judgments.Source: Figure 8 · p. 17, Figure 9 · p. 18
  • The STS examples range from strong agreement on STS12 (ρ = 0.77; humans 91.2%, best model 92.0%) to moderate agreement on SICK-R (ρ = 0.68; humans 82.6%, best model 94.1%).The SICK-R example combines semantic relatedness and entailment.Source: Figure 10 · p. 18, Figure 11 · p. 18

D.1 DATASET QUALITY ISSUES

  • The appendix groups dataset quality issues into categories that help explain observed human-model performance patterns.Source: §D.1 · p. 19

D.1.1 STS22-RUSSIAN

The Russian STS22 subset exposes dataset-quality problems that depress human performance, while the other analyzed datasets show ambiguity in sentiment, emotion, and academic-paper clustering. These issues can make benchmark scores unreliable, and the appendix describes metric-specific confidence intervals for human results.

D.1.1 STS22-RUSSIAN

  • 58.5% for humans versus 69.5% for models on STS22-Russian; gold labels mark context-expanded sentence pairs identical even when annotators judge them semantically different.Incomplete parsing and web-page artifacts also affect interpretation.Source: §D.1.1 · pp. 20–21

D.1.2 MULTILINGUAL SENTIMENT CLASSIFICATION-RUSSIAN

  • Russian sentiment samples often come from neutral company or government press releases, complicating positive-versus-negative classification; web-page parsing artifacts are also present.Such texts may be more informative than sentiment-bearing.Source: §D.1.2 · p. 21

D.1.3 EMOTION CLASSIFICATION

  • Emotion labels are inherently ambiguous (κ = 0.39), with mixed emotional states and sarcasm allowing multiple plausible interpretations.Examples include simultaneous anxiety and happiness and sarcasm that masks anger or frustration.Source: §D.1.3 · p. 21

D.1.4 ARXIV CLUSTERING CHALLENGES

  • ArXiv clustering has no human agreement (ARI = −0.001): interdisciplinary papers can plausibly fit several categories, and annotators may prioritize methodology or application domain differently.Emerging cross-domain research adds further category ambiguity.Source: §D.1.4 · p. 22

D.2 IMPACT ON EVALUATION

  • Low-quality labels can give models artificial advantages through consistent label-pattern matching, make benchmark scores unreliable, and affect non-English tasks disproportionately.When human experts cannot agree on answers, benchmark validity is questionable.Source: §D.2 · p. 22

E.1 CONFIDENCE INTERVAL METHODOLOGY

  • With N = 20–50, human-performance 95% confidence intervals use Wilson intervals for classification, Fisher z-transformation for STS, and annotator ranges for clustering and reranking.Model scores outside a human interval are treated as significantly different at p < 0.05.Source: §E.1 · pp. 22–23

E.2 DETAILED RESULTS WITH CONFIDENCE INTERVALS

Across 26 task-language pairs, model scores fall outside human confidence intervals in 14, but low annotator agreement complicates the interpretation of apparent model advantages. The detailed results also show agreement and performance variation across task categories and languages.

E.2 DETAILED RESULTS WITH CONFIDENCE INTERVALS

  • 14 of 26 tasks show model scores outside human 95% confidence intervals; low agreement can make apparent “superhuman” scores reflect annotation artifacts, while humans outperform models on Arabic sentiment and STS22.For EmotionClassification, the best model scores 75.4 versus human performance of 45.8 (CI:[32.6, 59.7]), with κ = 0.39.Source: Table 8 · p. 23, §E.2 · p. 23

F.1 AGREEMENT METRICS BY TASK CATEGORY

  • Agreement is assessed with task-appropriate metrics and interpreted using stated thresholds, including κ bands for classification and ρ bands for correlations.For example, κ > 0.8 is labeled excellent, while 0.2 < κ ≤ 0.4 is fair; ρ > 0.7 is strong.Source: §F.1 · p. 24

F.1.1 CLASSIFICATION TASKS

  • Multilingual sentiment agreement is reported only for English (κ = 0.24); Arabic, Norwegian, and Russian use one annotator, and human performance advantages occur in non-English variants.Other listed classification results report human and best-model scores alongside annotation counts and percentage agreement.Source: §F.1.1 · p. 24

F.1.2 CLUSTERING TASKS

  • For the reported language evaluation, English shows weak agreement (ARI = 0.15; 2 annotators, 30 items), while Arabic, Danish, and Russian have single annotators and no agreement metrics; performance varies significantly across languages.Reported examples include academic-paper categories (human 49.2%, best model 84.6%) and geographical entity groupings (human 97.6%, best model 100%).Source: §F.1.2 · p. 24

F.1.3 RERANKING TASKS

  • Two reported settings have mean Kendall tau 0.85 and 0.80 (binary kappa 0.83 and 0.78); Wikipedia Multilingual Reranking reports agreement only for English (ρ = 0.64).In the two settings, human and best-model performance was 92.7% and 100%, and 85.2% and 98.8%, respectively.Source: §F.1.3 · p. 25

F.1.4 STS TASKS

  • STS22 agreement is reported only for English (ρ = 0.75), while performance varies significantly by language; the listed STS examples report human and best-model scores by task.The three provided examples give human scores of 91.2%, 80.4%, and 82.6%, respectively.Source: §F.1.4 · p. 25

F.2.1 HIGH-AGREEMENT TASKS (RELIABLE BENCHMARKS)

High annotator agreement tends to accompany clear task specifications, adequate context, and limited quality problems, making such tasks more reliable evaluation targets. Low-agreement tasks often exhibit ambiguity, subjective judgments, or inconsistent labels.

F.2.1 HIGH-AGREEMENT TASKS (RELIABLE BENCHMARKS)

  • High-agreement tasks (κ > 0.6 or ρ > 0.7) are associated with clear specifications, adequate context, cultural and linguistic familiarity, defined criteria, and few processing issues.Examples include WikiCities clustering, several reranking tasks, STS12, and STSBenchmark.Source: §F.2.1 · p. 25
  • Agreement patterns distinguish the named reliable examples from problematic ones, with the former characterized by consistent annotator performance and the latter by ambiguity or inconsistent labels.The examples listed are WikiCities clustering, News21/Core17/Robust04 reranking, STS12, and STSBenchmark versus emotion classification, ArXiv clustering, and STS22-Russian.Source: §F.2.1 · p. 25, §F.2.2 · p. 26

F.2.2 LOW-AGREEMENT TASKS (PROBLEMATIC BENCHMARKS)

  • Tasks with low agreement (κ < 0.4 or ρ < 0.6) often exhibit ambiguous guidelines or subjective judgments, cross-cultural interpretation challenges, insufficient context, systematic dataset quality issues or processing artifacts, and inconsistent or contradictory gold labels.Source: §F.2.2 · p. 26

G ADDITIONAL HUMAN VS MODEL ANALYSIS

The additional analyses compare human performance with best and median models, categorize task difficulty, and examine consistency across models. The appendix also identifies evaluated models and provides task-wise LLM-as-annotator results for selected task types.

G ADDITIONAL HUMAN VS MODEL ANALYSIS

  • Humans outperform best models on 4 of 26 task-language pairs (15.4%), but win on 61.5% against median models; the comparison varies by task category, language, and baseline.The figure reports a 29% versus 0% human win rate for multilingual versus English-only tasks against best models.Source: Figure 12 · p. 26, Figure 13 · p. 27, Figure 14 · p. 28

H MODELS EVALUATED

  • Table 9 lists evaluated models, while detailed LLM-as-annotator tables cover classification, reranking, and STS; clustering was excluded because eliciting cluster assignments from generative models was difficult.The tables are identified as task-specific results, but the supplied passages do not give their scores.Source: Table 9 · p. 27, Table 10 · p. 31, Table 11 · p. 31

G ADDITIONAL HUMAN VS MODEL ANALYSIS

  • 69% of tasks are categorized as easy by human performance (≥0.7); the figure also relates model-performance ranges to consistency and human agreement.The two tasks below 0.5 have notably low inter-annotator agreement.Source: Figure 15 · p. 29, Figure 16 · p. 30

I.1 HUMAN-LLM TASK DIFFICULTY CORRELATION

Human and LLM performance is moderately correlated across task-language pairs, while the LLM usage statement limits manuscript assistance to formatting and presentation work.

I.1 HUMAN-LLM TASK DIFFICULTY CORRELATION

  • The analysis uses Spearman rank correlations across 19 task-language pairs, excluding clustering; positive correlation means high-scoring human tasks tend also to score high for LLMs.Source: §I.1 · p. 31
  • ρ = 0.52 (p = 0.023) indicates a moderate, statistically significant correlation between human and LLM performance across 19 task-language pairs, with clustering excluded.The authors interpret this as partially shared task-difficulty patterns.Source: §I.1 · p. 31, Table 13 · p. 32

J LLM USAGE STATEMENT

  • LLM assistance covered formatting, citation integration, writing polish, and code documentation, while research design, analysis, interpretation, and conclusions remained the authors’ work.The authors say LLM assistance did not influence the study’s scientific content or conclusions.Source: J LLM USAGE STATEMENT · p. 32
Loading 2510.10062v3…