HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly

Evaluation gap

• Existing evaluations often report NIAH or arbitrary subsets of other datasets, leaving unclear whether they represent diverse downstream applications and complicating model comparisons.The introduction identifies insufficient task coverage, short contexts, unreliable metrics, and incompatibility with base models as shortcomings of existing benchmarks.Source: §1 · p. 2

Benchmark shortcomings

• For long-document QA and summarization, many existing benchmarks rely on ROUGE, which the authors describe as noisy and unreliable.Many benchmarks also require instruction-tuned models, limiting their use for evaluating base models.Source: §1 · p. 2

Benchmark comparison

• At 128K, NIAH fails to distinguish most evaluated models, while RULER and ∞BENCH show unexpected rankings; HELMET provides more consistent frontier-model rankings.The reported anomalies include Gemini Flash outperforming Gemini Pro on RULER and Llama 70B underperforming Llama 8B on ∞BENCH.Source: §1 · p. 2, Figure 1 · p. 2

HELMET

• HELMET combines seven application-centric categories with controllable lengths up to 128K tokens, reference-based model evaluation, and refined prompts that support robust evaluation of base models.Its categories include synthetic recall, long-document QA, summarization, many-shot ICL, RAG, passage re-ranking, and generation with citations.Source: §1 · pp. 2–3

Study findings

• Across 59 LCLMs, synthetic tasks poorly indicate downstream performance, HELMET categories show distinct trends, and open models lag closed models on long-context reasoning and complex instructions, with the gap widening as length increases.The authors also find RAG offers a balance of ease of use, downstream correlation, and compatibility with base models.Source: §1 · p. 3

2.1 REALISTIC AND DIVERSE LONG-CONTEXT APPLICATIONS

• Open-domain QA represents RAG through answer-bearing passages and distractors; citation generation tests passage use and attribution, while requiring long answers with sentence-level citations.RAG examples use Natural Questions, TriviaQA, HotpotQA, and PopQA; citation generation uses ASQA and QAMPARI.Source: §2.1 · pp. 3–4
• Passage re-ranking tests relevance comparison and reasoning over shuffled MS MARCO passages, while many-shot ICL tests learning with balanced examples from datasets with large label spaces.ICL labels are mapped to numbers to test learning new tasks rather than reliance on pretrained label priors.Source: §2.1 · pp. 4–5
• Long-document QA uses NarrativeQA and ∞BENCH book QA and multiple-choice subsets; summarization uses legal and novel documents selected for extensive lengths.Documents are truncated to the chosen input length; NarrativeQA uses model-based evaluation, while summarization uses model-based scoring instead of ROUGE.Source: §2.1 · p. 5
• Synthetic recall includes RULER tasks and a more challenging JSON key-value retrieval task, with task selection intended to correlate with application-driven tasks.The benchmark reports the percentage of ground-truth answers appearing as substrings in the generation (SubEM).Source: §2.1 · p. 5

2.2 RELIABLE EVALUATION METRICS

• HELMET uses reference-based model evaluation for long-document QA and summarization, addressing the poor human-judgment correlation of overlap metrics and limitations of reference-free pairwise win rates.For summarization, GPT-4o checks claim support and fluency; for NarrativeQA, it scores fluency and correctness.Source: §2.2 · pp. 5–6, Figure 2 · p. 6
• The authors report that model-based metrics distinguish models and incoherent outputs more clearly than ROUGE, and track GPT-4o’s gains with longer input; human-study agreement on ∞BENCH Sum reaches κ = 0.91 for precision and κ = 0.76 for recall.For GPT-4o, ROUGE-L stays within a 2-point absolute difference as input length increases.Source: §2.2 · p. 6, Figure 2 · p. 6

2.3 ROBUST PROMPTING AND CONTROLLED EVALUATION SETTINGS

• Two-shot demonstrations address inconsistent output formats and benefit both base and instruction-tuned models; length-enhanced instructions are used for long-generation tasks.For long-document QA and summarization demonstrations, the original document is replaced with a placeholder phrase to reduce prompt tokens.Source: §2.3 · p. 6
• Input length is controlled by adjusting retrieved passages, demonstrations, or document truncation, enabling study at or beyond 128K tokens; the authors report HELMET rankings more consistent with human perception.The benchmark also uses length-instruction-enhanced evaluation for long-generation tasks.Source: §2.3 · pp. 6–7

3.1 SIMPLE SYNTHETIC TASKS ARE POOR PREDICTORS OF REAL-WORLD PERFORMANCE

• None of the synthetic tasks has an average correlation above 0.8; original NIAH correlations are all ≤0.8 and RULER average-score correlations are all <0.85.The analysis calculates Spearman rank correlations between synthetic and real-world tasks across 35 instruction-tuned models.Source: §3.1 · p. 7, Figure 3 · p. 7
• RULER MK’s distracting needles yield clearer model differentiation, while HotpotQA correlates more highly with other real-world tasks.RAG contexts include retrieved, relevant passages, making them more distracting and harder to saturate than synthetic recall contexts.Source: §3.1 · pp. 7–8

3.2 DIVERSE LCLM APPLICATIONS CALL FOR DIVERSE EVALUATION

• Different HELMET categories do not consistently correlate: RAG and re-ranking correlate moderately, while citation generation and ICL correlate poorly with other categories.Summarization and ICL test aggregation and learning from examples, capabilities described as orthogonal to long-context fact recall.Source: Figure 5 · p. 8, §3.2 · p. 8

3.3 MODEL PERFORMANCE ACROSS TASKS AND LENGTHS

• At 128K, closed-source models outperform open-source models on every task except ICL, with 30 to 40 absolute points separating them on citation generation and re-ranking.GPT-4o-08 and Gemini-1.5-Pro are notably strong.Source: §3.3 · p. 8, Figure 6 · p. 9
• No model leads every category: GPT-4o excels in recall and citation generation, Gemini in re-ranking and long-document QA, and many open models outperform closed ones on ICL.The authors suggest heavy instruction tuning may negatively impact ICL.Source: §3.3 · p. 9
• As task complexity rises, longer inputs cause greater performance degradation; open-source models collapse on citation generation at 128K while GPT-4o remains relatively stable.Most frontier models largely retain performance on recall and RAG as inputs lengthen, but re-ranking and citation generation degrade significantly.Source: §3.3 · p. 9, Figure 7 · p. 10

Long-context language models

• Long-context model work spans expanded context windows, longer-input training, position extrapolation, and efficient architectures.The passage names GPT-4, Gemini, and Claude among frontier models claiming context windows beyond 100K tokens.Source: §4 · p. 10

Synthetic tasks

• Synthetic tasks allow controlled, arbitrarily long inputs, but prior work had not studied how their results transfer to real applications.HELMET evaluates synthetic and downstream tasks together to examine their correlations.Source: §4 · p. 10

Long-context benchmarks

• Existing benchmarks are often short or lack rigorous evaluation, while many works address only specific domains; HELMET evaluates diverse downstream tasks at long lengths.The paper also presents a unified comparison and analysis across 59 LCLMs.Source: §4 · p. 10

5 CONCLUSION

• Synthetic tasks poorly predict downstream performance, HELMET categories show distinct capabilities and trends, and open-source models lag on complex tasks at longer lengths.The authors conclude that evaluating across diverse real-world tasks is essential.Source: §5 · p. 10

REPRODUCIBILITY STATEMENT

• The authors publicly released code and data for reproducibility and plan to release model-generated outputs and statistics for further analysis and comparison.Source: REPRODUCIBILITY STATEMENT · p. 11

• Without large-scale human studies, the authors say true LCLM rankings remain challenging to discern.Source: A COMPARISON WITH OTHER BENCHMARKS · p. 29
• They use prior work and empirical observations, including reported size-related performance patterns, to guide ranking expectations.The cited Gemini report describes larger models outperforming smaller ones on long-context handling across Gemini, GPT, and Claude families.Source: A COMPARISON WITH OTHER BENCHMARKS · p. 29
• The section compares benchmark rankings and performance and supports its examination of HELMET’s improvements with ablation studies.Source: A COMPARISON WITH OTHER BENCHMARKS · p. 29

A.1 RESULTS

• The reported ∞BENCH results were reproduced using the original authors’ code, while Table 5 reports 128K results across benchmarks.The authors say the numerical results for Figure 1 are presented in Table 5.Source: §A.1 · p. 29, Table 5 · p. 29
• Llama-3.2 models degenerate on ∞BENCH at 128K, yet rank well against open-source models on HELMET.The authors attribute their coherent, useful long-context generations to HELMET prompting strategies such as adding in-context examples.Source: §A.1 · p. 29
• Two-shot demonstrations substantially improve base-model performance, better reflecting real-world usage patterns.The study also reports that improved instructions and prompting let smaller Llama-3.2 models compete effectively with larger models.Source: §A.1 · p. 29

A.2 DISCUSSIONS

• Earlier evaluations often isolated task types, while many unified benchmarks remained limited by context length, metrics, or task coverage.The authors report evaluating 59 models across sizes and architectures to study correlations across tasks and the model landscape.Source: §A.2 · pp. 29–30
• HELMET’s prompting and metric design aims to make its broad task evaluation a more accurate reflection of model capabilities in practice.The discussion contrasts this unified approach with prior work that evaluated only a few domains or relied on ROUGE for summarization.Source: §A.2 · pp. 30–31

B.1 RETRIEVAL-AUGMENTED GENERATION

• For RAG, HELMET uses four open-domain QA datasets, and PopQA is filtered to reduce the influence of pretraining memorization.Its contexts include hard negatives retrieved from the same corpus as positive passages, making models distinguish relevant from irrelevant information.Source: §B.1 · pp. 31–32

B.2 GENERATION WITH CITATIONS

• Citation generation uses ALCE with Wikipedia retrieval and reports the average of correctness and citation quality.Models answer multifaceted questions with long text and cite supporting passage IDs at sentence ends.Source: §B.2 · p. 32

B.3 PASSAGE RE-RANKING

• Passage reranking uses MS MARCO examples with relevance labels, randomized passage order, and NDCG@10 evaluation.The setup balances label distribution and uses three passage permutations to reduce positional bias.Source: §B.3 · p. 32

B.4 IN-CONTEXT LEARNING

• ICL label mapping replaces natural-language labels with randomly assigned integers so classification relies on demonstrations rather than pretrained label knowledge.Examples are arranged in randomized rounds containing one example per label, and the evaluation uses 500 samples.Source: §B.4 · p. 32

B.5 SYNTHETIC TASKS

• Synthetic tasks include NIAH variants, multi-key and multi-value retrieval, variable tracking, word extraction, and question answering.The JSON KV task uses random UUID keys and values, with six queries at evenly spaced context positions; RULER data is standardized with the Llama-2 tokenizer.Source: §B.5 · pp. 32–33

B.6 MODEL-BASED EVALUATION

• The evaluation uses GPT-4o as a judge because ROUGE can be unreliable, and long-document QA outputs may be correct despite little lexical overlap with references.For NarrativeQA, the judge assesses fluency and correctness against the question and ground truth.Source: §B.6 · p. 33
• Model-based evaluation ranks Claude above Llama-3.2-3B-Inst, whereas F1 penalizes Claude’s extra assistant-like text.The authors say the judge catches this issue, which may better match expectations given the models’ sizes.Source: §B.6 · p. 33
• For summarization, the judge scores fluency, precision, and recall, then multiplies fluency by the precision-recall F1 score.Precision measures generated sentences supported by the reference; recall measures key points supported by the generation.Source: §B.6 · p. 33
• Human checks found substantial agreement with model judgments: κ = 0.76 for ∞BENCH Sum and κ = 0.72 for Multi-LexSum on recall.Precision agreement was κ = 0.91 for ∞BENCH Sum and κ = 0.83 for Multi-LexSum; fluency judgments matched human judgments 100% on sampled outputs.Source: §B.6 · p. 34
• Most disagreements arose in partially supported cases, where judges could be more lenient than human raters about omitted details.The authors note that human preferences about whether small details must be included are subjective.Source: §B.6 · p. 34

D EXPERIMENTAL SETUP

• Models are evaluated at 8,192, 16,384, 32,768, 65,536, and 131,072 tokens, but limited compute prevents some large models from running at 128K.Open-source evaluations use H100 GPUs with 80GB of memory; the available compute is eight H100 GPUs.Source: D EXPERIMENTAL SETUP · p. 34
• Evaluation sample sizes vary by dataset, from 100 examples for remaining datasets to 600 for JSON KV, NQ, PopQA, and TQA.The setup uses 300 examples for MSMARCO and HotpotQA and 500 for ICL.Source: D EXPERIMENTAL SETUP · p. 34
• The QA judge scores fluency as 0 or 1 and correctness from 0 to 3, then outputs both scores in JSON.Correctness distinguishes incorrect, partly correct, correct but not fully relevant, and correct and relevant answers.Source: D EXPERIMENTAL SETUP · p. 35

E.1 CORRELATION BETWEEN SYNTHETIC AND DOWNSTREAM TASKS

• RAG datasets are generally more correlated with downstream tasks than synthetic tasks, while greater synthetic-task complexity and noisier contexts accompany stronger correlations.Needle MK Needle/UUID, JSON KV, and NIAH MV generally correlate relatively highly with realistic datasets and test recall.Source: §E.1 · p. 35
• The HELMET synthetic-recall subset includes four selected datasets whose combined correlations with downstream tasks are generally higher than those of individual synthetic datasets.The authors note that more optimal selection methods may exist and leave them for future work.Source: §E.1 · pp. 35–36

E.2 CORRELATION BETWEEN DATASETS

• ALCE datasets lack correlation, suggesting citation writing differs from answering factual questions; long-document QA and ICL also show some low correlations.Datasets within each category are generally strongly correlated, while diversity may explain some low correlations in QA and ICL.Source: §E.2 · p. 36, Figure 8 · p. 37

E.3 POSITIONAL EMBEDDING EXTRAPOLATION REMAINS A CHALLENGE

• Performance drops sharply past L = 32768 for both positional-embedding model families, and changing embeddings may also reduce shorter-length ODQA and ICL performance.The trend also holds across model sizes from 8B to 70B.Source: §E.3 · pp. 37–38, Figure 9 · p. 38

E.4 PRESENCE OF LOST IN THE MIDDLE

• Models generally prefer recent context and struggle with middle and earlier facts, though at long inputs middle-position retrieval often exceeds retrieval at the start.The analysis places retrieval targets at six evenly spaced depths, extending to 128K tokens across JSON KV and question-answering tasks.Source: §E.4 · p. 39

E.5 COMPARISON BETWEEN BASE AND INSTRUCTION-TUNED MODELS

• Instruction tuning generally improves performance across tasks; Llama-3.1-8B-Inst outperforms Llama-3.1-70B on most tasks.HELMET supports both base and instruction-tuned models using in-context learning examples.Source: §E.5 · p. 39

E.6 PERFORMANCE OF CLAUDE

• Claude-3.5-Sonnet’s relatively low scores are attributed to not following task instructions, sometimes refusing, and occasionally producing truncated outputs.Examples include omitting citation markers, rankings, or classification labels.Source: §E.6 · p. 39

E.7 FULL RESULTS

• Detailed evaluation numbers for each dataset at each length are provided via the appendix’s linked resource.The passage supplies the link shorturl.at/5wjVl.Source: §E.7 · p. 39
• The appendix’s model table defines its length field as training length or, when unavailable, claimed context window.For MoE models it reports active and total parameters, and marks some inference-time RoPE changes with -θ.Source: Table 15 · p. 43
• Figures 11–14 report model performance by gold-item depth for JSON KV, Natural Questions, PopQA, and TriviaQA.The listed depth positions range from 0.0 at the context beginning to 1.0 at its end.Source: Figure 11 · p. 45, Figure 12 · p. 46, Figure 13 · p. 47