PaperlayerView exact revision on arXiv

Source-linked AI summary

Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries

Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, Rohan Anil, Ethan Dyer, Siamak Shakeri, Roopali Vij, Harsh Mehta, Vinay Ramasesh, Quoc Le, Ed Chi, Yifeng Lu, Orhan Firat, Angeliki Lazaridou, Jean-Baptiste Lespiau, Nithya Attaluri, Kate Olszewska

arXiv:2409.12640v2cs.CLcs.LG

TL;DR

Long-context benchmarks often test retrieval rather than synthesis and can be limited by leakage or short-circuiting. The paper introduces LSQ and Michelangelo, three minimal synthetic evaluations that query latent structures across arbitrarily long contexts. Across frontier models, the evaluations are high-signal but reveal substantial remaining difficulty in synthesizing long-context information.

  • Problem

    Reliable tests of whether models synthesize information across long contexts remain limited because retrieval-based evaluations may not require full-context reasoning and existing benchmarks can be leaked or short-circuitable.

    Source: §1 · p. 2, §6.2 · pp. 21–22

  • Method

    The paper develops LSQ and three minimal, synthetic, automatically regenerable tasks that hide latent structures among distractors and query the structures across natural-language and code settings.

    Source: §1 · pp. 1, 3

  • Results

    Frontier models show an initial significant degradation on these evaluations, although some retain non-trivial performance across contexts extending to 128K or 1M tokens.

    Source: §7 · p. 23

  • Takeaways & Limitations

    Michelangelo provides a high-signal diagnostic of beyond-retrieval long-context reasoning and exposes simple synthesis capabilities that current frontier models still do not reliably solve.

    Source: §1 · pp. 3–4, §7 · p. 23

    Some existing long-context reasoning benchmarks remain vulnerable to secret retrieval because their task construction does not require synthesizing information beyond retrieval.

    Source: §6.2 · p. 22

Abstract

from arXiv · show

We introduce Michelangelo: a minimal, synthetic, and unleaked long-context reasoning evaluation for large language models which is also easy to automatically score. This evaluation is derived via a novel, unifying framework for evaluations over arbitrarily long contexts which measure the model's ability to do more than retrieve a single piece of information from its context. The central idea of the Latent Structure Queries framework (LSQ) is to construct tasks which require a model to ``chisel away'' the irrelevant information in the context, revealing a latent structure in the context. To verify a model's understanding of this latent structure, we query the model for details of the structure. Using LSQ, we produce three diagnostic long-context evaluations across code and natural-language domains intended to provide a stronger signal of long-context language model capabilities. We perform evaluations on several state-of-the-art models and demonstrate both that a) the proposed evaluations are high-signal and b) that there is significant room for improvement in synthesizing long-context information.

1. Introduction

Michelangelo addresses the lack of reliable tests for long-context reasoning beyond retrieval by introducing LSQ, a framework for synthetic, unleaked evaluations that expose latent structure in arbitrarily long contexts. Its three primitives test context synthesis across natural-language and code settings, while initial results show substantial unresolved difficulty for frontier models.

Latent Structure Queries

  • LSQ treats a large context as a latent structure obscured by irrelevant information, requiring models to chisel away distractors and answer queries about the recovered structure rather than retrieve isolated values.This extends long-context testing beyond needle-in-a-haystack retrieval.Source: §1 · pp. 1–2

Motivation

  • Long-context evaluation remains difficult because retrieval success does not establish synthesis of information across the full context, while existing benchmarks may be short, artificial, labor-intensive, leaked, or short-circuitable.These limitations motivate a reliable evaluation that requires using the context rather than exploiting local or memorized information.Source: §1 · pp. 2–3

Minimality

  • Michelangelo is designed as a minimal benchmark whose tasks require beyond-retrieval understanding while remaining interpretable, easy to track, and informative about basic failure cases.Its minimality targets a small set of canonical tasks that remain challenging for present-day models.Source: §1 · p. 3

Canonical Primitives

  • Michelangelo provides three simple synthesis primitives: reproducing ordered context portions, tracking list modifications, and determining whether an answer is contained in context.The primitives span different synthesis capabilities and are instantiated in natural-language or code-based settings.Source: §1 · p. 3

Arbitrary Context Length

  • The tasks can extend to arbitrary context lengths while keeping the amount of relevant information fixed, separating intrinsic task complexity from difficulty caused by distracting context.Synthetic generation controls relevant and irrelevant information without logical contradictions or short-circuits.Source: §1 · p. 3

Results

  • Evaluations up to 1M tokens show non-trivial performance for GPT and Claude models up to 128K and Gemini models up to 1M, but all frontier models exhibit an initial sharp performance drop on these reasoning tasks.The results indicate progress in long-context generalization alongside unresolved beyond-retrieval reasoning gaps.Source: Figure 1 · p. 2, §1 · p. 4, Figure 2 · p. 4

2. Michelangelo Evaluation Tasks

Michelangelo evaluates long-context synthesis through three tasks that require reconstructing latent information from dispersed code or natural-language context rather than retrieving a single fact.

2. Michelangelo Evaluation Tasks

  • The evaluation section defines Michelangelo’s three task families and defers the unifying Latent Structure Queries framework to Section 3.The tasks span code-based list state tracking, natural-language co-reference resolution, and answerability judgments.Source: §2 · p. 4, §2.1 · p. 4, §2.2 · p. 6

2.1. Latent List

  • Latent List tracks a Python list through relevant and inert operations, then queries slices, aggregates, or length at three complexity levels.Irrelevant operations preserve context length without changing the latent list, while relevant operations are distributed throughout the input.Source: §2.1 · pp. 4–5
  • Latent List scores exact string matches for print views and normalized approximate error for numerical outputs.The approximate metric is bounded within [0, 1] and provides a wider dynamic range of signal.Source: §2.1 · p. 5

2.2. Multi-Round Co-reference Resolution (MRCR)

  • MRCR requires reproducing the response associated with a queried topic and format while resolving similar distractors and ordering across the conversation.Outputs are scored with the SequenceMatcher ratio after post-processing the model’s prefixed response.Source: §2.2 · pp. 6–7

2.3. IDK

  • IDK tests whether models recognize when a question is unanswerable from the context, using multiple-choice answers that include “I don’t know.”The evaluation mixes unanswerable cases with retrieval cases and marks explicit statements that the information is absent as correct.Source: §2.3 · pp. 7–8

3. Latent Structure Queries : A Novel Long-Context Evaluation Framework

Latent Structure Queries models a long context as updates to a hidden data structure, then tests understanding by querying slices after irrelevant information has been removed conceptually.

3. Latent Structure Queries : A Novel Long-Context Evaluation Framework

  • LSQ requires arbitrary-length extensibility, complexity indexed by relevant updates, leakage-resistant contexts, natural-language and code coverage, and minimal orthogonal evaluations.Its central abstraction treats context as a stream of updates to a latent structure.Source: §3 · p. 8
  • By querying slices rather than the entire structure, LSQ makes partial understanding testable while remaining tractable for complex contexts.The framework distinguishes beyond-retrieval synthesis from querying one or more independent keys and values.Source: §3 · p. 9, §3.1.2 · p. 10
  • Fixing relevant updates separates intrinsic task complexity from difficulty caused by long distracting context, while irrelevant updates enable arbitrary context extension.Generated filler can preserve realistic text distributions without changing the final answer.Source: §3 · p. 9
  • LSQ prevents pretraining leakage and context short-cuts by determining relevant updates and answer-preserving filler before constructing each instance.The authors contrast this control with entity substitution approaches that may retain circumstantial clues from known source material.Source: §3 · p. 9

3.1.1. Latent List

  • Latent List instantiates LSQ with a Python list whose state is updated by both meaningful and non-meaningful operations.The latent object is queried through different components of the resulting list.Source: §3.1.1 · p. 10

3.1.2. MRCR

  • MRCR extends needle retrieval by requiring ordering information across similar outputs, represented as a nested dictionary indexed by topic and writing format.The minimal version permits one additional doubly confounding instance, while deeper nesting and more distractors are straightforward extensions.Source: §3.1.2 · p. 10

3.1.3. IDK

  • IDK is the LSQ case where the queried latent structure contains no answer, so the model must rule out relevant information across the entire context.A missing dictionary key provides the simple operational analogy.Source: §3.1.3 · pp. 10–11

3.2. Context Stratification

  • Context stratification supplies cumulative subsets up to 32K, 128K, and 1M tokens, with bucket normalization for combining them.The design supports staggered model development and reduces variance when subsets are stacked.Source: §3.2 · p. 11

4. Experimental Results

Michelangelo evaluates ten frontier models on three long-context synthesis tasks up to 128K and 1M tokens, revealing task- and length-dependent differences in model performance.

4. Experimental Results

  • Gemini models perform best on MRCR at 128K, GPT models perform best on Latent List, and Claude-3.5 Sonnet performs best on IDK.The evaluations plot cumulative average scores against context length, with higher scores bounded between 0 and 1.Source: Figure 3 · p. 12, §4 · p. 12, Figure 4 · p. 13

4.1. Initial Degradation up to 128K

4.2. Gemini’s Non-Decreasing Performance from 128K to 1M

  • Gemini performance does not decrease from 128K to 1M context on the reported evaluations.The 1M-context plots include only Gemini 1.5 models, while other models are plotted up to 128K.Source: §4.2 · p. 12, §4 · p. 12
  • On MRCR, Gemini models remain non-degrading to 1M after an initial drop, while GPT and Claude models have more negative slopes at 128K.Gemini 1.5 Pro has the highest MRCR performance.Source: Figure 6 · p. 14
  • On Latent List, Gemini models remain non-degrading to 1M after an initial drop, while GPT and Claude models decline more sharply at 128K; GPT-4o leads at 128K.The passage states that GPT-4o’s performance at 1M is unclear.Source: Figure 7 · p. 14
  • On IDK, Gemini models remain non-degrading to 1M after an initial degradation, while all models appear flat afterward; Claude-3.5 Sonnet leads at 128K.Source: Figure 8 · p. 15

5. Discussion

Michelangelo’s tasks measure distinct aspects of long-context understanding and expose early degradation, cross-over behavior, complexity effects, and model-specific failure modes.

5.1. Measuring Different Aspects of Long Context Understanding

  • MRCR, Latent List, and IDK measure ordering and reproduction, latent-state tracking, and knowledge-boundary assessment, respectively.Their rank correlations across ten models provide quantitative evidence that they capture diverse capabilities.Source: §5.1 · p. 15
  • No frontier model is the clear winner across all three evaluations: Gemini leads MRCR, GPT leads Latent List, and Claude-3.5 Sonnet leads IDK.Source: §5.1 · p. 15

5.2. Degradation before 32K Context

  • Michelangelo often detects a sharp super-linear performance drop by around 32K context, followed by flattening or roughly linear degradation.Because task complexity remains fixed, the initial drop is attributed to long-context processing rather than increased task difficulty.Source: §5.2 · p. 16

5.3. Cross-Over Behaviors with Increasing Context Length

  • Models can cross over as context grows: Gemini 1.5 Flash eventually outperforms GPT and Claude on MRCR and IDK through 1M tokens.On MRCR, GPT and Claude initially outperform Gemini below 8K but show faster length-generalization decay.Source: §5.3 · p. 16

5.4. Parallel Model Family Curves on MRCR

  • Parallel MRCR curves suggest similar training-process effects across some model families, while MRCR remains robust without prompt changes.The task produces verifiable, low-variance outputs across models.Source: §5.4 · p. 17

5.5. Stratifying Latent List by Complexity

5.6. Slicing IDK Performance When the Answer is “I Don’t Know”

  • Restricting IDK to cases whose answer is “I don’t know” produces steeper degradation because the model must assess the full context for absent information.Gemini 1.5 Pro slightly exceeds Claude 3.5 Sonnet on sufficiently long contexts for this subtask.Source: §5.6 · p. 17, Figure 13 · p. 19

5.7. An Interesting Failure Mode: GPT Models Hallucinate Puzzles in IDK

  • GPT-4 models sometimes hallucinate hidden riddles in random-letter contexts, selecting an answer choice and substantially worsening IDK performance.Figure 14 provides GPT-4o examples of this failure mode.Source: §5.7 · p. 18, Figure 14 · p. 20

6. Related Work

Existing long-context evaluations often reduce reasoning to retrieval, suffer from leakage or shortcutting, or use unrealistic distractors; Michelangelo is designed to avoid these weaknesses.

6.1. Broader Work on Long Context Evaluation

  • Long-context evaluation has largely emphasized retrieval and many-shot learning, while reliable tests of reasoning over long contexts remain limited.The paper focuses on sparse relevant information that must be synthesized rather than merely retrieved.Source: §6.1 · p. 21
  • Perplexity trends on long documents may not proxy accuracy on complex long-reasoning tasks because perplexity and evaluation error move in opposite directions.Source: §6.1 · p. 21

6.2. Challenges with Developing Reliable and Minimal Long Reasoning Evaluations

  • Existing evaluations can permit short-circuiting, secretly reduce to needle retrieval, use out-of-distribution distractors, or require training on toy synthetic tasks.These properties can let models perform well without representing full-context utilization.Source: §6.2 · pp. 21–22
  • The benchmark complements SQL-focused SPIDER with orthogonal long-reasoning tests across broader settings.Source: §6.2 · p. 22
  • Michelangelo avoids these pitfalls by making relevant information unique, irrelevant information unrelated, and distractors generally similar in distribution to relevant context.Source: §6.2 · p. 22

7. Conclusion

Michelangelo introduces LSQ-based evaluations for long-context synthesis and reasoning beyond retrieval, finding substantial shortcomings in current frontier models. The evaluations are extensible, avoid leaked pre-existing content, and reveal degradation and persistent capability gaps at long context lengths.

  • Michelangelo introduces three minimal LSQ tasks—Latent List, MRCR, and IDK—that expose beyond-retrieval reasoning failures even at 32K context.The framework supports arbitrary context lengths and complexity levels while avoiding leaked content from pre-existing evaluations.Source: §7 · p. 22
  • Evaluations of the Gemini 1.5 family up to 1M context show significant initial behavioral degradation, followed by non-trivial flattening for many, but not all, frontier models.The results suggest some long-context capabilities, such as retrieval, but a significant gap remains relative to reasoning over long contexts.Source: §7 · p. 23
  • MRCR is robust across frontier models and prompting choices, produces high signal for smaller and non-post-trained models, and is recommended as a default replacement for Needle-in-a-Haystack.The recommendation reflects MRCR’s simple beyond-retrieval primitive and harder distractor contexts for retrieval.Source: §7 · p. 23

8. Contributions

The contributions section records the project’s contributors and provides prompt formats for the three Michelangelo evaluations. These formats operationalize latent-list transformations, multi-conversation retrieval and reconstruction, and uncertainty-aware question answering.

8. Contributions

  • The project distinguishes a corresponding author, core contributors, and contributors, with names listed under each category and no ordering within categories.The corresponding author led the project and created many evaluations and the overall framework.Source: §8 · p. 23

A.1. Latent List

  • Latent List prompts present Python-style list updates and ask the model to apply a view operation to a specified slice.Examples include insertion, removal, sorting, reversal, and appending operations before querying the resulting list.Source: §A.1 · pp. 28–29

A.2. MRCR

  • MRCR prompts provide multiple conversations whose follow-up requests ask the model to add a specified sentence to a designated prior response.The format includes several worked conversation examples before the target query.Source: §A.2 · pp. 29–30

A.3. IDK

  • IDK prompts pair a natural-language scene with long sequences of letters and ask a multiple-choice question that includes an explicit I-don’t-know option.The example asks which city contains the park and offers Fresno, Berkeley, Sacramento, or I don't know.Source: §A.3 · p. 30

B.1. Estimating the Chance Rate for Latent List

The Latent List chance-rate analysis estimates how well random outputs could score under its approximate metric. The resulting average chance rate is 12.2%, decreasing as task complexity increases.

  • The chance-rate estimate samples plausible list contents and slices under operation-specific assumptions, then averages scores across complexity levels 1, 5, and 20.The analysis uses different random-output assumptions for length, print, sum, min, and max operations.Source: §B.1 · p. 30
  • Figure 15 plots the per-example chance-rate histogram for MRCR under a simple random model using approximate edit distance.Source: Figure 15 · p. 31
  • 12.2% average chance rate is estimated for Latent List, with 16.9% at complexity 1, 11.3% at complexity 5, and 8.5% at complexity 20.The values are averaged over uniformly represented complexity levels.Source: §B.1 · pp. 30–31

B.1.1. Estimating the Chance Rate for MRCR

The MRCR analysis estimates chance performance under two increasingly permissive random-output assumptions. Allowing topic or format matches raises the estimated chance rate from 4% to 9%.

  • 4% is the MRCR chance rate when a random model uniformly outputs one of the possible conversation-history outputs.This estimate does not account for partial reconstruction, refusal, or less relevant outputs.Source: §B.1.1 · p. 31
  • The MRCR chance rate is 9% when computed over instances where a random model is assumed to output one matching either the topic or the format.Source: §B.1.1 · p. 31

B.2. Chance Rate for IDK

The MRCR multiple-choice query has a 25% chance rate, while Figure 16 characterizes per-example chance rates under a stronger random baseline.

  • 25% is the chance rate for the four-option multiple-choice MRCR query.Source: §B.2 · p. 31
  • Figure 16 plots per-example chance-rate histograms for a stronger random model using topic- or format-matched writing pieces and approximate edit distance.Source: Figure 16 · p. 32
  • The stronger random baseline uniformly selects among writing pieces matching at least one topic or format criterion.Source: Figure 16 · p. 32

C. Appendix: More Plots

The appendix extends Michelangelo’s model coverage with additional plots, showing improvements for newer Gemini models, persistent long-context trends, and mixed IDK behavior across model families.

More Plots

  • Figure 17 reports substantial MRCR improvements for newer Gemini models, with Gemini 1.5 Pro (08/27) surpassing competitors after around 8K context.Source: Figure 17 · p. 33
  • Gemini 1.5 Pro (08/27) outstrips all competitor models after around 20K context in MRCR.Source: C. Appendix: More Plots · p. 33, Figure 17 · p. 33
  • Newer Gemini models improve Latent List performance, especially Gemini 1.5 Flash, while generalization through 1M context remains unchanged.Source: Figure 18 · p. 33
  • IDK performance retains its context-length generalization trend through 1M context despite regressions for some newer Gemini models.Source: Figure 19 · p. 34
  • In the Unknown Answer setting, GPT models perform poorly and GPT-4o regresses relative to GPT-4T, whereas Claude performance generally rises with model size and Claude-3.5 Sonnet is the strongest exception.Source: Figure 20 · p. 34

D. Appendix: Detailed Challenges with Developing Reliable and Minimal Long Reasoning Evaluations

The appendix identifies recurring weaknesses in long-context reasoning evaluations, including short-circuiting, retrieval-only tasks, unrealistic distractors, leakage, and poor control over complexity or generalization.

Short-circuiting

Secret retrieval tasks

Out of distribution distractor context

Training on toy synthetic tasks

Design goals and trade-offs

Related papers

Loading 2409.12640v2…