PaperlayerView exact revision on arXiv

Source-linked AI summary

Load-Bearing Context: The Question Damage Score for Evaluating Context Reliance in Linguistic Reasoning

Neh Majmudar, Elena Filatova

arXiv:2608.27756v1cs.CL

TL;DR

The paper asks whether LLM answers come from supplied context or from prior knowledge, memorization, or other inference. It introduces single-example deletions and the Question Damage Score on 53 UKLO puzzles, then finds that frontier models rarely abstain after load-bearing context is removed. The framework exposes these divergences without determining their cause and supports targeted follow-up studies.

  • Problem

    Existing benchmarks reveal little about whether correct LLM answers derive from provided context rather than prior knowledge or memorization, while commonly removing entire contexts or constructing inherently unanswerable questions.

  • Method

    The paper deletes one context example using uniform random or ECC-inspired targeted deletion across 53 UKLO puzzles and uses the Question Damage Score to identify load-bearing examples.

  • Results

    Three frontier LLMs rarely abstain after structurally important context is removed and often answer correctly on structurally unsolvable variants.

  • Takeaways & Limitations

    The framework provides a controlled setting for separating memorization, prior linguistic knowledge, and inference beyond intended puzzle structure through targeted follow-ups.

  • Takeaways & Limitations

    The study does not separate memorization, prior linguistic knowledge, and other non-derivation explanations, and its structural criterion uses a conservative whitespace-token approximation.

Abstract

from arXiv · show

Determining whether large language models derive answers from context or prior knowledge remains a fundamental challenge. Self-contained linguistic olympiad puzzles provide a controlled setting where all answers derive solely from expert-designed context examples without external knowledge. Removing individual context examples can eliminate information needed for specific questions while leaving the rest of the puzzle unchanged. We leverage this to introduce a diagnostic framework for analyzing individual context examples. Using 53 UK Linguistics Olympiad puzzles, we generate two modified variants by deleting a single context example: (1) uniform random deletion, and (2) targeted deletion (inspired by error-correcting codes) to remove a structurally load-bearing example uniquely carrying necessary information. We formalize this impact using a Question Damage Score to classify puzzles as fragile or robust. Evaluating three frontier LLMs under instructions to abstain when information is insufficient, we find they rarely abstain, often continuing to produce correct answers after load-bearing context is removed. These findings motivate further investigation into context-based reasoning, prior knowledge, memorization, and linguistic inference. Beyond abstention, the framework enables fine-grained analyses of context reliance, including causal interventions, stopping-set analysis, targeted contamination studies, and mechanistic interpretability.

1 Introduction

The paper introduces single-example context deletion as a diagnostic for whether LLM answers rely on provided evidence rather than prior knowledge or memorization. Across UKLO puzzles, models rarely abstain after structurally necessary context is removed, motivating broader context-reliance analyses.

  • Existing benchmarks measure accuracy but reveal little about whether answers derive from provided context, prior knowledge, or memorization.
  • The framework deletes one context example from each of 53 UKLO puzzles using uniform random and ECC-inspired targeted strategies.Targeted deletion seeks examples that uniquely carry information required by at least one question.
  • The Question Damage Score counts questions made unanswerable by deleting an example and classifies puzzles as fragile or robust.Fragile puzzles have max_j DQ(j) > 0, whereas robust puzzles have max_j DQ(j) = 0.
  • Three frontier LLMs rarely abstain after structurally important context is removed and often answer correctly on structurally unsolvable variants.The study evaluates original and modified puzzles under explicit instructions to abstain when context is insufficient.
  • The framework supports future analyses of context reliance, including causal interventions, mechanistic interpretability, and targeted contamination studies.

2 Related Work and Positioning

Prior unanswerable benchmarks generally remove all supporting context or construct questions without answers. This work instead creates matched solvable and unanswerable puzzle variants through a single controlled deletion, while using self-contained linguistic puzzles to study context dependence.

  • Existing unanswerable benchmarks remove entire contexts, omit supporting information, or construct questions with no correct answer option.These designs make unanswerable items independently of answerable counterparts.
  • Each modified puzzle differs from its original by exactly one deleted context example, creating matched solvable and unanswerable counterparts.
  • TreeCut removes an edge from a known dependency tree in synthetic arithmetic problems, whereas linguistic puzzles require dependencies to be recovered from surface form.The linguistic structural criterion is therefore approximate rather than exact.
  • The contribution is positioned as controlled context perturbation of published linguistic puzzles rather than a new puzzle collection.The supplied passage truncates before stating the full contrast.
  • Rosetta Stone puzzles are self-contained and allow individual examples to be removed while preserving the remainder of the reasoning task.

3 Dataset Overview

The dataset comprises curated UKLO Rosetta Stone puzzles designed to test rule induction from examples without external knowledge. Its modular structure supports controlled single-example perturbations while balancing corpus scale against curation quality.

  • UKLO Rosetta Stone puzzles require inferring linguistic rules from examples and applying them to questions without relying on prior knowledge of the target language.
  • The study analyzes 53 puzzles selected from 106 UKLO puzzles published between 2010 and 2025.Puzzles lacking support for controlled single-example perturbations were excluded.
  • The resulting corpus contains an average of 16 context examples per puzzle, ranging from 6 to 38.
  • The relatively small corpus deliberately trades scale for the quality of expert design, committee review, and empirical difficulty measurement.
  • Puzzle modularity enables consequential and inconsequential deletions to be distinguished while preserving the reasoning task.

4 Deletion as a Diagnostic Probe

The diagnostic removes one context example at random or by maximum structural damage, measuring how many questions lose uniquely required information. It distinguishes fragile from robust puzzles, while acknowledging that whitespace-token damage is conservative and not directly comparable across puzzles.

  • Load-bearing deletion: Deleting Pali example 3 removes the only occurrence of hoti and makes questions 3.1.b and 3.2.f unanswerable from the remaining context.
  • Redundant deletion: Deleting Abawiri example 3 preserves solvability because its tokens and lexical-possession feature occur elsewhere.
  • Deletion strategies: Each of 53 puzzles yields two n − 1-example variants through uniform random deletion and ECC-inspired targeted deletion.Random deletion is a baseline and may remove redundant information.
  • Question Damage Score: The Question Damage Score DQ(j) counts questions requiring singleton tokens uniquely provided by context example l_j.Targeted deletion removes the example with the largest score, with ties broken arbitrarily.
  • Puzzle classification: 8 of 53 puzzles are robust, while 45 are fragile under the DQ classification.Robust puzzles have max_j DQ(j) = 0; fragile puzzles have max_j DQ(j) > 0.
  • Caveats: Whitespace-token structural damage is a conservative lower bound because singleton tokens may decompose into recurring morphemes.An LLM-as-judge cross-check distinguishes clearly destructive cases from borderline ones.
  • Caveats: DQ magnitudes are descriptive rather than directly comparable across puzzles because puzzle sizes vary, so cross-puzzle classification uses only whether DQ(j) equals zero.The implementation is a first-order approximation based on whitespace tokens and question-token dependencies.

5 Experimental Design

The experiments test whether LLMs solve UKLO puzzles and abstain when context is structurally insufficient, using original puzzles and two deletion variants. The design combines exact-match evaluation with auxiliary LLM judging, while recognizing that structural damage does not by itself establish model-perceived unsolvability.

  • Experimental setup: The study runs two experiments: direct evaluation of puzzle solving and abstention, followed by LLM-as-a-Judge assessment of whether modified puzzles remain solvable.The judge receives the original puzzle, answer key, and modified puzzle to assess whether the original answers remain derivable.
  • Evaluation: Three frontier models are evaluated through APIs using default settings and a single zero-shot pass per puzzle.The evaluated models are Claude Sonnet 4.6, Gemini 3 Flash, and GPT-5.4.
  • Experimental setup: Models are evaluated on original puzzles, random-deletion variants, and ECC-inspired targeted-deletion variants in a zero-shot structured-output setup.Each instance contains the preamble, context examples, and questions; models are instructed to use only the context and output N/A when information is insufficient.
  • Evaluation: Performance is measured with Exact Match against official UKLO answers, with abstentions additionally recorded for the modified puzzles.Results are compared across original, random-deletion, and targeted-deletion conditions.
  • Scope and validation: Structural damage identifies information removed by deletion but cannot determine whether alternative reasoning, redundancy, or prior linguistic knowledge preserves the original answers.Thus, the structural criterion and model behavior are analyzed as distinct questions rather than treated as equivalent.
  • Scope and validation: Expert solvability adjudication is confounded because language specialists may rely on known morphology rather than puzzle-internal derivability, especially across many languages.The study therefore uses LLM judging as auxiliary semantic validation rather than replacing the structural criterion.

6 Results

The evaluation compares structural damage, model judgments, abstention, and puzzle-solving behavior after single-example deletions. Aggregate EM changes little, but per-puzzle analyses reveal substantial divergences between structural solvability and model behavior.

  • Evaluation design: The evaluation combines structural Question Damage Scores, LLM-as-a-judge verdicts, and solver abstention behavior on modified puzzles.These perspectives distinguish intended structural sufficiency, model-perceived solvability, and observed answers.
  • Abstention: Abstention remains rare after deletion: GPT-5.4 abstains twice, Gemini six times, and Claude thirty times across 106 modified puzzles.After excluding pre-existing abstentions, deletion-driven abstentions reduce to two Gemini puzzles and twenty-one Claude puzzles.
  • LLM-as-a-judge: Gemini identifies 32 of 45 fragile puzzles as unsolvable, achieving 71% recall, 94% precision, and an F1 of 81%.Claude is more conservative, while GPT identifies fewer than half of fragile puzzles.
  • Contradictions analysis: 21 of 53 Gemini puzzles match or exceed original performance on judge-confirmed unsolvable variants, including seven strict improvements.Cherokee rises from 0.700 to 0.900 after targeted deletion, while Karelian rises from 0.310 to 0.540 after random deletion.
  • Contradictions analysis: Deleting redundant context can also reduce EM by as much as 0.4 while preserving structural solvability.Together, the patterns show failures both to exploit sufficient context and to restrict answers to what remaining context supports.
  • Contradictions analysis: The seven improvements and fourteen ties satisfy both deterministic structural damage and independent Gemini judgments that the modified puzzles are insufficient.These two arguments support interpreting the cases as contradictions to context-based deduction alone.
  • Aggregate performance: 0.7–7 EM points separate original and ECC-modified runs, but these aggregate gaps are within expected sampling variability for 53 puzzles.Many deleted examples are redundant, so aggregate averages can obscure destructive deletions.

7 Discussion and Conclusion

Targeted deletion exposes systematic divergences among structural sufficiency, model-perceived solvability, and solver behavior. The framework supports controlled investigation of these divergences without deciding whether they reflect memorization, prior knowledge, or other inference.

  • Discussion and conclusion: Frontier LLMs rarely abstain after load-bearing context is removed and can match or exceed original performance on judge-confirmed damaged variants.The framework does not adjudicate whether these outcomes arise from memorization, prior linguistic knowledge, or inference beyond the intended structure.
  • Future uses: The framework represents each puzzle as context-example→question dependencies that can be probed through controlled, one-node-at-a-time interventions.The released corpus includes 53 puzzles with per-example DQ scores and structural annotations.

Limitations

The study’s limitations concern corpus scale, deletion scope, tokenization, unexplained non-contextual behavior, and model coverage.

  • The corpus contains 53 puzzles, with one random and one ECC-targeted variant per puzzle.
  • Aggregate Exact Match differences are small and sensitive to additional random draws, so the authors avoid treating them as point estimates.A larger random-deletion ensemble would tighten the baseline, while the ECC procedure is deterministic.
  • Whitespace tokenization can classify morpheme-recoverable puzzles as fragile, especially in agglutinative languages.The LLM-as-judge cross-check helps identify such cases but is not a substitute for expert annotation.
  • Single-example deletion does not expose composition structures formed by chains of examples.The authors identify multi-example deletion and stopping-set analysis as extensions.
  • The contradiction pattern does not distinguish memorization, contamination, or high-resource-language heuristics.The framework instead surfaces candidate cases for targeted follow-up studies.
  • The evaluation covers three frontier models, excluding reasoning-mode ablations and additional models such as larger Gemini variants.

Data and Code Availability

The authors release the corpus, structural annotations, model judgments, prompts, and evaluation scripts in an anonymized repository.

  • The release contains 53 UKLO Rosetta Stone puzzles in structured JSON with random-deletion and ECC-targeted variants.
  • It includes Base Damage Scores, Question Damage Scores, fragility labels, and judge verdicts from all three models.
  • The repository also provides the prompts and evaluation scripts used to produce the reported results.

A UKLO Linguistic Puzzle Examples

The examples span UKLO puzzles with different years, formats, linguistic topics, language families, authors, and participant difficulty scores.

  • The Pali puzzle combines Morphology and Syntax in a Rosetta format for an Indo-European, Indo-Aryan language.Its Foundation and Intermediate difficulty scores are 39% and 61%.
  • The Waama puzzle is a Syntax-focused Match-Up task from an Atlantic-Congo, Gur language.Its Breakthrough and Foundation difficulty scores are 42% and 54%.

Rosetta Stone Example with Pictorial Context

The appendix combines UKLO puzzle examples, formal damage-score definitions, tokenization caveats, evaluation procedures, and related context-removal approaches.

  • Rosetta Stone Example with Pictorial Context: The Chinese Characters puzzle is a 2016 Rosetta task on a Sino-Tibetan writing system, with Foundation and Intermediate scores of 74% and 92%.
  • Rosetta Stone Example with Pictorial Context: The Longgu puzzle is a 2021 Rosetta task on Austronesian semantics, with an Intermediate difficulty score of 37%.
  • Damage scores: UKLO Rosetta puzzles use context examples paired with translations and questions requiring translation in either direction.Tokens are defined as whitespace-delimited character sequences, and their distribution is represented by a binary coverage matrix.
  • Damage scores: The Base Damage Score D(j) counts singleton tokens uniquely provided by context example l_j.Deleting an example with D(j) > 0 permanently removes at least D(j) tokens from the context.
  • Damage scores: The Question Damage Score DQ(j) counts questions made unanswerable by deleting context example l_j.DQ(j) > 0 implies that at least one question requires a singleton token uniquely carried by that example.
  • Damage scores: DQ is an absolute count rather than a normalized fragility rate, so cross-puzzle magnitudes are not directly comparable.The meaningful cross-puzzle comparison is whether DQ(j) equals zero.
  • Tokenization: Whitespace tokenization is language-agnostic and conservative, but can label puzzles fragile when morphemes remain recoverable.This issue is especially relevant for agglutinative languages such as Permyak; the LLM-as-judge experiment is intended to catch such cases.
  • Evaluation: The study evaluates puzzle performance and abstention across original, random-deletion, and targeted-deletion conditions using three frontier models.Exact Match is measured against official answers, while Wilcoxon and McNemar tests address paired puzzle- and question-level comparisons.

H LLM Judge Verdicts

An LLM judge independently cross-checks whether deleted-context puzzles are insufficient, then compares those judgments with model performance to identify recall and brittleness. Across 53 puzzles, the analysis examines random and ECC-inspired deletions, including statistically significant performance differences.

  • Judge-based solvability assessment: The LLM judge provides an independent, language-aware empirical cross-check of the deterministic Question Damage Score solvability assessment.The judge tests whether modified contexts are truly insufficient.
  • Judge-based solvability assessment: Table 6 reports Valid or Unsolvable verdicts for all 53 puzzles under Random and ECC-inspired deletion strategies.Bold cases mark puzzles where Exact Match increased after deletion.
  • Behavioral patterns: Models sometimes matched or exceeded their original Exact Match on puzzles judged unsolvable, providing item-level evidence consistent with pretraining recall.The derivation path was judged provably broken despite the model producing answers.
  • Behavioral patterns: Models also showed strict Exact Match drops when redundant context was removed from puzzles judged structurally solvable, indicating brittle deductive performance.These cases are reported as the inverse failure mode of successful answers on unsolvable puzzles.
Loading 2608.27756v1…