PaperlayerView exact revision on arXiv

Source-linked AI summary

MWE-ECL: Recoverable Long-Range Context Does Not Always Override Local Lexical Priors

Wei He, Aline Villavicencio, Rodrigo Wilkens, Zhenyun Deng

arXiv:2609.27590v1cs.CL

TL;DR

Long-context evaluations can establish whether distant evidence is recoverable without showing whether it changes a decision opposed by a local expectation. MWE-ECL uses matched retrieval, no-anchor prior, and interpretation prompts to test that distinction. Its results show that near-ceiling retrieval can coexist with lower prior-conflict override, motivating evaluations that measure behavioral influence as well as access.

  • Problem

    Long-context evaluations often test distant-evidence recovery, but whether recoverable evidence changes a decision opposed by a strong local expectation remains a separate question.

    Source: §1 · p. 1

  • Method

    MWE-ECL is a bilingual diagnostic using matched interpretation, anchor-retrieval, and no-anchor prior-control prompts to measure anchor-conditioned decisions, explicit recoverability, and model-observed defaults.

    Source: §1 · p. 1

  • Results

    On 146 shared 0–64K prior-conflict rows, DeepSeek V4 Pro reached 0.993 retrieval, 0.890 interpretation, and 0.768 override (69 conflicts).

    Source: §6.1 · p. 5

  • Takeaways & Limitations

    Long-context benchmarks should test whether recoverable distant evidence changes a competing local decision, not only whether it can be recovered.

    Source: §7 · p. 7, §9 · p. 7

    The diagnostic uses only 25 English and 10 Chinese MWE families; Chinese difficulty is unmatched, and maximum-endpoint ECL is right-censored tested-range coverage.

    Source: §8 · p. 7

Abstract

from arXiv · show

Long-context evaluations often test whether a model can recover distant evidence, but recoverability does not guarantee behavioral influence. We test the prediction that a distant discourse anchor can remain explicitly recoverable yet fail to change the locally preferred reading of a familiar multiword expression; such failures should concentrate when the model's no-anchor default conflicts with the anchor, while prior-correct decisions remain largely preserved. We introduce Multiword Expression Effective Context Length (MWE-ECL), a bilingual diagnostic whose matched anchor-retrieval, no-anchor prior, and interpretation prompts measure explicit recoverability, model-observed defaults, and anchor-conditioned decisions, respectively. Across eight English deployment panels on a shared 0-128K grid, retrieval-control accuracy on prior-conflict items is 0.989-1.000, prior-conflict override spans 0.806-1.000 (0.809-1.000 after conditioning on correct retrieval), and preservation of prior-correct decisions remains 0.977-1.000. A same-call control querying retrieval and interpretation in one prompt reproduces the gap for DeepSeek V4 Pro (1.000 retrieval versus 0.900-0.920 interpretation), showing that separate invocations are not its sole explanation; smaller or absent gaps in the other two models bound its generality. For DeepSeek V4 Flash, separate prompt-fit tests retain perfect retrieval with lower interpretation at 512K and 1M, while foil-consistent cues shift the no-anchor prior far more than retrieval; cross-model cue effects are heterogeneous. A separately reported 10-family Chinese subset shows similar descriptive gaps, but imperfect retrieval for some models prevents an integration-only attribution. MWE-ECL therefore evaluates whether explicitly recoverable distant context changes a competing local semantic decision.

1 Introduction

MWE-ECL tests whether distant evidence that a model can recover changes a decision opposed by a strong local expectation. Matched prompts distinguish anchor access, the model’s local default, and its anchor-conditioned interpretation.

1 Introduction

  • The paper frames effective context length around whether recoverable distant evidence changes a decision conflicting with a strong local expectation.A million-token window specifies nominal capacity, not guaranteed effective use.Source: §1 · p. 1
  • MWE-ECL uses multiword expressions to test prior override: a distant discourse anchor can license a reading that conflicts with the familiar local reading.The same local trigger appears across senses, so the anchor—not lexical familiarity—alters the decision.Source: §1 · p. 1
  • Matched interpretation, anchor-retrieval, and no-anchor prior-control prompts distinguish anchor access, local defaults, and anchor-conditioned decisions.ECL is reported as right-censored tested-range coverage, while prior-conflict override is the targeted context-use metric.Source: §1 · p. 1
  • The research questions test distant-anchor access, override of a conflicting no-anchor choice, and robustness across distances, interference, languages, and same-call controls.The section notes that a short current turn may depend on a preference, policy exception, or state update established much earlier.Source: §1 · p. 1
  • MWE-ECL is a bilingual diagnostic that separates explicit anchor recoverability, model-observed defaults, and anchor-conditioned decisions.Its measures distinguish correction of a wrong local default from access and corruption of an already-correct default.Source: §1 · p. 1
  • The paper reports a recurring behavioral access–use dissociation in English across eight deployment panels and context scales, with descriptive supporting evidence under semantic interference and in Chinese.Figure 1 depicts the matched prompt heads; high retrieval with lower interpretation indicates insufficient contextual influence at the behavioral level.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

Prior long-context work broadens evaluation of access, integration, and retrieval, while knowledge-conflict studies examine context competing with existing evidence. MWE-ECL positions its contribution as a controlled lexical-semantic test of whether distant licensing evidence changes an interpretation.

Long-context access and use

  • Long-context suites broaden evaluation from access to aggregation, reasoning, and low-overlap retrieval, while MWE-ECL asks whether served context changes behavior.Prior work also reports position effects and harder retrieval as lexical overlap and evidence locality decrease.Source: §2 · p. 2

Beyond retrieval: integration and knowledge conflict

  • Beyond-retrieval and knowledge-conflict work motivates testing whether accessible context changes a response when local or parametric evidence supports an alternative.MWE-ECL contributes a controlled lexical-semantic version, with no-anchor and retrieval heads measuring the competing default and explicit scenario access.Source: §2 · p. 2

Retrieval-Augmented Generation and Agent Memory

  • Unlike RAG evaluations, MWE-ECL places the anchor directly in context and tests whether it licenses a reading of a later ambiguous expression.This removes external document selection as a source of failure and differs from an anchor that simply contains the final answer.Source: §2 · p. 2

Multiword expressions

  • MWE-ECL separates a visible phrase from its licensing evidence, pairs literal and figurative cases, and inserts controlled long-distance filler.Matched retrieval and model-specific prior controls distinguish failure to access the licensing scenario from failure to use it.Source: §2 · p. 2

3 Benchmark Design

MWE-ECL is a diagnostic stress test that separates explicit anchor recoverability, model-observed defaults, and anchor-conditioned decisions. Its controlled prompts measure whether distant context overrides a conflicting local default while tracking preservation and retrieval-conditioned performance.

3 Benchmark Design

  • MWE-ECL stress-tests familiar expressions where a conventional local reading competes with an alternative licensed by distant discourse, separating access failure from insufficient contextual influence.It is not a benchmark of average-case language understanding; its conclusions are behavioral rather than hidden-state claims.Source: §3 · pp. 2–3

3.1 Task Structure

  • The task holds the trigger and answer options fixed while varying the distant anchor, with neutral filler increasing context size and anchor–decision distance.The primary grid does not disentangle those two effects, trading document naturalism for a controlled access–use contrast.Source: §3.1 · p. 3, Table 1 · p. 3

3.2 Controls and Hard-Cue Extension

  • Retrieval asks for anchor-scenario identification, while the no-anchor control preserves the trigger and options to elicit an operational model default; hard-cue variants add target-consistent, foil-consistent, or mixed cues.Because the main heads use separate calls, their contrast is between-call; a same-call control tests whether the gap persists within one invocation.Source: §3.2 · p. 3

3.3 Metrics

  • Joint ECL is right-censored coverage at the furthest tested distance where retrieval and interpretation both pass, not an estimated failure boundary.Override measures correctness on prior-conflict identities, while preservation measures correctness on prior-correct identities; corruption is the complement of preservation.Source: §3.3 · pp. 3–4
  • Retrieval-conditioned override uses prior-conflict identities with a non-error retrieval prediction, but success in a separate matched call does not establish retrieval by the interpretation call.These rates are model-specific; cross-model comparisons additionally use conflict-set overlap and a fixed consensus-hard subset.Source: §3.3 · pp. 3–4

4 Dataset

The bilingual core contains matched English and Simplified Chinese subsets, with newly authored benchmark prompts and separate reporting across languages. The English catalog intentionally emphasizes expressions with strong figurative defaults, limiting generalization to weaker-prior constructions.

4 Dataset

  • The core contains 50 English cases across 25 MWE families and 20 Chinese cases across 10 families, with one literal and one figurative case per family.The English expressions span four construction types; 38 were cross-checked against DICE and MAGPIE, while all anchors, triggers, questions, and options were newly authored.Source: §4 · p. 4
  • The idiom-heavy English catalog intentionally stresses strong figurative defaults, so results may not generalize to constructions with weaker or domain-specific priors.The separate 60-case expansion broadens coverage but is not difficulty-matched, human-validated, pooled, or used for ranking.Source: §4 · p. 4
  • The 20-case Chinese subset uses the same three tasks at seven distances through 128K but is reported separately because language difficulty is not matched.Its 10 families were selected from XMPIE, retaining source expressions and sense evidence but not source sentences.Source: §4 · p. 4
  • Blind annotators matched all intended anchor-licensed labels, with 100/100 English judgments and 40/40 Chinese judgments, and flagged no items.Automated checks also covered schema, class balance, provenance, prompt construction, filler generation, and static leakage.Source: §4 · p. 4

5 Experiments

Eight deployment panels share the same seven-distance English grid, while separate stress regimes examine longer contexts and cue conditions. The evaluation protocol treats these regimes as distinct and limits cross-model comparisons to descriptive deployment snapshots.

5.1 Models

  • Eight API deployment panels completed the same seven-distance 0–128K English grid, with the corresponding Chinese grid reported separately.Comparisons across mismatched grids or endpoint configurations are descriptive deployment snapshots, not architecture rankings.Source: §5.1 · p. 4

5.2 Implementation

  • Prompts instruct models to use the entire document and answer with one letter, using temperature 0 and a four-token answer budget.Completed but unparsed outputs count as incorrect, while provider failures are excluded and reported separately; five panels disable extended thinking and three use provider defaults.Source: §5.2 · p. 4

5.3 Evaluation Protocol

  • Primary within-model tests compare retrieval and override on identical conflict rows, with whole-family bootstrap intervals treated as primary and Holm-corrected sign-flip tests as inferential checks.Other headline panels are single-pass, and the bootstrap intervals quantify item/family variation rather than API nondeterminism; same-call, hard-cue, prompt-fit, expansion, and Chinese analyses remain separate regimes.Source: §5.3 · p. 5

6 Results

Across the shared English grid, models often retrieve distant anchors accurately without overriding a conflicting local default, while usually preserving prior-correct decisions. Same-call and stress tests reproduce or qualify this access–use gap, with heterogeneous cue effects and descriptive Chinese results.

6.1 Near-Ceiling Recoverability Does Not Ensure Prior-Conflict Correction

  • At τ = 0.8, all eight panels pass at 128K, so joint ECL is right-censored; at τ = 0.9, it is 512 tokens for DeepSeek V4 Pro and 64K for MiniMax-M3.Prior-control accuracy is 0.486–0.514, while 0K-to-128K no-anchor semantic agreement is 0.820–0.960; near-0.5 accuracy does not mean defaults are random or wholly unstable.Source: §6.1 · p. 5, Table 3 · p. 6
  • On 146 shared 0–64K rows, DeepSeek V4 Pro retains 0.993 retrieval, 0.890 interpretation, and 0.768 override after balanced-label and same-order checks.At 128K, interpretation ranges from 0.860 to 1.000 across the listed panels and models.Source: §6.1 · p. 5

6.2 The Gap Is Not Solely a Separate-Call Artifact

  • In the same-call control, DeepSeek V4 Pro retrieves all 50 anchors correctly but makes four or five interpretation errors in each 50-case cell.Order effects are small, with family-bootstrap confidence intervals including zero; the GLM-5.1 ceiling result bounds generality.Source: §6.2 · p. 5, Table 4 · p. 6

6.3 Failures Concentrate Where Local Defaults Conflict

  • Prior-correct preservation is 0.977–1.000, while failures concentrate on prior-conflict decisions; all 19 cold feet errors pair correct literal-anchor retrieval with figurative choices.For DeepSeek V4 Pro, MiniMax-M3, and Qwen3.6-35B, respectively, 4/175, 2/170, and 1/173 prior-correct rows are corrupted; Holm-adjusted sense tests remain descriptive and do not support a universal sense effect.Source: §6.3 · p. 5

6.1 Near-Ceiling Recoverability Does Not Ensure Prior-Conflict Correction

  • Retrieval is 0.989–1.000, while prior-conflict OR is 0.806–1.000 and OR|Ret is 0.809–1.000 across the shared grid.DeepSeek V4 Pro has retrieval 0.989 and OR 0.806 over 175 conflicts, a 0.183 gap; its OR|Ret is 0.809.Source: §6.1 · p. 5, Figure 2 · p. 6

6.4 Stress Tests Reproduce and Bound the Dissociation

  • The Chinese subset shows descriptive gaps, but imperfect retrieval in some panels and only 10 families prevent an integration-only attribution or cross-language difficulty comparison.DeepSeek V4 Pro flips 8 of 70 prior-correct rows in Chinese (Pres. 0.886); the subsets are reported separately and are not pooled or difficulty-matched.Source: Table 3 · p. 6, §6.4 · p. 6
  • The 60-case expansion shows override 0.867–0.897 versus retrieval 0.961–0.983 across four models, but is a lexical-coverage check, not ranking evidence.The small-N position pilot does not rule out a start-position advantage.Source: §6.4 · pp. 6–7, Table 5 · p. 6
  • At 256K, 512K, and 1M, DeepSeek V4 Flash retains 1.000 retrieval with lower interpretation, but endpoint-specific prompt fits are not a monotone extension of the matched grid.With 4 and 16 foil-consistent cues, its interpretation is 0.873 and 0.867; at 16 cues, the no-anchor prior shifts more than retrieval or retrieval-conditioned integration.Source: §6.4 · pp. 5–7, Table 6 · p. 7, Table 7 · p. 7

7 Discussion

The discussion frames MWE-ECL as a controlled way to test whether accessible context changes a plausible local decision. It recommends pairing access controls with conditional correction and preservation measures, while limiting the claims to the diagnostic's scope.

7 Discussion

  • Long-context evaluations should pair an access control with a task where retrieved evidence must alter a plausible local decision.The proposed pattern includes a distant state update, a direct state query, and a no-state control that preserves the local decision surface.Source: §7 · p. 7
  • Conditional correction and preservation distinguish inaccessible state, accessible but behaviorally inert state, and harmful overwriting; these metrics should accompany overall accuracy.Overall accuracy alone cannot separate correction from preservation of an already-correct default.Source: §7 · p. 7
  • MWE-ECL is one controlled instantiation, not a claim that MWE errors predict agent failures or fully measure long-context reasoning.Override uses a model-specific denominator, so deployment-level rankings remain descriptive despite agreement across the analyses.Source: §7 · p. 7

8 Limitations

The evaluation supports a behavioral access–use dissociation, but its controls do not expose the interpretation call’s hidden retrieval state. Its findings are also bounded by a small, idiom-heavy set and deployment-specific conditions.

8 Limitations

  • The same-call control omits the prior head, and forced-choice decisions do not reveal underlying mechanisms.Because the no-anchor note and semantic anchor differ in content and length, their contrast is a benchmark-specific behavioral effect, not a semantics-only intervention on an internal prior.Source: §8 · p. 7
  • The diagnostic covers only 25 English and 10 Chinese MWE families, with Chinese difficulty unmatched to English and no matched human validation for the larger expansion.The set uses idiom-heavy strong-default MWEs and neutral administrative filler.Source: §8 · p. 7
  • Six headline panels are single-pass API snapshots, and maximum-endpoint ECL values are right-censored tested-range summaries rather than estimates of a monotone failure boundary.Coverage reflects API availability during the run window, and reasoning modes vary by endpoint.Source: §8 · p. 7

9 Conclusion

MWE-ECL separates distant-anchor recoverability, no-anchor defaults, and anchor-conditioned interpretation. Its behavioral findings motivate evaluating whether recoverable distant evidence changes a competing local decision, within the scope of the paper’s deployment snapshot.

9 Conclusion

  • MWE-ECL separates distant-anchor recoverability, no-anchor defaults, and anchor-conditioned interpretation; near-ceiling retrieval can coexist with substantially lower contextual override.The conclusion describes the findings as behavioral and deployment-specific, while calling for long-context benchmarks to test whether distant evidence changes a competing local decision.Source: §9 · p. 7

A Scope and Reading Guide

  • The evaluation uses matched case–distance identities, with override conditioned on a model’s wrong no-anchor prediction and retrieval-conditioned override additionally requiring correct matched retrieval.Intervals use whole-family bootstrap resamples, and paired family-level tests use Holm correction across eight prespecified headline panels.Source: A Scope and Reading Guide · p. 10

B Evaluation Snapshot

  • The English headline grid contains 50 cases, 25 MWE families, three task heads, and seven distances from 0 to 128K per model.The API-interface panel is a time-stamped deployment comparison, not an architecture ranking; stress, hard-cue, expansion, and Chinese regimes are reported separately.Source: B Evaluation Snapshot · p. 10

C Primary Statistical Accounting

The statistical accounting shows that interpretation failures concentrate on prior-conflict correction, while prior-correct decisions are more often retained. It also distinguishes tested-range ECL coverage from a failure-boundary estimate and specifies how model-specific conflict sets are handled.

C Primary Statistical Accounting

  • Every joint-ECL entry passes at the largest tested endpoint under τ = 0.8, so it is a right-censored tested-range summary rather than a monotone failure-boundary estimate.Table S2 provides the interval accounting omitted from the compact main table.Source: C Primary Statistical Accounting · p. 11, Table S2 · p. 12
  • Override uses each model’s own no-anchor prediction to define conflict rows; the accounting also reports pairwise conflict-set overlap and a fixed 182-identity consensus-hard subset as sensitivity analyses.Neither cross-model analysis substitutes for within-model override.Source: C Primary Statistical Accounting · p. 11, Table S4 · p. 12
  • Across the five non-ceiling panels, prior-conflict correction failures range from 11 to 34, while prior-correct corruption ranges from zero to four.Family-macro override closely tracks the micro-pooled estimate, indicating the result is not driven by one unusually frequent MWE family.Source: C Primary Statistical Accounting · p. 11, Table S5 · p. 12
  • At 128K, no-anchor semantic choices agree with their 0K counterparts on 0.820–0.960 of cases, and retention among items correct at 0K and retrieval-correct at 128K ranges from 0.872 to 1.000.Raw option-A rates remain between 0.260 and 0.400, separating semantic default movement from a fixed answer-position bias.Source: C Primary Statistical Accounting · p. 11, Table S6 · p. 13

D Same-Call and Semantic-Interference Controls

The controls test whether the access–use pattern persists in a shared call, under semantic cues, across repeated runs, and near serving limits. They support a bounded behavioral comparison rather than a general ranking or positional curve.

D Same-Call and Semantic-Interference Controls

  • In the same-call control, retrieval is perfect in all DeepSeek V4 Pro and GLM-5.1 cells and three of four MiniMax-M3 cells, while interpretation remains below retrieval for DeepSeek V4 Pro and MiniMax-M3 under both query orders.Both decisions are decoded from one response, and query order is counterbalanced; the control does not expose the prior head.Source: D Same-Call and Semantic-Interference… · p. 14, Table S7 · p. 14
  • Foil cues reduce interpretation for DeepSeek V4 Pro and Qwen3.6-Plus and shift no-anchor priors more strongly for most panels; retrieval drops for DeepSeek V4 Pro and MiniMax-M3.The cue comparisons support within-panel contrasts, not a model ranking, because pilot and all-family runs differ in catalog coverage and execution window.Source: D Same-Call and Semantic-Interference… · p. 14, Table S8 · p. 15

E Repeatability and Position Pilot

  • All four 128K interpretation/retrieval repeatability cells have identical labels across three passes; the position pilot shows no monotone late-to-early ordering but is too small to estimate a general positional curve.At 8K, override ranges from 0.70 to 0.90 across positions, and at 64K it ranges from 0.60 to 0.70; the pilot cells contain 20 items.Source: E Repeatability and Position Pilot · p. 15, Table S10 · p. 16

F Prompt-Fit Stress Beyond 128K

  • All four prompt-fit rows retain perfect explicit retrieval, while interpretation ranges from 0.860 to 0.940; the DeepSeek V4 Flash 1M row is not worse than its 512K row.These endpoint-specific stress cells are descriptive extensions, not a monotone degradation curve.Source: F Prompt-Fit Stress Beyond 128K · p. 15, Table S11 · p. 16

G Dataset Validation and Catalog Expansion

The section reports validation and separately scoped English and Chinese catalog results, then clarifies what the controls support and where the benchmark’s conclusions are bounded.

G Dataset Validation and Catalog Expansion

  • The primary English catalog was fixed before evaluation, with two annotators matching all 50 intended labels and automated checks for prompt structure and leakage.The annotators were blind to model outputs; the static audit found no verbatim answer-option or target-MWE leakage from neutral fillers.Source: G Dataset Validation and Catalog… · p. 16, Table S12 · p. 16
  • The 60-case English expansion broadens lexical and constructional coverage but is not difficulty-matched, is evaluated separately at 0/8K/64K, and is excluded from headline rankings.Its table reports 540 records per model, with zero request errors and zero unparsed responses; primary OR is shown only for context and catalogs are not pooled.Source: G Dataset Validation and Catalog… · p. 16, Table S13 · p. 16
  • Several Chinese panels show lower override than retrieval, while three ceiling panels remain at 1.000; the smaller, independently sourced subset is descriptive, not difficulty-matched to English.It contains 20 cases across 10 families and is reported only in aggregate.Source: G Dataset Validation and Catalog… · p. 16, Table S14 · p. 17

H Prompt and Release Boundary

  • The controlled-release Chinese inventory is not published row by row, so its 20 cases across 10 families are reported only in aggregate.Native-Chinese annotators matched all intended labels with no flags; transformation and validation code are included in the package.Source: H Prompt and Release Boundary · p. 17

I Interpretive Boundary of the Controls

  • Hard-cue results indicate sensitivity to semantic interference, but retrieval decreases in some panels, precluding a pure integration-only interpretation.Repeated runs constrain deployment variation at selected endpoints; position-pilot and prompt-fit rows remain descriptive because their grids are smaller or configuration-specific.Source: I Interpretive Boundary of the Controls · p. 17
  • The controls support a behavioral access–use dissociation but do not identify a neural mechanism, prove every error contains a recoverable hidden anchor, or establish a universal context-length ranking.The same-call control rules out invocation separation for its three tested deployments but does not reveal the latent state producing either answer.Source: I Interpretive Boundary of the Controls · p. 17

J Residual Limitations

  • The idiom-heavy forced-choice benchmark does not estimate average language understanding, and deployment drift, unmatched language difficulty, and non-held-out expansion bound its claims.The stated scope is behavioral access and use under the reported deployments.Source: J Residual Limitations · p. 17
Loading 2609.27590v1…