Source-linked AI summary
Learning When Not to Listen: Selective Anti-Interference Pretraining for Language Models
Jinchang Zhu, Haowei He, Yi Ding, Rong Fu, Nie Xiaojian, Shuangyong Song, Zhongjiang He, Menglin Yang
TL;DR
Language models can let irrelevant distant prefixes alter predictions already supported by local context, while the training objective for resisting that interference remains underspecified. SPAR uses a short-context gate and gated distribution matching to stabilize locally supported predictions, and improves benchmark performance across continued training and pretraining evaluations.
Problem
How training should teach models to use relevant evidence while resisting irrelevant preceding text remains underspecified, particularly in long, packed, or distractor-heavy contexts.
Source: Introduction · p. 1Method
SPAR uses a short-context sufficiency gate to select locally predictable suffix tokens and applies gated KL matching across original and far-prefix-corrupted inputs.
Source: Introduction · p. 1Results
SPAR improves RULER across five pretrained base models and improves both RULER and NoLiMa in pretraining experiments.
Source: Introduction · p. 2Takeaways & Limitations
The findings support selective anti-interference as an objective-level signal for robust context use without uniformly suppressing long-range dependence.
Source: Conclusion · p. 7, Discussion · p. 7
Abstract
from arXiv · showhide
Language models can over-condition on irrelevant preceding text: predictions already supported by local context may still change when distant, unrelated prefix tokens are perturbed. This interference is especially consequential in long, packed, or distractor-heavy contexts, where useful evidence and irrelevant spans coexist. We propose Selective Prefix Anti-Interference Regularization (SPAR), a pretraining objective for selective anti-interference. SPAR runs the original sequence and a corrupt-prefix input in which only the far prefix is changed, then uses a short-context sufficiency gate and a gated KL objective to stabilize locally supported suffix predictions. The gate operationalizes a model-based estimate of whether the far prefix supplies additional information about the target token. Mechanism analyses show that the gate identifies locally sufficient tokens and sharply reduces prefix sensitivity on gate-selected suffix tokens. In continued training on pretrained base models, SPAR improves RULER across Qwen2.5-0.5B, Qwen2.5-3B, Llama-3.2-1B, Llama-3.1-8B, and GPT2-XL under equal counted training compute; pretraining experiments further show gains on both RULER and NoLiMa. These results show that selective anti-interference is an effective objective-level signal for robust context use.
Introduction
SPAR addresses far-prefix interference by explicitly stabilizing predictions when the local suffix is sufficient, while preserving the usefulness of distant evidence when it matters. The introduction reports gains across continued-training and pretraining evaluations, alongside mechanism evidence that the stabilization is selective.
Introduction
- Standard causal language modeling trains on the full prefix without explicitly requiring locally predictable suffix tokens to resist changes to irrelevant far-prefix text.The paper frames this as selective conditional dependence: nuisance prefixes should not alter locally sufficient predictions, while decisive distant evidence should remain useful.Source: Introduction · pp. 1–2
- SPAR corrupts only the far prefix, preserves the local suffix, and uses a short-context sufficiency gate to apply clean-to-corrupt KL matching selectively alongside ordinary CLM.The gated invariance term targets positions where the recent suffix supports the target and the far prefix contributes little additional likelihood.Source: Introduction · p. 1
- Mechanism analyses find that the gate identifies locally sufficient targets and that prefix sensitivity decreases sharply on the selected suffix tokens.The introduction describes fixed-reference gate analysis and prompt-level probes as tests of whether robustness gains concentrate in the intended regime.Source: Introduction · pp. 1–2
- SPAR improves RULER across five pretrained base models and improves RULER and NoLiMa in pretraining across 0.3B–1B scales.The introduction also reports continued-training gains on HELMET across five pretrained base models.Source: Introduction · pp. 1–2
Related Work
Prior work has expanded the lengths models can process and sharpened evaluation of effective context use. SPAR instead makes selective stability under irrelevant far-prefix changes a training objective.
Making longer context computable
- Long-context work extends computable length through attention, positional methods, efficient deployment, and training recipes; SPAR adds an objective-level signal for stable local predictions.Examples include Transformer-XL, Longformer, FlashAttention, RoPE, ALiBi, Position Interpolation, LongLoRA, StreamingLLM, LongAlign, and LongCite.Source: Related Work · pp. 2–3
Long-context benchmarks reveal selective context-use challenges
- Long-context benchmarks test retrieval, reasoning, long dependencies, and distractor robustness, showing that accepting more tokens does not ensure reliable use of distant evidence.RULER provides 13 structured stress-test tasks, while NoLiMa removes literal lexical matching between questions and evidence.Source: Related Work · p. 3
Training objectives for useful context dependence
- LongPPL and LongCE use long–short contrast to identify and upweight key tokens, while SPAR targets locally sufficient tokens for stability under irrelevant far-prefix changes.Other training-signal approaches include long-context alignment data and batching, position-gap synthesis, and supervision distributions that affect output length and citation faithfulness.Source: Related Work · p. 3
Irrelevant context, distractibility, and far-prefix interference
- Distractors, position sensitivity, and benchmark disagreements frame context use as a selection problem; SPAR trains locally determined predictions to resist irrelevant far-prefix changes.It constructs clean and far-prefix-corrupted views and applies the stability objective only when a gate indicates local sufficiency.Source: Related Work · p. 3
Consistency and invariance training
- SPAR adapts consistency regularization to perturbations that change the far prefix but preserve the local suffix, matching predictions only where a short-context gate indicates sufficiency.Related consistency methods include Virtual Adversarial Training, Mean Teacher, UDA, and R-Drop.Source: Related Work · p. 3
Selective Prefix Anti-Interference · Regularization
SPAR targets suffix positions where local context already supports the target and the far prefix supplies little additional predictive information. It compares clean and corrupt-prefix predictions, applying gated consistency while leaving tokens that depend on long-range context to ordinary language-model training.
Far-Prefix Interference and Conditional Dependence
- SPAR targets positions with high local predictability and little additional predictive information from the far prefix, where nuisance prefix changes should leave predictions stable.The pointwise conditional-information estimate compares the target probability under full context with that under local context; SPAR focuses on its low-information region.Source: Far-Prefix Interference and… · p. 4
Clean and Corrupt-Prefix Views
- The paired views preserve the target suffix while corrupting the far prefix, turning interference at locally sufficient positions into a direct training signal.The corruption perturbs far-prefix chunk order; both views predict the same suffix tokens and retain the same immediate suffix evidence.Source: Clean and Corrupt-Prefix Views · p. 4
Short-Context Sufficiency Gate
- The sufficiency gate selects locally predictable tokens only when full context adds little evidence, leaving tokens with genuine long-range dependence to ordinary CLM.High short-context likelihood avoids regularizing uncertain positions; the full–short gap filters out positions where the far prefix contributes decisive evidence.Source: Short-Context Sufficiency Gate · p. 4
Gated Invariance Loss
- On gated positions, SPAR applies a directional consistency loss, using the stop-gradient clean distribution as the anchor and sending the consistency gradient through the corrupt-prefix branch.The ordinary clean-context CLM loss remains the source of token-likelihood learning; shared parameters let corrupt-branch gradients update the model used at clean-context inference.Source: Gated Invariance Loss · p. 4
Evaluating Selective Context Dependence
- Prefix-sensitivity analysis measures the change in gold-token log likelihood under clean versus corrupt-prefix inputs, both across all tokens and on gate-selected positions.Gated sensitivity is the closer mechanism check because it evaluates the positions selected by the same local-sufficiency criterion used in training.Source: Evaluating Selective Context Dependence · p. 5
- To avoid model-dependent subset shift in gate-region analysis, evaluation fixes token bins and masks with a frozen CE reference model and applies them to both CE and SPAR.This keeps the evaluated tokens, bins, and mask identical across the two models.Source: Evaluating Selective Context Dependence · p. 5
Experimental Setup
The experiments evaluate SPAR in continued training and pretraining, using long-context benchmarks and analyses of prefix sensitivity and gate selectivity.
Continued training
- Continued-training experiments compare SPAR with ordinary CLM (CE) at equal counted training compute across five pretrained base models.Qwen2.5-0.5B, Qwen2.5-3B, Llama-3.2-1B, Llama-3.1-8B, and GPT2-XL are evaluated directly after training.Source: Experimental Setup · p. 5
Pretraining
- Pretraining experiments evaluate 0.3B, 0.6B, and 1B models on RULER and NoLiMa to test SPAR before continued training.Source: Experimental Setup · p. 5
Benchmarks and metrics
- RULER is the primary continued-training benchmark, while HELMET provides a secondary long-context evaluation and NoLiMa tests retrieval beyond literal matching in pretraining.RULER spans 13 retrieval, multi-hop tracing, aggregation, and QA-style synthetic tasks; task macro averages are reported by context length, with macro deltas against the corresponding baseline.Source: Experimental Setup · p. 5
Mechanism analyses
- Mechanism analyses use 4K contexts with 3K-token far prefixes and 256 entity–value prompts to test selectivity under nuisance changes and far-dependent evidence switches.All candidates are scored by sequence log probability; the far-dependent condition switches the only answer-bearing evidence in the far prefix.Source: Experimental Setup · p. 5
- Gate-region analysis uses 256 held-out pretraining sequences and a frozen CE model to define a shared evaluation gate for both target models.It averages document replacement and chunk-shuffle corruption; a fixed mask keeps CE and SPAR on the same selected token subset.Source: Experimental Setup · p. 5
Evaluation windows
- Qwen and Llama evaluations at 24K and 32K are extrapolations beyond their 16K training windows, whereas GPT2-XL uses a 32K training window.The trained-window claim covers lengths up to each model’s training window; full reported deltas include all completed lengths when stated. Table 5 reports Qwen2.5-0.5B batch-fraction sensitivity at fixed λinv = 0.004, and Figure 2 plots per-prompt robustness gains against gains in following changed answer evidence.Source: Experimental Setup · p. 5, Table 5 · p. 5, Figure 2 · p. 5
Results
SPAR improves long-context benchmark results in pretraining and continued training, while token- and prompt-level analyses link the gains to selective robustness against irrelevant prefix changes. Its benefits depend on selecting locally sufficient positions and using a moderate-strength invariance signal.
Pretraining Improves RULER and NoLiMa
- SPAR improves both RULER and NoLiMa at every tested pretraining scale, and improves RULER in every pretrained model family and HELMET for all five base models.The pretraining results span 0.3B, 0.6B, and 1B scales; continued-training gains are described as small but consistent for Qwen2.5-3B, larger for Qwen2.5-0.5B and Llama-3.2-1B, and strongest for GPT2-XL at 32K.Source: Pretraining Improves RULER and NoLiMa · p. 6, Continued Training Improves RULER… · p. 6
Gate Selection Matters
- The SPAR gate yields the largest average RULER gain for every model, with gains of +2.15, +3.38, and +3.43 versus +0.98, +1.56, and +1.55 for random gating on three models.Ungated KL is weaker across those same models; the selection rule targets suffix positions where short-context evidence suffices for prediction.Source: Gate Selection Matters · p. 6, Table 6 · p. 6
Public Benchmark Transfer
- SPAR improves average zero-shot public-benchmark scores at all three pretraining scales, with gains concentrated on PIQA, HellaSwag, ARC, Open-BookQA, Lambada, and MMLU.The decoder-only Transformer base models were evaluated before instruction or QA fine-tuning.Source: Public Benchmark Transfer · p. 6
Evidence for Selective Context Dependence
- On the fixed CE-reference mask, prefix JS falls by 0.0049 in selected tokens versus 0.0005 in unselected tokens, while sensitivity falls by 0.0820 versus 0.0040.Selected-token clean-view NLL increases by +0.0186 while corrupt-view NLL decreases by -0.0634; the same tokens are used to evaluate CE and SPAR.Source: Evidence for Selective Context… · pp. 6–7, Table 7 · p. 7
- Prompt-level gains concentrate where SPAR is both less sensitive to irrelevant prefix changes and more responsive to changed answer-bearing facts, with evidence-switch fixes substantially outnumbering losses.This pattern aligns with the gate-region results: nuisance invariance improves while responsiveness to answer-changing evidence is retained.Source: Evidence for Selective Context… · p. 7
SPAR Strength Sensitivity
- At fixed λinv = 0.004, Qwen2.5-0.5B shows small gains at 10%–30% SPAR batches and a broad high-gain region at 50%–90%, with largest gains at 8K–16K.For Llama-3.2-1B at a fixed 10% SPAR batch fraction, gains rise as λinv increases from 0.008 to 0.016, remain high near 0.020, then decline at larger weights.Source: SPAR Strength Sensitivity · p. 7, Figure 4 · p. 7
Discussion
SPAR selectively stabilizes locally supported predictions under changes to irrelevant distant text, using the full–short log-probability gap to identify where invariance is desirable. Its gate targets locally sufficient positions rather than suppressing long-range dependence uniformly.
Selective anti-interference
- The full–short log-probability gap gives SPAR a conditional-dependence interpretation: low-gap targets are locally predictable, while high-gap positions indicate predictive evidence from the far prefix.Mechanism analyses connect this token-level criterion to local sufficiency and reduced corrupt-prefix sensitivity in the selected region.Source: Discussion · p. 7
Why the gate matters
- On the balanced answer-token set, the gate reaches 0.988 precision and a 0.008 far-dependent false-positive rate, while analyses show fewer target flips and reduced selected-region prefix sensitivity.These findings support lower irrelevant-prefix dependence where the local suffix already supports the prediction.Source: Discussion · p. 7
Relation to key-token objectives
- Unlike key-token objectives such as LongCE, which emphasize positions improved by long context, SPAR targets locally sufficient predictions where stability under far-prefix changes is desired.The discussion frames anti-interference training as complementary to long-context token upweighting.Source: Discussion · p. 7
Objective-level view
- SPAR adds to CLM a selective pressure to damp irrelevant far-prefix variation only where short-context evidence is sufficient, rather than suppressing long-range dependence uniformly.This local training signal discourages nuisance-text sensitivity even when useful evidence and nuisance text coexist in a sequence.Source: Discussion · p. 7
Conclusion
SPAR uses a short-context sufficiency gate to stabilize locally supported predictions against far-prefix corruption. The conclusion reports gains in continued training and from-scratch pretraining, framing selective anti-interference as an objective-level signal for robust context use.
Conclusion
- SPAR yields benchmark gains in continued training on pretrained base models and in from-scratch pretraining.The conclusion presents these results as evidence for selective anti-interference pretraining as an objective-level signal for robust context use.Source: Conclusion · p. 7
- A short-context gate identifies locally sufficient targets and stabilizes them under far-prefix corruption.Mechanism analyses show conservative gate selection and lower prefix sensitivity in the selected region.Source: Conclusion · p. 7
- SPAR adds a complementary signal for what models should ignore without changing architecture or requiring task-specific supervision.The conclusion contrasts learning what to ignore with learning what to retrieve.Source: Conclusion · p. 7