Revisiting Long-context Modeling from Context Denoising Perspective

1 INTRODUCTION

• Existing approaches mainly use high-quality synthetic long-context data for post-training, but can be inefficient or ineffective under limited resources.Long-context models are also described as enabling applications such as LLM agents and project code analysis.Source: §1 · p. 1
• LongCE improves by nearly 13 points per 1B tokens, while Prolong-64K-Base improves by only 0.3 points per 1B tokens.The comparison is presented as evidence of differences in training efficiency; the figure also compares real-world task performance and uses bubble size for relative training data volume.Source: Figure 1 · p. 1, §1 · p. 2
• In the retrieval-then-generation process, irrelevant tokens can overwhelm critical tokens, while uniform token-wise cross-entropy supervision cannot distinguish them.The paper identifies detecting critical tokens, diminishing context noise, and strengthening the connection between critical tokens and predictions as key to better long-context modeling.Source: §1 · p. 2
• IG scores detect critical tokens, and subtracting gradient values associated with irrelevant tokens from embeddings reduces noise at the model input.The authors report that IG improves critical-token detection accuracy over the traditional attention-based method.Source: §1 · p. 2
• Across four task types and two long-context training settings, CDT consistently surpasses other methods; Llama3.1-8B-Instruct scores 50.92 versus GPT4o's 51.00 on LongBench-E.The reported average gain is 2 points on 12 LongBench-E tasks and 13 long synthetic RULER tasks.Source: §1 · p. 2

2.1 RETRIEVAL-THEN-GENERATION MECHANISM OF LONG-CONTEXT MODELS

• Long-context models are described as retrieving salient information and using it for prediction, but excessive irrelevant tokens can overwhelm critical information.Prior work also reports positional bias in locating key information, including the lost-in-the-middle phenomenon.Source: §2.1 · p. 2
• Prior responses to retrieval and noise challenges include architecture changes, information-extraction mechanisms, and training-objective optimization.The paper positions its work as revisiting critical-information location through context denoising.Source: §2.1 · p. 2

2.2 LONG-CONTEXT POST-TRAINING

• Long-context post-training includes context-window scaling and alignment; scaling approaches include positional extrapolation and architecture modification.Alignment work also targets capturing critical information and challenges such as hallucination.Source: §2.2 · pp. 2–3
• Prior work had not simultaneously considered training efficiency and effectiveness under both settings; only a few preliminary studies explored token re-weighting or critical-token pruning.These preliminary approaches sought a trade-off between long-context processing efficiency and effectiveness.Source: §2.2 · p. 3

3 PRELIMINARY STUDY

• The study uses a synthetic long-form reasoning task as a controlled proxy because real-world testing data with labeled critical-token positions is lacking.Contexts contain supporting facts, misleading interference facts, low-frequency words, and irrelevant documents; sample lengths range from 0K to 64K.Source: §3 · p. 3

3.2 EFFECT OF MANUAL CONTEXT NOISE RESTRAINT

• Manual denoising raises attention scores on critical tokens nearly ×10 times while slightly reducing scores on irrelevant tokens.The method subtracts gradients for low-IG tokens from their input embeddings.Source: Figure 4 · p. 4, §3.2 · p. 5

3.1 CRITICAL TOKENS DETECTION

• IG estimates information flow between context tokens and predicted tokens, while FR measures attention allocated to context-token types.A higher aggregate IG score indicates a larger contribution from a token segment to the prediction.Source: §3.1 · p. 4
• IG detects significantly less irrelevant-context noise than FR, and critical tokens receive higher IG scores than irrelevant tokens regardless of response correctness.Figure 3 compares attention-based distributions with information flow on the critical-token location task.Source: §3.1 · pp. 4–5, Figure 3 · p. 4

4.1 CRITICAL TOKEN DETECTION

• Full IG computation is memory-intensive for long sequences, so CDT approximates it with token embedding gradients, which are proportional to the IG score.The passage reports a maximum computable sequence length of 12K even with 8×92GB GPUs for the full IG score.Source: §4.1 · pp. 5–6
• With model parameters frozen, CDT backpropagates cross-entropy through token embeddings and compares each L2-normalized gradient with the sequence average to identify critical tokens.The identifier marks a token as irrelevant when I(x_i) = 1.Source: §4.1 · p. 6

4.2 EMPHASIZING TRAINING

• CDT subtracts gradients from irrelevant-token embeddings while leaving critical tokens unchanged, then unfreezes the model and trains on the denoised embeddings with cross-entropy.The learning rate and β control the update and denoising level, respectively.Source: §4.2 · p. 6
• Denoising runs online during training, adding computational overhead; the authors describe the process as an expectation-maximization manner of bootstrapping long-context capabilities.The model identifies critical tokens, diminishes noise, and thereby ultimately enhances information flow.Source: §4.2 · p. 6

5.1 EXPERIMENTAL SETUPS

• Evaluation covers LongBench-E, LongPPL, BABILong, and RULER, using short-context models scaled to 64K and long-context models requiring alignment.Training uses PG-19 for context-window scaling and LCM-Base post-training, and LongMiT plus LongAlpaca for LCM-Instruct alignment.Source: §5.1 · pp. 6–7

5.2 RESULTS

• Nearly 4.7 points on average: CDT outperforms LongCE on LongBench-E with the same training data, while LCM-Instruct CDT improves over its backbone by more than 2 points on average.For LCM-Base, CDT avoids significant drops across subtasks; the reported comparison also notes a nearly 4-point Few-shot drop for LongCE versus the backbone.Source: Table 1 · p. 7, §5.2 · p. 7
• CDT achieves the best RULER performance from 32K to 128K context lengths and the lowest LongPPL on GovReport.Table 2 reports average RULER scores across 13 sampled subtasks; Llama3-8B-Base is the LongPPL evaluator.Source: Table 2 · p. 8, §5.2 · p. 8
• CDT has the highest overall BABILong score in each group and does not compromise short-context performance at 4K and 8K.BABILong evaluates reasoning from short context through long context.Source: Table 2 · p. 8, §5.2 · p. 8
• CDT achieves the best critical-token detection performance among methods on the synthetic task.Figure 7 compares critical-token detection capability across methods.Source: Figure 7 · p. 8

6.1 COMPARISON OF CRITICAL TOKEN DETECTION

• CDT identifies the largest number of critical tokens while suppressing irrelevant-token detections; attention-based detection finds supporting and interference tokens but also many irrelevant ones, while LongPPL misses supporting tokens.Source: §6.1 · p. 9

6.2 IMPACT OF CONTEXT DENOISING STRENGTH

• Attention to critical tokens rises after context denoising and increases further during Emphasizing Training; gains grow with learning rate before plateauing around 8e-5.The main experiments use lr = 1e-5 and β = 5, giving lr × β = 5e-5.Source: §6.2 · p. 9

6.3 TRAINING BUDGETS AND EM PROCESS

• With approximately 0.5 additional hours on 8×A100 GPUs per 50 steps, CDT consistently improves performance over 250 steps, while same-step DPO yields marginal gains and SFT declines.Figure 9 shows performance improvement alongside training duration for each 50-step interval.Source: §6.3 · p. 9, Figure 9 · p. 9
• CDT’s performance improvement is attributed largely to its EM process, which has a convergence boundary after approximately 250 steps.Figure 10 illustrates progressively improving information flow and attention distribution during training.Source: Figure 10 · p. 9, §6.3 · p. 10

• Long-context models may have retrieval context overwhelmed by excessive irrelevant tokens, motivating analysis of contextual noise.Source: §7 · p. 10
• The paper introduces the IG score to identify critical tokens and Context Denoising Training to increase attention to them and strengthen their association with predictions.Source: §7 · p. 10
• Experiments across four task types and different models demonstrate the method’s superiority; an open-source 8B model achieves performance comparable to GPT-4o on real-world long-context tasks.Source: §7 · p. 10

• The authors state that the work adheres to ethical research practices.Source: ETHICS STATEMENT · p. 10
• All data and LLMs used are publicly available, including in API format, and properly cited.Source: ETHICS STATEMENT · p. 10
• The authors report that no human subjects were involved.Source: ETHICS STATEMENT · p. 10

A ILLUSTRATION OF TRAINING EFFICIENCY OF CURRENT METHODS

• LongCE achieves a 13-point gain per 1B tokens versus ProLong’s 0.3-point gain per 1B tokens in the controlled comparison.The comparison uses Llama3-8B-Base on 12 LongBench-E tasks, with matched data, optimizer, batch size, learning rate, and 8 × A100 GPUs.Source: A ILLUSTRATION OF TRAINING EFFICIENCY… · p. 16, Table 3 · p. 16

B.1 PRELIMINARY TASK CONSTRUCTION

• The original BABILong qa4 samples can be answered from one fact, so the authors build their 4-hop task from qa3 data to require sequential search across four supporting facts.Source: §B.1 · p. 16
• For the 4-hop evaluation data, four supporting facts are inserted into noise context with distractor facts while preserving their relative order; 3-hop data use the original qa3 format with similar processing.The construction also places at least one distractor after the final supporting fact and uses the same noise context for samples of a given length.Source: §B.1 · pp. 16–17
• A BABILong qa4 sample with 0k context illustrates why that source task alone does not require four-hop reasoning.Source: §B.1 · p. 16, Figure 11 · p. 17

B.2 DESIGN OF IG SCORE

• Averaging IG scores across all attention heads is adopted for simplicity and ease of deployment, since using the top 30 heads gives results highly consistent with averaging all heads.Source: §B.2 · p. 17, Table 5 · p. 17

C DERIVATION OF RELATION BETWEEN INFORMATION FLOW AND EMBEDDING GRADIENTS

• Information flow in attention combines the attention distribution with its corresponding gradient; increasing attention can increase loss sensitivity and embedding derivatives.The derivation also notes that long-context attention scores are unlikely to approach 0.5.Source: C DERIVATION OF RELATION BETWEEN… · pp. 17, 19
• The derivation uses one embedding layer and attention module, omits activation layers for simplicity, and states that the conclusions extend to stacked blocks.The input embedding matrix has sequence-length and model-dimension axes, denoted n and d.Source: C DERIVATION OF RELATION BETWEEN… · pp. 18–19

D.1 TRAINING DETAILS

• Experiments use OpenRLHF, Ring-flash-attention, and DeepSpeed; LongCE uses an 8192-token sliding context window and its recommended hyperparameters.Language-model post-training and long-context SFT are applied to Llama3.1-8B-base and Llama3.1-8B-Instruct, respectively.Source: §D.1 · pp. 20–21
• Context-window scaling trains Llama-3-8B-base directly after increasing the RoPE base from 500,000 to 20,000,000 to scale context from 8K to 64K.This is an 8× context-window increase.Source: §D.1 · p. 20
• Long-context SFT and CDT use 7,000 LongMiT and 1,000 LongAlpaca samples, with LongMiT instances restricted to 16K–128K tokens and balanced by supporting-passage count.LongAlpaca adds single-evidence QA examples around 16K tokens.Source: §D.1 · p. 21

D.2 EVALUATION DETAILS

• HELMET evaluates seven long-context subtasks, with test contexts from 0 to 128K tokens; inputs beyond 128K are truncated to the model limit.The paper reports that CDT achieves the best performance on HELMET.Source: §D.2 · p. 21
• LongBench-E covers real-world long-context tasks across wider context lengths than LongBench, using 8K truncation for Llama3-8B-base and 32K for other models.Source: §D.2 · p. 21
• Language-model evaluation calculates LongPPL and PPL on GovReport.Source: §D.2 · p. 22
• RULER tests six synthetic long-context categories at 32K and 64K, while BABILong evaluation selects multiple-support-fact tasks and QA1.Source: §D.2 · p. 22

D.3 BASELINE ILLUSTRATION

• Baseline comparisons include YaRN, CE, LongCE, SFT, and LOGO, alongside ProLong-512K-Instruct, NExtLong-512K-Instruct, and LLaMA-3.1-8B-SEALONG.The listed methods include positional scaling, language-modeling and instruction-tuning objectives, and DPO-based approaches.Source: §D.3 · p. 22

E.4 EXPERIMENT STATISTICAL SIGNIFICANCE

• Paired-samples t-tests on LongBench-E show CDT significantly outperforming the baseline at the 5% significance level.Source: Table 14 · p. 23, §E.4 · p. 23

E.1 ANALYSIS OF RESULTS ON REAL-WORLD LONG-CONTEXT TASKS

• CDT substantially improves Code-task performance for LLaMA-3.1-8B-Instruct and LLaMA-3-8B-Base, with code completion gains linked to local-context focus and training examples.The discussion notes 263 code-completion examples from LongMIT and smaller relative gains for LLaMA-3.1-8B-Base, which already performs strongly on code.Source: §E.1 · pp. 22–23

E.2 GENERALIZING CDT TO MORE MODELS

• CDT significantly improves performance on real-world long-context, long synthetic, and long-form reasoning tasks across Qwen2.5-7B-Instruct and Mistral-V0.3-Instruct; Mistral scores more than 30 points on the long-form reasoning task.Source: §E.2 · p. 23

E.3 EVALUATION RESULTS ON LANGUAGE MODELING TASKS

• The language-modeling evaluation reports PPL scores in addition to LongPPL.Source: Table 13 · p. 22, §E.3 · p. 23

F ANALYSIS OF ATTENTION MAP BEFORE AND AFTER CDT

• After CDT, attention maps show increased attention to relevant context while retaining attention to the question and the model’s original attention characteristics.The visualization aggregates attention over 46 chunks of a 12,000-token input at layer 24.Source: F ANALYSIS OF ATTENTION MAP BEFORE AND… · pp. 23–24, Figure 14 · p. 24

G LIMITATION AND FUTURE WORK

• CDT adds context-noise detection costs that theoretically grow with model size, and its gains on complex reasoning are less pronounced and not yet understood.Source: G LIMITATION AND FUTURE WORK · p. 24

H USE OF LLMS

• The authors used Qwen models to refine clarity and fluency in early Abstract and Introduction drafts, then reviewed and edited the final text.Source: H USE OF LLMS · pp. 24–25

I ERROR ANALYSIS

• The error-analysis section uses colored text in generation examples to mark correct and incorrect prediction segments.The sample table compares Llama3.1-8B-Instruct, its CDT version, and GPT4o.Source: I ERROR ANALYSIS · p. 25, Table 15 · p. 26