PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding the Effect of Noise in LLM Training Data with Algorithmic Chains of Thought

Alex Havrilla, Maia Iyer

arXiv:2402.04004v2cs.LG

TL;DR

The paper studies how the type and intensity of noise in CoT training data affect task performance, using a controlled algorithmic setting. It introduces TInt to generate customizable noisy traces and finds that fine-tuned models are highly robust to static noise but more vulnerable to dynamic noise, while prompted models are more noise-sensitive. These results support prioritizing removal of globally destructive dynamic errors over low or moderate static noise.

  • Problem

    The paper asks how CoT noise quantitatively affects task performance and whether its effects differ across training phases such as fine-tuning and prompting.

    Source: §1 · p. 1

  • Method

    The paper uses TInt to generate customizable algorithmic execution traces and injects static or dynamic noise with controlled contamination levels and intensities.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Fine-tuned models are remarkably robust to static noise but substantially more sensitive to dynamic noise, while prompted models show higher overall noise sensitivity.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Filtering should prioritize samples containing large amounts of dynamic global noise, whereas low or moderate static noise may be less necessary to remove.

    Source: §5 · p. 8

    The study evaluates fine-tuning and prompting but does not examine pretraining or how pretraining-data distributions affect downstream noise sensitivity.

    Source: §5 · p. 8

Abstract

from arXiv · show

During both pretraining and fine-tuning, Large Language Models (\textbf{LLMs}) are trained on trillions of tokens of text of widely varying quality. Both phases of training typically involve heuristically filtering out ``low-quality'' or \textit{noisy} training samples, yet little is known quantitatively about how the type or intensity of noise affects downstream performance. In this work, we study how noise in chain of thought (\textbf{CoT}) impacts task performance in the highly-controlled setting of algorithmically solvable tasks. First, we develop the Traced Integer (\textbf{TInt}) framework to generate highly customizable noised execution traces for any arithmetic function on lists of integers. We then define two types of noise: \textit{static} noise, a local form of noise which is applied after the CoT trace is computed, and \textit{dynamic} noise, a global form of noise which propagates errors in the trace as it is computed. We then evaluate the test performance of pretrained models both prompted and fine-tuned on noised datasets with varying levels of dataset contamination and intensity. We find fine-tuned models are extremely robust to high levels of static noise but struggle significantly more with lower levels of dynamic noise. In contrast, few-shot prompted models appear more sensitive to even static noise. We conclude with a discussion of how our findings impact noise filtering best-practices, in particular emphasizing the importance of removing samples containing destructive dynamic noise with global errors.

1. Introduction

The paper asks how CoT noise affects task performance across training phases and develops a controlled framework to study this question. It finds strong robustness to static noise but substantially greater vulnerability to dynamic noise.

  • The study quantifies how CoT noise affects task performance and how its effects differ across pretraining, fine-tuning, and prompting.Source: §1 · p. 1
  • TInt generates customizable execution traces for arithmetic functions on lists of integers while controlling which trace components are visible.Source: §1 · p. 2
  • Static noise is applied after computation and preserves the high-level computation, whereas dynamic noise accumulates during computation and changes future state values.Source: §1 · p. 2
  • Fine-tuned models tolerate high contamination from static noise, while dynamic noise is more destructive because downstream tokens become unreliable.Source: §1 · p. 2
  • The findings motivate filtering samples with destructive global errors while treating low or moderate static noise as less necessary to remove.Source: §1 · p. 2

2. Related Work

Prior work establishes CoT as a tool for improving reasoning and studies robustness to noisy NLP data, but offers little quantitative evidence about noise in structured CoT training data.

  • CoT prompting improves language-model problem solving, and related work studies why intermediate reasoning helps and how to distill it into smaller models.Source: §2 · p. 2
  • Existing noisy-data research commonly examines grammatical errors, jargon, mistranslations, or task labels rather than structured CoT noise.Source: §2 · p. 2
  • This paper initiates a quantitative study of how noise in algorithmic CoT training data affects task learning performance.Source: §2 · p. 2

3. Methods: Generating Noisy Algorithmic Chain of Thought

The methods create controllable algorithmic CoT traces and inject localized or propagating errors at specified stages. TInt supports granular trace visibility, while separate noise parameters control contamination and intensity.

3. Methods: Generating Noisy Algorithmic Chain of Thought

  • Algorithmic CoT distributes computation across easier-to-predict tokens, potentially improving model performance, length generalization, and task composability.Source: §3 · pp. 2–3

3.1. The TInt Framework

  • TInt represents non-negative integers while recording programmable traces of arithmetic and list-based computations.Source: §3.1 · p. 3
  • TInt examples are generated by sampling inputs, executing a function while retaining selected trace pieces, and cleaning the accumulated trace into formatted text.Source: §3.1 · p. 3

Step 1: Choosing a good integer sampling distribution

3.2. Adding noise to algorithmic CoT

  • Dynamic noise is injected during trace accumulation and globally affects subsequent computation, whereas static noise is added during post-processing and remains local.Source: §3.2 · p. 4

4. Experiments

The experiments compare algorithmic CoT with direct prediction under static and dynamic noise, using fine-tuning and prompting across arithmetic and median tasks. Fine-tuned models tolerate substantial static noise, whereas dynamic noise and few-shot prompting produce sharper performance degradation.

4.1. Fine-tuning on Noisy Chain of Thought

  • Algorithmic CoT significantly improves fine-tuning performance across arithmetic and median tasks, especially for multiplication and median finding.The comparison uses direct training as a baseline and reports results in Figure 3.Source: §4.1 · p. 4, Figure 3 · p. 4
  • Fine-tuned addition models remain perfect with up to 70% of digits corrupted, failing only when 90% of digits are corrupted in every sample.Direct training is much more sensitive, with performance decreasing toward zero as dataset contamination increases.Source: §4.1 · p. 5, Figure 4 · p. 5
  • Median finding is more sensitive to character noise because its traces require harder primitives such as division by two, especially for even-length lists.At nc = 1 and nd = 1, accuracy reaches 0, while higher-noise cases often retain the ability to copy the middle number.Source: §4.1 · p. 5, Figure 4 · p. 5
  • Line noise is generally tolerated, but deletion of most—not all—intermediate lines is especially destructive, producing the lowest addition accuracy of 0.6 at nd = 1 and nl = 0.75.When all lines are deleted, accuracy recovers to 0.91 because the model effectively predicts the answer directly.Source: §4.1 · p. 5, Figure 5 · p. 6
  • Dynamic noise is more destructive than static noise: intensities above ndl = 0.5 destroy performance on all tasks, while median finding fails above ndl = 0.3.Propagated errors and longer corrupted traces make it harder to select reliable intermediate computations.Source: §4.1 · pp. 6–7, Figure 7 · p. 7
  • The authors attribute static-noise robustness to CoT’s sample efficiency and the model’s ability to aggregate multiple prior computations, whereas dynamic noise corrupts later steps globally.Algorithmic CoT may therefore be especially dependent on preserving samples without global propagated errors.Source: §4.1 · pp. 6–7

4.2. Prompting with Noisy Chain of Thought

  • In prompting, GPT-4 is more sensitive to both noise intensity and dataset contamination than fine-tuned models, with dynamic noise destroying performance even at small contamination levels.GPT-4 follows addition better than median or multiplication, while GPT-3.5 cannot follow the algorithmic CoT framework.Source: §4.2 · pp. 7–8, Table 1 · p. 7

5. Conclusion and Broader Impact

The study finds that fine-tuned models tolerate substantial static CoT noise but are much more vulnerable to dynamic global errors, while prompted models are more noise-sensitive. These findings motivate filtering destructive dynamic noise while retaining some lower-level static noise, although the study does not examine pretraining or scale.

Conclusion and Broader Impact

  • Fine-tuned models are remarkably robust to static local noise, whereas dynamic noise is much more destructive and prompted models show higher overall sensitivity.The authors attribute static-noise robustness to sample efficiency and attention to multiple prior computations for self-correction.Source: §5 · p. 8
  • Filtering documents with large amounts of dynamic global noise is critical during both pretraining and fine-tuning.The authors consider low or moderate static noise less necessary to filter and potentially helpful when its intensity is not too high.Source: §5 · p. 8
  • CoT data generated by larger models may contain substantial static noise but comparatively less dynamic noise, helping explain successful distillation to smaller models.This interpretation is presented as a likely property of such generated data, not as a directly measured result here.Source: §5 · p. 8
  • The study focuses on fine-tuning and prompting, leaving scale effects and pretraining-data distribution as future work.The authors expect larger models may be more robust to all noise types, but this expectation is not tested.Source: §5 · p. 8

A. Baselines for Noise-Free Algorithmic Chain of Thought

The noise-free baselines show that algorithmic CoT improves compositional and length-generalization behavior across arithmetic and median tasks, while its benefits depend on training-length coverage and trace design.

Experimental setup

  • The TInt framework traces arbitrary integer functions while controlling which computation steps remain visible in training CoT.The baseline tasks include addition, subtraction, multiplication, division, and median finding over integer lists.Source: A. Baselines for Noise-Free… · p. 11, Figure 9 · p. 12

A case study in addition

  • CoT improves addition performance even without independent digit tokenization, likely because the trace isolates digits by decimal place.Space-separated tokenization also improves performance for the evaluated decoder-only models.Source: A. Baselines for Noise-Free… · p. 11, Table 2 · p. 11

Sample efficiency

  • Algorithmic CoT generalizes perfectly across all lengths in the training distribution using one epoch and only 1000 samples.The authors interpret this as evidence that CoT encourages learning the underlying algorithm rather than an approximate length-specific procedure.Source: A. Baselines for Noise-Free… · p. 11

Length generalization

  • A 410m-parameter direct model achieves near-perfect accuracy on exactly 50-digit addition but only 0.36 when lengths are sampled uniformly.This contrast indicates strong fixed-length overfitting in direct training.Source: A. Baselines for Noise-Free… · p. 11
  • Algorithmic CoT significantly improves length generalization over direct training, but fixed-length training yields poor generalization even with CoT.Beyond the training range, remaining failures often arise from copying inputs into the algorithm format rather than executing the algorithm incorrectly.Source: A. Baselines for Noise-Free… · p. 11

Multi-task training on arithmetic

  • Across multi-task arithmetic, algorithmic CoT substantially improves multiplication and division, while direct models often fail on inputs exceeding two digits.Multiplication remains imperfect because addition is treated as atomic and must be learned without CoT.Source: Table 4 · p. 11, A. Baselines for Noise-Free… · p. 12

Median finding

  • Algorithmic CoT enables median finding by composing sorting and addition, whereas the direct model struggles despite learning those routines independently.The median procedure first sorts the list and then extracts the median.Source: A. Baselines for Noise-Free… · p. 12

Sample efficiency

  • CoT training is reported to be highly sample-efficient, with accuracy plotted against sample count in the accompanying sample-efficiency figure.The supplied passage states that the experiment confirms strong sample efficiency but provides no numerical curve values.Source: A. Baselines for Noise-Free… · p. 12

B. Additional Results for Noisy Fine-tuning

Fine-tuned models remain robust to static line-deletion noise until the dataset is fully contaminated, but intermediate deletion rates can cause severe performance collapse.

Static line noise

  • Line deletion is especially destructive for median finding, whose sorting procedure relies on multiple control flows rather than addition’s single control flow.At n_l = 0.5, performance is worse than training without algorithmic CoT; at extreme deletion, models can recover direct-task performance.Source: B. Additional Results for Noisy… · p. 12
  • The most destructive line-noise regime is around 50% deletion, where diverse partial traces make reconstructing the full algorithm difficult.At very high deletion levels, the model can ignore irrelevant lines and eventually approach its direct baseline.Source: B. Additional Results for Noisy… · p. 12

C. Additional Results for Noisy Prompting

Few-shot prompting achieves strong clean addition accuracy but is more sensitive to noisy CoT than fine-tuning, with complexity and intermediate line deletion shaping performance.

Experimental setup

Prompting baselines

  • Six-shot prompting reaches 0.92 accuracy on 10-digit addition, compared with 0.86 for three-shot prompting.The study therefore uses six-shot prompting for subsequent experiments.Source: C. Additional Results for Noisy… · p. 13
  • Clean six-shot accuracy is 0.92 for addition, 0.37 for median finding, and 0.27 for multiplication.The lower median and multiplication scores are interpreted as reflecting greater operational complexity than addition.Source: C. Additional Results for Noisy… · p. 13

Noisy prompting

  • For multiplication, moderate line noise produces the worst performance among the three tested line-noise levels.The passage links this observation to the lower baseline performance of multiplication relative to addition.Source: C. Additional Results for Noisy… · p. 13

D. CoT Design

The TInt framework evaluates algorithmic CoT design choices, favoring compact variable-state traces that preserve local information while controlling context length. The section also describes decomposition and visibility choices needed to scale traces to complex arithmetic operations.

CoT Design

  • Variable-state assignment without code or natural-language lines was the most effective CoT design, while removing those lines also saved context.Experiments found code lines irrelevant at best and detrimental at worst, with similar effects for natural-language text.Source: D. CoT Design · p. 13
  • Because full sub-routine traces can exceed training context, the framework omits routines whose visibility is explicitly disabled, including addition during multiplication.Only very low-level operations are invisible by default, such as equality checks and shifts.Source: D. CoT Design · p. 14
  • Multiplication traces expose evolving state through assignments such as res, carry, term, and digit-processing variables.The listed trace examples show intermediate assignments progressing toward the final result.Source: D. CoT Design · p. 14
  • Complex operations can be shortened by replacing sub-traces with calls to simpler routines, such as invoking addition during multiplication.This decomposition reduces trace sizes and supports models with smaller context lengths.Source: D. CoT Design · p. 14
  • The best-performing trace prints only variable-value deltas, omitting unchanged variables to improve context efficiency.For example, if only x changes from 5 to 7 while y remains 6, the trace prints x = 7 and omits y.Source: D. CoT Design · p. 14
  • The preferred design maintains locality by stripping processed digits from input variables instead of repeatedly attending to the original arguments.This locality is illustrated as a key property of the most context-efficient design.Source: D. CoT Design · p. 14, Table 5 · p. 15
Loading 2402.04004v2…