PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying Error Tolerance in Synthetic Data: An Atomic-level Operand vs. Operator Perturbation Study

Jiaxiang Liu, Chenhao Yuan, Shuwen Xu, Boxuan Xing, Xiusheng Huang, Yinhao Xu, Hao Liu, Wenhao Teng, Xiangwen Liao, Pengfei Cao, Jun Zhao, Kang Liu

arXiv:2608.29144v1cs.CL

TL;DR

Synthetic-data filtering lacks a quantitative criterion for distinguishing tolerable from harmful errors. ATOM decomposes responses into atomic operations and controls operand and operator perturbations. The experiments show strong robustness to operand changes but substantial degradation from operator changes, while ATOM-guided datasets outperform standard baselines across multiple backbones and tasks.

  • Problem

    Synthetic-data filtering lacks a principled quantitative criterion for determining which errors supervised fine-tuning can tolerate.

  • Method

    ATOM decomposes instruction-response pairs into atomic functions and generates data with controlled operand and operator perturbations.

  • Results

    2.0% relative decrease follows operand perturbations, whereas operator perturbations cause an 18.2% relative decrease; ATOM-guided datasets also outperform standard baselines across four backbones and 10 tasks.

  • Takeaways & Limitations

    Useful synthetic data should preserve valid, diverse operators rather than require strict operand precision.

  • Takeaways & Limitations

    Experiments exclude models with 32B parameters or larger and do not extend atomic decomposition to code generation or other domains.

Abstract

from arXiv · show

Synthetic data generation has become a cornerstone for advancing large language models. However, the lack of the quantitative analysis for error tolerance became a critical bottleneck. Consequently, current filtering strategies fluctuate between two extremes: they are either overly aggressive, risking the exclusion of potentially valuable samples, or overly permissive, failing to eliminate erroneous samples effectively. To bridge this gap, this paper introduces Atomic Tree Operation Modeling (ATOM), a framework that decomposes data into functional units ($f(x)\rightarrow y$). ATOM distinguishes benign Operand $x$ perturbations from fatal Operator $f$ perturbations. The former are needlessly discarded by aggressive filtering, while the latter slip through permissive filtering. Our experiments reveal a double dissociation: models are robust to operand perturbations but collapse under operator perturbations. By prioritizing operator over aggressive operand precision, our ATOM-synthesized data outperforms rigorous baselines (e.g., +3.1% gain over LIMA), suggesting that operator diversity matters more than operand precision. Our code is available at https://github.com/Lut-hub/ATOM.

1 Introduction

ATOM addresses the lack of a quantitative error-tolerance criterion in synthetic-data filtering by decomposing responses into atomic operations and distinguishing operand from operator perturbations. Experiments find strong operand robustness but substantial operator sensitivity, motivating operator-centered synthesis and broad operator coverage.

  • Teacher-model hallucinations introduce factual errors into synthetic training data, creating a need to quantify which errors supervised fine-tuning can tolerate.
  • ATOM decomposes instruction-response pairs into atomic operations f(x) → y, enabling controlled error types and perturbation ratios.The framework preserves or violates the logical operator to distinguish operand and operator perturbations.
  • 2.0% relative decrease follows operand perturbations, compared with an 18.2% relative decrease after operator perturbations.Internal representations after operand perturbation remain aligned with those learned from original Atomic-QA.
  • ATOM-guided Atomic-QA and Atomic-LIFD preserve broad operator coverage and consistently outperform standard baselines across four model backbones and 10 benchmark tasks.The datasets retain structural diversity often discarded by aggressive filtering.
  • Atomic-X extends the error-tolerance findings to existing instruction and mathematical datasets through controlled perturbations.
  • Models are robust to operand perturbations but sensitive to operator perturbations.Operand changes preserve the operator, whereas operator changes violate the input-output relation.

2 Related Work

Prior work emphasizes high-quality data and uses generation, filtering, heuristics, or contribution estimation to construct synthetic datasets. Research on noisy supervised fine-tuning examines robustness to perturbed instructions and erroneous reasoning, while challenging purely superficial accounts of alignment.

  • Prior studies show that supervised fine-tuning can achieve effective alignment from small, high-quality corpora rather than sheer data scale.
  • Synthetic-data pipelines commonly generate then filter samples, use heuristic selection, or estimate sample contributions.
  • Noisy supervised fine-tuning studies report effects from perturbed instructions and investigate robustness to erroneous reasoning chains.
  • Recent evidence challenges the superficial alignment hypothesis by linking supervised fine-tuning to reasoning improvements and pretraining scaling laws.

3 Methodology

ATOM represents data as atomic functions and organizes their operators into a hierarchy. It then generates verified atomic data from leaf operators and creates controlled operand- and operator-perturbed variants.

  • Atomic Function: ATOM defines an atomic function as a minimal logically indivisible triplet A = ⟨x, f, y⟩ with f(x) → y.
  • Atomic Function: The operator f specifies the logical rule, the operand x is the substitutable input, and the output y is the deterministic result.Examples include CapitalOf(France) → Paris and Contains(primary colors) → {red, blue, yellow}.
  • Perturbation Design: Operand perturbation jointly changes x and y while preserving f(x′) = y′, producing factually different but logically coherent samples.
  • Perturbation Design: Operator perturbation changes y while keeping x fixed, producing an output that violates the operator’s logic.
  • Atomic Tree Construction: ATOM organizes operators in a tree hierarchy whose leaf nodes represent concrete atomic operators and whose branches provide increasingly specific categories.The hierarchy uses depth expansion for granularity and breadth expansion for coverage.
  • Atomic Data Synthesis: Atomic-QA generation samples diverse operands for each leaf operator, computes outputs, deduplicates overlapping operands, and verifies logical validity.

4 Experimental Setup

The experiments test which component dominates training, whether ATOM data outperforms aggressively filtered baselines, and whether error tolerance generalizes to real synthesis data. They use seeded Atomic-QA construction, controlled perturbations, multiple benchmarks, and four model configurations.

  • The experiments address operator-versus-operand importance, ATOM performance against aggressively filtered baselines, and generalization to real synthesis data.
  • Figure 3 compares Atomic-QA with operand- and operator-perturbed variants, using dashed lines to mark each setting’s average performance.
  • Atomic-QA contains 1.37M instances and is accompanied by operand-perturbed and operator-perturbed variants.The datasets use FB15k, ZsRE, and wiki_recent as seed sources.
  • Atomic-Alpaca and Atomic-MetaMath inject operator errors at controlled rates into real-world instruction and mathematical data through Atomic-X.
  • The evaluation covers LLaMA3.1 8B, Qwen2.5 7B and 14B, and Qwen3 8B.

5 Operator–Operand Asymmetry

The study separates operand changes, which preserve operator logic, from operator changes, which disrupt it, revealing robust behavior under operand noise but sharp degradation under operator noise.

  • Operand perturbations replace factual inputs with random strings while preserving the operator, producing structurally coherent but factually meaningless data.
  • Models remain robust to operand corruption across tasks and model structures, with Atomic-QA performance closely tracking the unperturbed baseline.On ARC-Challenge, performance is 0.587 versus 0.610; on HotpotQA, it is 0.159 versus 0.133.
  • Operator perturbations disrupt structural logic, causing performance to fall to 50.4%.This corresponds to a 9.2% absolute decline and an approximately 18.2% relative drop.
  • Behavioral and representational analyses show that operand-perturbed models learn similarly to standard Atomic-QA models.The Qwen2.5-7B models reach a CKA similarity of 0.9497, while their accuracy curves rise in unison.
  • These findings motivate prioritizing operator coverage over strict operand filtering in synthetic-data construction.

6 Effectiveness of Atomic-QA

Atomic-QA is evaluated against established synthetic-data baselines across multiple backbones and benchmarks, with results associated with retained operator diversity rather than sheer data volume.

  • 6.1 Atomic-QA Outperforms Baselines: Atomic-QA is compared with standard baselines across 10 benchmarks and four model backbones.
  • 6.1 Atomic-QA Outperforms Baselines: Atomic-QA achieves a 3.1% absolute gain over LIMA on LLaMA3.1-8B, reaching 56.5% average performance.The gains are 1.7% on Qwen2.5-7B and 2.5% on Qwen3-8B.
  • 6.2 Exclusion of Potentially Valuable Samples: Atomic-LIFD achieves competitive results despite using minimal training tokens, indicating that retaining diverse operator structures can preserve performance.
  • 6.1 Atomic-QA Outperforms Baselines: Atomic-QA reaches 61.1% on Qwen2.5-7B versus 56.8% for WizardLM, a difference of over 4%.The strongest gains appear on reasoning-intensive tasks such as OpenBookQA and ARC-Challenge.
  • 6.2 Exclusion of Potentially Valuable Samples: Atomic-QA achieves the highest scores across all six reported corpus-diversity metrics and encompasses all baselines.The paper states that aggressive filtering contracts diversity, whereas tree-based operator enumeration preserves structural samples.

7 Generalization to Existing Datasets

Atomic-X decomposes existing datasets, perturbs their atomic operations at controlled rates, and reconstructs outputs to measure operator-noise tolerance beyond Atomic-QA.

  • 7 Generalization to Existing Datasets: Atomic-X uses an Encoder–Perturber–Decoder pipeline to decompose instruction-response pairs, alter atomic operations, and reconstruct perturbed outputs.
  • 7 Generalization to Existing Datasets: The encoder produces an ordered sequence of atomic operators by recursively parsing responses into logical branches.
  • 7 Generalization to Existing Datasets: Performance follows a two-stage nonlinear degradation pattern as operator noise increases.Accuracy declines gradually from 0% to 40% noise, then drops steeply beyond roughly 60% noise.
  • 7 Generalization to Existing Datasets: Oracle filtering fully recovers the noise-free baseline, but existing filtering methods cannot approach this upper bound.Their negative Cohen’s Kappa scores indicate difficulty distinguishing faulty from correct samples.

8 Conclusion

The study quantifies error tolerance in synthetic data beyond binary quality labels. It finds that models learn atomic logic even with randomized operands, motivating operator-focused synthesis.

  • Models learn functional atomic operators while specific operand values remain changeable variables.
  • Synthetic data should favor operators over operands for generalizability.

Limitation

The paper’s main scope boundary is that ATOM is evaluated only on models below 32B parameters and on instruction-following and mathematical reasoning tasks.

  • Experiments do not include models with 32B parameters or larger because of computational resource constraints.
  • ATOM has not yet been extended to code generation or other domains.
  • The framework is evaluated for general instruction following and mathematical reasoning tasks.

C.2 Details about Model Training and Evaluation

Experiments fine-tune four open-source LLMs across Atomic-QA variants and established synthetic-data baselines, evaluating multiple comprehension and reasoning capabilities. Atomic-QA remains strong across model backbones, while Atomic-LIFD matches the full dataset despite reduced scale.

  • Model training and evaluation: Four backbones—LLaMA3.1-8B, Qwen3-8B, Qwen2.5-7B, and Qwen2.5-14B—are fine-tuned with LoRA and evaluated using LM Evaluation Harness.All models use rank r = 16, weight decay 0.01, and 2 training epochs.
  • Datasets: Atomic-QA is compared with its variants, official Instruct versions, and six open-source datasets including LIMA, Alpaca, Magpie, WizardLM, and Self-Instruct.
  • Model training and evaluation: The evaluation suite covers text comprehension, information integration, knowledge-driven reasoning, commonsense reasoning, and mathematical reasoning.Benchmarks include BoolQ, LongBench-HotpotQA, RACE, SQuADv2, ARC, OpenBookQA, PIQA, SIQA, WinoGrande, GSM8K, Math500, and MathQA.
  • Results: Atomic-LIFD achieves performance comparable to full Atomic-QA, indicating that data quantity is not the decisive factor behind Atomic-QA’s superiority.
  • Results: 61.2% on Qwen2.5-7B and 62.5% on Qwen3-8B are the highest average performances across baselines on those backbones.

D.3 Measuring Operator–Operand and Dominance

Across models, operand perturbations preserve performance and representational alignment, whereas operator perturbations produce sharp degradation. The results also show that surface fluency does not reliably indicate logical correctness, and that structurally diverse filtered data retains strong performance.

  • Perturbation robustness: Operand-Perturbed Atomic-QA consistently matches standard Atomic-QA across 7B, 8B, and 14B models, with only a 0.002 average drop for Qwen3-8B.
  • Perturbation robustness: At nearly 100% operand error, performance remains largely unaffected, while 100% operator error causes significant decline across Alpaca and MetaMath.
  • Representational alignment: The last-layer CKA gap between clean and operator-perturbed models is 0.935 versus 0.832 for LLaMA and 0.950 versus 0.899 for Qwen.
  • Surface fluency and logic: Operator-Perturbed data has perplexity 35.80 versus 36.53 for Atomic-QA on LLaMA3.1-8B, despite its logical corruption.Operand-Perturbed data with random strings instead has perplexity 99.76.
  • Statistical significance: Atomic-QA improves over baseline means significantly on every model family, with gains from +0.0264 to +0.0377 and p < 0.01.
  • Filtering and scale: Atomic-LIFD-20K consistently outperforms LIMA while using approximately 15% fewer tokens.

E Beta-Distribution Based Error Assignment

The paper assigns atomic-function errors stochastically across samples using a Beta distribution while preserving a prescribed global error rate. This approach introduces realistic sample-level heterogeneity and corrects rounding and normalization effects so the final assignment matches the target.

  • Problem Formulation: The method targets a global atomic-function error rate r across M samples containing a total of N atomic functions.For each sample, the selected error count is constrained so aggregate errors match round(r · N).
  • Candidate Construction: Each sample receives one candidate version containing k corrupted atomic functions, with successive versions adding one corruption at a time.The procedure pre-generates n_i + 1 candidate versions and selects exactly one version per sample.
  • Motivation: Unlike deterministic uniform assignment, stochastic sampling creates heterogeneous error rates across samples, reflecting variation in data quality.The naive method assigns approximately the same error rate to every sample, whereas Beta sampling allows some samples to be nearly perfect and others to contain multiple errors.
  • Stochastic Sampling Phase: Beta-distribution sampling assigns each sample an error probability based on the target rate r and concentration parameter κ.The parameterization uses α = rκ and β = (1 − r)κ; κ controls distribution shape.
  • Normalization, Adjustment, and Quantization: Normalization, clipping, quantization, and residual adjustment align sampled probabilities and integer error counts with the prescribed global total.The adjusted probabilities are clipped to [0, 1], rounded to integer counts, and iteratively corrected for the residual between the target and assigned totals.
Loading 2608.29144v1…