PaperlayerView exact revision on arXiv

Source-linked AI summary

Truth as a Compression Artifact in Language Model Training

Konstantin Krestnikov

arXiv:2603.11749v3cs.CLcs.AI

TL;DR

The paper asks why language models prefer correct answers when contradictory answers appear equally often, and tests whether error compressibility rather than truth explains that preference. Controlled contradictory corpora and natural-language transfer experiments show that models favor structurally incoherent errors less often, while coherent false systems can match truth; the authors frame this as a compression-consistency hypothesis whose large-scale generality remains open.

  • Problem

    It is unclear why the training objective favors one answer when correct and incorrect information appear at equal frequency, limiting prediction of failures on consistent errors.

  • Method

    The paper trains transformers from scratch on contradictory mathematical and natural-language corpora while systematically varying error structure under controlled frequency, format, and domain conditions.

  • Results

    Across controlled settings, correctness preference appears for random errors but disappears for a single coherent false rule, returns with competing rules, and reproduces on Wikipedia text.

  • Takeaways & Limitations

    In the studied settings, models track the description length and consistency of competing answer systems rather than truth per se.

  • Takeaways & Limitations

    Extension to production-scale pretraining remains unconfirmed because the scale-up uses models far below production sizes and a single seed per condition.

Abstract

from arXiv · show

Why do language models trained on contradictory data prefer correct answers? In controlled experiments with small transformers (3.5M--86M parameters), we show that this preference tracks the compressibility structure of errors rather than truth per se. We train GPT-2 style models on corpora where each mathematical problem appears with both correct and incorrect solutions -- a denoising design that directly models conflicting information about the same fact. When errors are random, models extract the correct signal with accuracy scaling from 65% to 85% with model size. When errors follow a coherent alternative rule system, accuracy drops to chance (~45--51%): the model cannot distinguish the false system from truth. A multi-rule experiment reveals a sharp crossover: a single coherent alternative rule eliminates truth bias entirely, but adding a second competing rule restores most of it (47%->78%), with continued growth through N=10 (88%). The same pattern reproduces on real Wikipedia text (71% vs 46%). We propose the Compression--Consistency Principle as an explanatory hypothesis: in these settings, gradient descent favors the most compressible answer cluster, not truth per se. Truth bias emerges only when falsehood is structurally incoherent. Whether this principle extends to large-scale pretraining remains an open question.

1 Introduction

The paper asks why models prefer correct answers amid equally frequent contradictions and hypothesizes that error structure, through compression, determines this preference. Controlled experiments show that random errors preserve correctness preference, whereas coherent false rules can eliminate it.

  • Contradictory training data can contain equally frequent correct and incorrect answers, leaving unexplained why the training objective favors one.The motivation is predicting when filtering of noisy information will fail, especially when errors are internally consistent.
  • Cross-entropy minimization is framed as code-length minimization, linking model training to the Minimum Description Length principle.The proposed explanation treats compact answer systems as favored by compression.
  • The experiments train transformers from scratch on problems paired with correct and incorrect solutions while varying only error structure.Frequency, format, and domain are held constant to isolate error structure as the independent variable.
  • The contribution is an experimentally supported hypothesis that contradictory corpora favor consistency rather than truth under the compression objective.The reported tests include random, coherent, and multi-rule errors, Wikipedia entity substitution, and larger natural-text training.
  • 65% to 85% correctness preference occurs for random errors, while a coherent false rule can be equally compact and remove that preference.Random errors require individual memorization; coherent errors form a structured alternative system.

2 Related Work

Related work connects language-model factual behavior to compression, internal representations, data statistics, simplicity bias, and noisy-label learning. The paper identifies systematic variation of error compressibility in a denoising setting as an unstudied gap.

  • Compression research links prediction objectives to universal compression, benchmark performance, knowledge acquisition, scaling, and hallucination.The cited literature spans Shannon, Solomonoff, MDL, and recent empirical and theoretical work on LLM compression.
  • Prior studies find structured internal representations and truth-correlated representations, while this work focuses on behavioral effects caused by data-level conflicts.The paper positions its analysis at the behavioral rather than representation level.
  • Research on truthfulness and conflicting knowledge reports frequency, document-count, personas, formality, and surface-quality effects on model preferences.These findings provide complementary explanations for how models resolve conflicting information.
  • Simplicity-bias and noisy-label studies show that neural networks can prefer simple functions, memorize random labels, and remain robust to substantial label noise.The paper extends this line by testing structured noise whose errors follow coherent rules.
  • Systematic variation of error compressibility in a denoising setting had not previously been studied, according to the paper.

3 Methodology

The methodology creates contradictory mathematical and natural-language corpora, varies error structure at controlled frequency, and evaluates whether models prefer correct completions. It combines MDL-based task design with paired and generative evaluation.

  • Corpus and hypothesis: The experiments compare random, coherent, and multi-rule errors while varying error structure and holding contradiction frequency central to the design.Natural-language transfer uses Wikipedia entity substitution to test beyond formal mathematics.
  • Corpus and hypothesis: When a false system is less compressible than the true system, MDL favors truth; when both are equally compact, it provides no basis for preference.With multiple false rules, a selector adds log N bits to the description length.
  • Corpus and hypothesis: “Truth” denotes mathematical correctness and Wikipedia factual accuracy, while the compressibility gap is domain-dependent and smaller in natural language.
  • Models and training: The models are custom GPT-2-like decoder-only transformers, with internal size labels that do not correspond to OpenAI GPT-2 checkpoints.The configurations are summarized in Table 1.
  • Corpus design: Each denoising corpus contains 5,000 unique generated and verified mathematical problems, each appearing with contradictory answers.The primary tokenizer is character-level, while BPE robustness is separately verified.
  • Corpus design: Wikipedia transfer uses 20,000 paragraphs with random or coherent same-type entity substitutions and evaluates 2,000 held-out paired paragraphs.Coherent substitution preserves grammatical structure and cross-entity consistency, whereas random substitution breaks co-occurrence patterns.
  • Evaluation: 85% accuracy at 86M parameters occurs with random wrong answers, whereas coherent wrong answers remain near 50% across scales.This is the central denoising comparison summarized in the paired-evaluation table and Figure 1.
  • Evaluation: Paired evaluation compares completion NLLs conditioned on a shared prompt, reporting the fraction of pairs where the correct completion receives lower NLL.Length matching checks that pair accuracy is not driven by completion length; greedy decoding provides a generative evaluation.

4 Results

Across controlled mathematical and natural-text settings, models prefer correct answers when errors are random but lose that preference when false answers follow a coherent rule. Adding competing rules, changing tokenization or architecture, and scaling to 1B parameters clarify that compressibility and consistency govern the effect.

  • 4.1 The Central Contrast: Random vs Coherent Errors: 85% accuracy is reached at 86M parameters with random wrong answers, while coherent wrong answers remain near chance across scales.Random accuracy rises monotonically from 65% to 75%, 81%, and 85%; coherent accuracy stays near chance.
  • 4.2 Multi-Rule Errors: The Sharp Crossover: 78% accuracy at N=2 follows 47% at N=1, then accuracy grows gradually through N=10.Each alternative rule is compact, but randomly selecting which rule applies makes the problem-to-rule mapping unpredictable.
  • 4.3 Transfer to Wikipedia Text with Entity Substitution: 71% accuracy for random substitution contrasts with 46–49% for coherent substitution on Wikipedia text.The effect is weaker than in math and saturates near 70% across the tested range.
  • 4.4 Robustness Checks: 75.9% random accuracy with BPE versus 65.3% with character-level tokenization shows the effect survives and strengthens under tokenizer changes.Coherent accuracy remains at chance: 49.3% in denoising and 45.9% in standard evaluation.
  • 4.4 Robustness Checks: 74.5% ± 0.4% matched-random accuracy remains far above 47.2% coherent accuracy despite both corrupting approximately three derivation steps.This controls for the possibility that coherent errors appear different only because they affect more steps.
  • 4.6 Cross-Architecture Validation: 86.8% accuracy for random errors versus 50.6% for coherent errors reproduces the contrast in a Qwen3-0.6B architecture.The result is therefore not specific to the GPT-2-like architecture used in the primary experiments.
  • 4.7 Scaling to 1B on Real Text: 76.8% paired accuracy versus 46.7% at 1B parameters reproduces the contrast when contradictory math is only 8% of a naturalistic corpus.The random condition reports ΔLoss = +0.098 with p < 10^-6, while coherent errors remain at chance.

5 Discussion

The experiments support a compression-based account in which consistency, rather than truth, determines preference when contradictory answer systems compete. Truth bias appears for incoherent errors but disappears for compact coherent falsehoods, with frequency and rule assignment determining the boundary.

  • Truth bias emerges only when false alternatives are structurally incoherent; internally consistent false rule systems compress comparably to the true system.
  • The role of frequency: 67% accuracy persists with a 10/90 correct-to-incorrect ratio for random errors, whereas coherent errors yield 72% false-system preference at 40/60 and 91% at 20/80.
  • The role of frequency: Equal-frequency coherent falsehood can neutralize truth bias, while frequency favors whichever equally compressible system appears more often.
  • What the multi-rule experiment shows: The N=1→2 transition indicates that unpredictable rule assignment, not individual rule complexity, introduces the incompressible information restoring truth preference.
  • Implications and scope: Cross-domain verification partially restores truth bias, but verification accuracy falls from 71% at 3.5M to 61% at 86M parameters.
  • The analogy to Popper’s falsifiability is limited because models do not test theories against evidence, though both favor lower description length.

6 Conclusion

The paper finds that correctness preferences in its experiments track the description length of false answer systems rather than truth itself. This mechanism reproduces across error structures, evaluation modes, natural-language data, and architectures, but its extension to large-scale pretraining remains open.

  • Models prefer correct answers only when errors are structurally incoherent; a coherent alternative removes this preference, while competing rules restore it.
  • The pattern reproduces on Wikipedia paragraphs, in generative evaluation, and across architectures, including experiments reaching approximately 1B parameters.

Limitations

The evidence is constrained by model scale, domain specificity, generation-versus-discrimination differences, limited seed counts, and imperfect causal matching. The authors also identify future tests involving internal representations, RLHF, and in-context learning.

  • Model scale: Primary experiments use 3.5M–86M-parameter models, while the ∼1B-parameter extension is only a single-seed scale-up sanity check.Replication at multi-billion scale with compute-matched training and multiple seeds remains an important next step.
  • Domain specificity: Mathematics offers an unusually crisp correct/incorrect distinction, but the effect weakens in natural language, where errors can remain locally fluent.The reported natural-language comparison is 71% vs 85%; domains with competing real-world knowledge systems remain untested.
  • Discriminative vs generative gap: Paired accuracy reaches 85% versus 53% generative accuracy at large scale, leaving a gap because real-world language-model behavior depends on generation.The gap narrows from 49 percentage points for tiny models to 32 for large models, suggesting scale partially bridges it.
  • Seed counts: Core conditions use four seeds, while some use two, supporting directional stability but not tight confidence intervals.The limited seed counts constrain statistical precision.
  • Causal identification: Random and coherent corruptions may differ in affected derivation steps and surface-form simplicity, weakening causal identification despite partial multi-rule controls.A fully matched control for corruption load, lexical diversity, and output length would strengthen the causal claim.
  • Future directions: Future work could probe truth and coherence directions and test interactions with RLHF and in-context learning.These extensions would connect the behavioral findings to internal-representation research and broader model behavior.

Ethical Considerations

The paper argues that truth preference in controlled contradictory-data settings reflects compressibility rather than truth itself, while cautioning against direct production-model conclusions.

  • Scope and ethical relevance: The controlled findings do not directly demonstrate vulnerability in production models.The authors frame transparent reporting of truth-bias failure conditions as useful for data curation and model evaluation.
  • Compression–Consistency Principle: Random errors require individual memorization, whereas coherent false rules can be as compact as correct rules.The MDL framing predicts a preference for correct rules only when alternatives are structurally less compressible.
  • Compression–Consistency Principle: A single coherent alternative eliminates truth preference, while competing false rules restore it as selector information becomes less compressible.The false-system description length grows as O(n log N) when randomly assigning N rules to problems.
  • Compression measure: A gzip compression analysis across nine conditions finds a positive association between compression ratio and paired accuracy.The reported Spearman correlation is ρ = 0.68 with p = 0.042; the authors present this as supporting evidence.

D Standard (Non-Denoising) Experiments

Standard training presents each problem once, unlike denoising’s within-problem contradictions; although standard accuracy is higher, the difference narrows with scale and coherent errors remain near chance.

  • Standard versus denoising: 47–52% standard coherent accuracy remains at chance, preserving the random-versus-coherent contrast.The table summary states that coherent models follow pure frequency across proportions.
  • Noise ratios: At a 4:1 noise ratio, denoising accuracy degrades gracefully and plateaus for medium and large models.This result is reported for increasing noise ratios in the denoising setup.
  • NLL distribution: Random-error NLL differences are right-skewed, whereas coherent-error differences are symmetric.Random errors have mean +0.048, median +0.025, and 81.5% positive differences; coherent errors have 45.5% positive.

E Additional Transfer Experiments

Additional experiments test transfer beyond formal mathematics and show that cross-domain tasks or verification dependencies can restore some truth preference under coherent errors.

  • Synthetic world: 57.7% random accuracy exceeds 46.6% coherent accuracy in a synthetic world with 50 entities, 4 types, and 15 deterministic rules.Per-type performance varies, with minerals at 68.7% and potions near chance at 49.1%.
  • Wikipedia transfer: Wikipedia transfer evaluates random versus coherent entity substitutions across 20,000 paragraphs.The supplied table caption identifies accuracy by entity type for random substitution, while the setup describes both corruption modes.
  • Cross-domain tasks: 35% →56% derivative accuracy follows adding correct cross-domain tasks at 25% to a coherent corpus.The effect is non-monotonic and drops at 50% because of dilution.

F Robustness Checks

Robustness checks show that the central contrast survives tokenizer changes and additional evaluation settings, while verification effects weaken with model size; the paper is primarily empirical and reproducibility information is documented.

  • Verification dependencies: 70.9% tiny-model accuracy follows embedding verification dependencies, versus 43% for standard coherent conditions.The effect decreases to 64% at small and 61% at large model sizes.
  • Training dynamics: All model sizes reach a behavioral plateau by steps 3000–4000, with the large model at 88.8% at step 3000.The large-model result remains stable through step 5000.
  • Theoretical scope: The paper explicitly presents its three MDL predictions as informal predictions with proof sketches, not formal theorems about neural-network training.The work is characterized as primarily empirical.
  • Reproducibility: All hyperparameters, architectures, generation procedures, evaluation protocols, seeds, and generation templates are reported for reproduction.The paper also reports error-bar definitions, statistical tests, and per-seed breakdowns.
Loading 2603.11749v3…