Source-linked AI summary
The pitfalls of next-token prediction
Gregor Bachmann, Vaishnavh Nagarajan
TL;DR
The paper asks whether next-token learning can faithfully support planning, distinguishing autoregressive inference from teacher-forced training. It identifies a teacher-forcing failure mechanism, demonstrates it on an easy graph-planning task across Transformer and Mamba, and reports teacherless multi-token training as a promising workaround. The conclusions motivate alternatives to the next-token prediction paradigm while leaving broader settings open.
Problem
The paper addresses the gap between criticisms of autoregressive error accumulation and the less-examined possibility that teacher-forced training itself fails to learn accurate next-token predictors for lookahead tasks.
Method
The paper distinguishes inference-time autoregression from teacher-forced training, analyzes a Clever Hans shortcut mechanism, and evaluates a minimal graph-planning task with Transformer, Mamba, and teacherless training.
Results
Teacher-forcing fails on the straightforward planning task for both Transformer and Mamba, while teacherless training predicts multiple future tokens and circumvents the failure in some settings.
Takeaways & Limitations
The findings support treating teacher-forced training and autoregressive inference as distinct and motivate exploration of training paradigms beyond next-token prediction.
Takeaways & Limitations
The authors note that teacherless training is promising only in some settings, and the broader extension to unstructured tasks such as story-writing remains conjectural.
Abstract
from arXiv · showhide
Can a mere next-token predictor faithfully model human intelligence? We crystallize this emerging concern and correct popular misconceptions surrounding it, and advocate a simple multi-token objective. As a starting point, we argue that the two often-conflated phases of next-token prediction -- autoregressive inference and teacher-forced training -- must be treated distinctly. The popular criticism that errors can compound during autoregressive inference, crucially assumes that teacher-forcing has learned an accurate next-token predictor. This assumption sidesteps a more deep-rooted problem we expose: in certain classes of tasks, teacher-forcing can simply fail to learn an accurate next-token predictor in the first place. We describe a general mechanism of how teacher-forcing can fail, and design a minimal planning task where both the Transformer and the Mamba architecture empirically fail in that manner -- remarkably, despite the task being straightforward to learn. Finally, we provide preliminary evidence that this failure can be resolved using _teacherless_ training, a simple modification using dummy tokens that predicts multiple tokens in advance. We hope this finding can ground future debates and inspire explorations beyond the next-token prediction paradigm. We make our code available under https://github.com/gregorbachmann/Next-Token-Failures
1. Introduction
The paper separates autoregressive inference from teacher-forced training and argues that teacher-forcing itself can fail on lookahead tasks. A minimal graph-planning task exposes this failure in Transformer and Mamba models, while teacherless multi-token training shows promise as a remedy.
- Failure mechanism: Teacher-forcing can induce a Clever Hans cheat by using revealed answer prefixes to fit later tokens while making earlier tokens harder to learn.The resulting shortcuts remove supervision about the full answer from the earlier planning decisions.
- Implications: The findings motivate more careful debate about next-token prediction and exploration of training paradigms beyond it.The authors specifically connect the straightforward-task failure and positive teacherless results to future work on more complex tasks.
- Core distinction: Next-token prediction debates often conflate autoregressive inference with teacher-forced training, which can produce different failure modes.The paper argues that existing critiques mainly target inference while assuming teacher-forcing learned an accurate predictor.
- Empirical demonstration: The authors design a minimal lookahead task and find teacher-forcing fails for both Transformer and Mamba despite the task being easy to learn.The empirical failure occurs in a graph path-finding setup and is described as complete in-distribution failure.
- Alternative objective: Teacherless training predicts multiple future tokens at once and shows promise for circumventing teacher-forcing failures in some settings.The paper presents this as a simple multi-token modification and preliminary evidence rather than a universally established solution.
2. The Two Modes of Next-Token Prediction
The paper defines two distinct uses of next-token prediction: autoregressive inference consumes the model’s own outputs, whereas teacher-forced training consumes ground-truth prefixes.
- Setup: A next-token model assigns probability to one future token given an input sequence, while joint sequence probability follows by combining conditional predictions.The notation treats an exact token-by-token match as ˆr = r.
- Autoregressive inference: During autoregressive inference, the model samples each token using the prefix plus its previously generated tokens.The model therefore feeds its own outputs back as inputs at inference time.
- Teacher-forced training: During teacher-forced training, the model receives ground-truth response prefixes and is trained to predict the next ground-truth token.The objective maximizes next-token log-probabilities under those ground-truth prefixes.
- Key consequence: Teacher-forcing exposes the model to preceding ground-truth response tokens, a property central to the failure mechanism developed later.The training inputs differ from the model-generated inputs used during autoregressive inference.
3. Failure due to Auto-Regressive Inference
The chain-rule defense concerns expressive power, while snowballing errors concern inference-time execution; neither addresses whether teacher-forced learning produces a good planner.
- Chain-rule defense: The chain rule shows that any sequence distribution can be represented by an appropriately expressive next-token predictor.This establishes representational capacity, not that training will discover the required predictor.
- Snowballing errors: Small per-token errors during autoregressive generation can accumulate and sharply reduce the probability of an exact long-sequence match.The paper frames this as a shortcoming of autoregressive execution.
- Interpretive boundary: Snowballing errors do not rule out a model having learned a good plan that it fails to execute during inference.Verification, backtracking, or feedback-based wrappers could potentially recover such a plan.
- Remaining gap: The paper argues that existing defenses and criticisms address architecture or inference, not shortcomings in learning how to plan.This motivates separating teacher-forced training failures from autoregressive execution failures.
4. Failure due to Teacher-Forcing
Teacher-forcing can fail on lookahead tasks because answer prefixes enable shortcuts that remove supervision needed to learn the underlying plan. In a minimal path-finding task, this mechanism produces in-distribution failure and may extend to structured tasks such as story-writing.
- Path-Star Task: The path-star graph is a minimal, straightforward lookahead task designed to isolate the planning mechanism without irrelevant confounding factors.The response is the unique path from v_start to v_goal, and a right-to-left solution can compute and reverse that path.
- Mechanism: Teacher-forcing decomposes learning p → r into token-level mappings from (p, r<i) to r_i, rather than learning the response from p alone.The revealed ground-truth prefix supplies additional information that changes the learning problem.
- Clever Hans Cheat: Answer prefixes enable Clever Hans cheating, especially for later tokens, by allowing trivial mechanisms to use revealed response information.These shortcuts fit easy tokens without requiring lookahead.
- Indecipherable Token: Once easy tokens are fit, gradient starvation deprives the model of supervision needed to learn the true right-to-left planning solution.The remaining supervision can be incomplete and statistically harder to use for recovering a composed plan.
- Indecipherable Token: With only early-token supervision, composing l discrete subroutines can require exponential search of Ω(|C|^l) under the proposition’s assumptions.An incorrect subroutine can alter the final output completely, creating an all-or-nothing loss surface.
- Empirical Scope: Teacher-forcing causes complete in-distribution failure in the path-finding setup for both Transformer and Mamba architectures, while the proposed story-writing extension remains conjectural.The paper presents story-writing as a conceptual application whose empirical testing is not straightforward.
5. Experimental Verification
Experiments on path-star graphs show that teacher-forced models can fit training data yet fail completely in-distribution, while teacherless and reversed-target objectives can help on easier settings.
- Teacher-forced failure: Teacher-forced Transformer and Mamba models fit the training data but exhibit complete in-distribution failure during autoregressive evaluation.This remains true despite identical training and test topology and training accuracy reaching 100%.
- Experimental setup: The experiments use path-star graphs with fixed topology across training and testing, randomized labels and adjacency orders, and 200k samples with N = 100.The setup is intended to test algorithm learning rather than memorization or length generalization.
- Teacher-forced failure: Teacher-forcing exploits revealed ground-truth prefixes by predicting each node from the preceding true node rather than learning the path from the graph description.The held-out cheat accuracy is approximately 100% across most settings, except challenging high-degree graphs.
- Teacher-forced failure: The first path node remains indecipherable under teacher-forcing, with Acc1st(LMθ) approximately 1/d because the model lacks information about subsequent targets.The model therefore emits one of the d neighbors of the start node essentially at random.
- Alternative objectives: Teacherless training replaces ground-truth prefixes with repeated dummy tokens, forcing predictions to rely on the graph description while retaining the original targets for supervision.Inference conditions on dummy tokens rather than feeding previous predictions autoregressively.
- Alternative objectives: Teacherless models are often too difficult to fit, but on easier graphs they fit and generalize, while later tokens achieve higher accuracy earlier as models learn right-to-left.Reversing targets also significantly boosts learning, including for models trained from scratch.
6. Related Work
Related work distinguishes support for next-token prediction from criticisms focused on planning and inference, while this paper attributes failures to the objective rather than a particular architecture.
- Arguments in support of next-token prediction: Prior work argues that language redundancy supports next-token prediction and reports language models outperforming humans on next-token prediction in OpenWebText.These findings do not rule out poor planning performance.
- Arguments in support of next-token prediction: Theoretical and empirical studies show that autoregressive Transformers and chain-of-thought supervision can increase expressive power or enable otherwise unlearnable multi-hop tasks.The paper states that its negative result does not contradict these positive learnability results.
- Arguments against next-token prediction: The snowballing failure mode concerns inference-time execution after errors, whereas this paper studies an immediate failure in learning the next-token predictor itself.In the studied setting, the error occurs at the beginning rather than accumulating over time.
- Arguments against next-token prediction: Existing work reports failures on planning, arithmetic, summarization, and creative-generation tasks framed as evidence that autoregressive next-token predictors may be ill-suited for planning.This paper positions its path-finding counterexample as a formalization of that emerging intuition.
- This work: The paper introduces the Clever Hans cheat and Indecipherable Token failure, observing them in both Transformer and Mamba models.It therefore attributes the reported failures to next-token prediction rather than specifically to the Transformer architecture.
- Beyond next-token prediction: Alternative paradigms explored in related work include non-autoregressive, energy-based, diffusion, reordered-token, all-at-once, and lookahead-training approaches.Teacherless training was previously proposed for image captioning and parallel speculative sampling under different motivations.
7. Conclusion
The paper argues that next-token prediction has important limits as a model of intelligence, especially for planning, and distinguishes teacher-forcing failures from autoregressive execution failures. It proposes teacherless training and related alternatives as directions for addressing these limits.
- Next-token prediction may be insufficient for modeling human thought because human problem-solving involves imagining, curating, and backtracking plans.
- Teacher-forcing and autoregressive inference must be distinguished because they produce different failure modes.
- Teacher-forcing failures can occur even in straightforward planning tasks, rather than arising from autoregressive error accumulation or architecture alone.
- A teacher-forced model trained on fiction may still struggle to generate plot twists, although this possibility is presented speculatively.
- Teacherless training and chain-of-thought supervision are proposed as possible alternatives, though their applicability to unstructured story-writing remains unclear.
Impact Statement
The paper identifies broad potential societal consequences because its results concern a foundational technique at the heart of modern AI systems, but does not single out specific consequences.
- The results concern a foundational technique at the heart of modern AI systems.
- The paper acknowledges many potential downstream societal consequences of such foundational work.
- No specific societal consequence is highlighted in the impact statement.
A. Limitations
The study is limited by its empirical and conceptual scope, incomplete evaluation across model scales and task families, and the presence of specialized workarounds for its minimal task.
- The arguments are empirical and conceptual rather than formally proved.
- Failure has not been demonstrated for very large models such as Llama2 or Mistral.
- Specialized workarounds may solve the path-star task, including pretraining, in-context learning, multimodal learning, or parallelizable solutions.
- Beyond minimal path-finding, the range of problems affected by teacher-forcing-induced failure has not been demonstrated or characterized.
- It remains unclear whether the failure generalizes to ordinary text-generation tasks.
B. Teacher-Forcing Failure and Snowballing Failure are Distinct
Teacher-forcing-induced failures and autoregressive snowballing are distinct planning pathologies with different remedies. The paper illustrates this distinction using path-finding and contrasts teacherless training with several task-specific workarounds.
- The two planning failures are orthogonal and therefore require different solutions.
- A teacher-forced model can fail immediately at the first path-finding step, without errors accumulating over sequence length.
- A model with small per-step errors can learn the correct plan yet fail during long autoregressive inference through snowballing.
- Snowballing may be addressed with backtracking-and-planning wrappers, whereas teacher-forcing pathology cannot be solved post-hoc.
- The story-writing example requires planning dependencies across events, including a resolution that must guide earlier setup and conflict generation.
- Teacherless training can require computing and storing the complete path solution before producing the first token.
E. Experiment Notations
The experiments define metrics for exact-match autoregressive performance, first-token accuracy, snowballing after teacher-forcing the first token, and teacherless or reversed responses. They also measure Clever Hans cheating across graph types.
- Metrics: Exact-match autoregressive accuracy compares each generated path with its ground-truth path.The evaluation generates solutions autoregressively and scores exact agreement.
- Metrics: First-token accuracy measures whether the model predicts the first token after the start node.This isolates the difficult initial prediction in the path-finding task.
- Training variants: Teacherless responses use repeated dummy tokens instead of the model’s previously generated outputs during inference.This matches the uninformative-input design used for teacherless training.
- Metrics: Snowball accuracy evaluates generation after teacher-forcing the first token, testing whether later errors compound during inference.The experiments use graphs with varying path lengths.
- Snowball Failure: Snowball accuracy remains barely affected for paths up to L = 40, while cumulative success decays exponentially but stays high on G2,40.These results indicate that snowballing is not prominent in this setting.
- Metrics: Clever Hans cheating accuracies quantify teacher-forced performance across graph types; they are nearly perfect in most cases except the high-degree graph G20,5.The exception reflects difficulty fitting that training distribution.
F.3. More Detailed Accuracies
Detailed results compare standard, teacherless, and reversed training on graph and arithmetic tasks. Teacherless training can solve many graph tasks, reversal performs near perfectly, and teacherless training improves sample efficiency for 3-digit addition, though its cheating mechanism is less clear.
- Graph accuracies: Standard next-token prediction is difficult on the graph tasks, with performance limited to approximately 1/d.Here d is the graph degree.
- Graph accuracies: Teacherless training works very well with GPT2-Large, which solves most graph tasks perfectly, while from-scratch models generally struggle.GPT-Mini succeeds only on the simplest graph, G2,5.
- Graph accuracies: Reversing the response significantly simplifies graph learning, producing near-perfect accuracies across all graphs.The comparison is reported in the reversed-accuracy results.
- Arithmetic tasks: Teacherless training yields more sample-efficient learning than standard training for 3-digit addition, alongside reversed training as the comparison condition.Test accuracy is tracked over training iterations.
- Arithmetic tasks: For 3-digit addition, the precise form of teacher-forcing cheating is harder to characterize because leading answer digits can inform subsequent digits.The passage gives carry-related digit information as one possible shortcut.
- Indecipherable Token: The model fails to learn the first path token even when that token is trained in isolation, supporting the role of Clever Hans cheating in the original failure.The isolated task removes the rest of the path and becomes classification.
G. Other experimental details
The paper specifies graph tokenization, model configurations, and related distinctions around next-token prediction. Its evidence separates inference-time snowballing from teacher-forcing failures and connects teacherless training to hindsight supervision.
- Experimental details: Graph inputs are tokenized as a shuffled edge list, start and goal nodes, and a target sequence containing the full path.The prefix is not predicted, whereas the repeated start-goal path is the learning target.
- Experimental details: Experiments use GPT-Mini, GPT2-Large, and Mamba to test whether the failure depends on architecture.GPT-Mini is trained from scratch, GPT2-Large is pretrained, and Mamba provides a recurrent comparison.
- Conceptual distinctions: The paper distinguishes its teacher-forcing failure from snowballing, which assumes that teacher-forcing has already learned an accurate next-token predictor.Snowballing concerns execution of a learned plan during inference, whereas the paper challenges learning itself.
- Conceptual distinctions: The Clever Hans shortcut maps teacher-forced prefixes to the next token and can persist even with unlimited training data.It differs from shortcuts mapping the original input prefix directly to the ground-truth response.
- Related supervision: Teacherless models sometimes exploit hindsight chain-of-thought even though the supervision appears after the end target.This differs from related work that provides chain-of-thought before the end target.
- Related supervision: Reversal simplifies addition but can hurt language-model perplexity, so its effect is task-dependent.The paper contrasts prior addition findings with natural-language results.