Source-linked AI summary
The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity?
Alexander Hägele, Aryo Pradipta Gema, Henry Sleight, Ethan Perez, Jascha Sohl-Dickstein
TL;DR
As AI systems become more capable and handle more consequential tasks, it is unclear whether their failures will reflect coherent pursuit of unintended goals or incoherent behavior. The paper uses a bias–variance decomposition to measure error-incoherence and finds that longer reasoning and action sequences consistently make failures more incoherent, while model scale has experiment-dependent effects.
Problem
The paper addresses whether AI failures increasingly reflect systematic pursuit of unintended goals or inconsistent behavior as model intelligence and task complexity grow.
Method
The paper decomposes model errors into bias and variance and measures error-incoherence as variance's proportion of total error across test-time randomness and several task types.
Results
Across multiple-choice, coding, and safety tasks, longer reasoning increases error-incoherence, while larger models are often more incoherent and scale effects vary by task difficulty.
Takeaways & Limitations
The results suggest that advanced AI systems performing complex tasks may fail inconsistently rather than through pursuit of a stable goal, informing comparisons among AI risk scenarios.
Takeaways & Limitations
The paper does not experimentally or theoretically investigate the specific mechanisms behind increasing error-incoherence with trajectory length and sometimes model size.
Abstract
from arXiv · showhide
As AI becomes more capable, we entrust it with more general and consequential tasks. The risks from failure grow more severe with increasing task scope. It is therefore important to understand how extremely capable AI models will fail: Will they fail by systematically pursuing goals we do not intend? Or will they fail by being a hot mess, and taking nonsensical actions that do not further any goal? We operationalize this question using a bias-variance decomposition of the errors made by AI models: An AI's \emph{error-incoherence} on a task is measured over test-time randomness as the fraction of its error that stems from variance rather than bias in task outcome. Across all tasks and frontier models we measure, the longer models spend reasoning and taking actions, \emph{the more incoherent} their failures become. Error-incoherence changes with model scale in a way that is experiment dependent. However, in several settings, larger, more capable models are more incoherent than smaller models. Consequently, scale alone seems unlikely to eliminate error-incoherence. Instead, as more capable AIs pursue harder tasks, requiring more sequential action and thought, our results predict failures to be accompanied by more incoherent behavior. This suggests a future where AIs sometimes cause industrial accidents (due to unpredictable misbehavior), but are less likely to exhibit consistent pursuit of a misaligned goal. This increases the relative importance of alignment research targeting reward hacking or goal misspecification.
1 INTRODUCTION
The paper asks whether increasingly capable AI systems fail through consistent pursuit of unintended goals or through inconsistent, incoherent behavior. It introduces a bias–variance framework and reports that longer reasoning and more sequential actions increase error-incoherence across several settings.
- AI is increasingly used for consequential tasks, making it important to understand how capable systems fail.
- Misalignment risk concerns systems that pursue an unintended goal with superhuman competence.
- The paper contrasts coherent optimization of the wrong objective with failures that appear random and do not further any coherent goal.
- The authors ask how bias and variance contribute to deviations from intended behavior as model intelligence and task complexity increase.
- The paper defines error-incoherence as the proportion of total error attributable to variance rather than bias.
- Across multiple-choice, coding, and safety tasks, longer reasoning is associated with greater incoherence, even when task difficulty is controlled.
2 BACKGROUND
The background develops bias–variance decomposition as the paper’s framework for separating systematic error from stochastic deviations. It defines error-incoherence for fixed models under test-time randomness and motivates model size and reasoning length as scaling axes.
- Bias–variance decomposition expresses expected prediction error as squared bias, variance, and irreducible noise.
- For classification, bias reflects error in the mean or mode prediction, while variance measures deviations from that prediction.
- The paper assumes zero irreducible noise when decomposing expected cross-entropy error into KL-BIAS and KL-VARIANCE.
- Unlike classical analyses based on retraining, this study averages over input and output randomness for a fixed model and task.
- Error-incoherence is the proportion of variance in total error, ranging from 0 for fully consistent errors to 1 for entirely inconsistent errors.
- The study examines model size, dataset size, compute, and reasoning length as scaling dimensions, with reasoning treated as sequential action steps.
3 EXPERIMENTS
Across benchmark, coding, safety, and synthetic tasks, longer reasoning or action sequences are associated with greater error-incoherence. Model scale has task-dependent effects: larger models are often more incoherent on harder tasks, while ensembling and reasoning budgets can reduce incoherence.
- Experimental setup: Experiments span multiple-choice, agentic coding, safety, and synthetic optimization tasks with targets suitable for measuring incoherence.The evaluated tasks include GPQA, MMLU, SWE-BENCH, Model-Written Evals, and a controlled quadratic-optimization setting.
- Reasoning and action length: Longer reasoning and action sequences increase error-incoherence across all tested conditions.The trend appears across frontier models and QWEN3, although baseline levels and slopes differ by model and task.
- Reasoning and action length: Natural overthinking produces substantially higher incoherence while average accuracy and SWE-BENCH scores remain similar.For each question or task, samples above the median reasoning length or action count are compared with samples below it.
- Model scale and task complexity: Larger models become more coherent on easy questions but more incoherent on the hardest questions, although the hardest-task trend is noisy.Performance improves with model size, with the fastest improvement on the hardest questions, while incoherence changes in opposite directions by difficulty.
- Synthetic optimization: In the synthetic task, increasing model size reduces bias faster than variance, producing increasingly variance-dominated rollouts.All models show rising incoherence over rollout steps, while smaller models can reach lower plateaus after failing to follow the correct trajectory.
- Error correction: Ensembling reduces error-incoherence, while larger reasoning budgets reduce it only slightly and less than natural variation does.For ensembles, variance falls approximately as 1/E without changing bias; reasoning budgets improve performance and slightly reduce incoherence for most tested models.
4 RELATED WORK
Prior work examines reasoning-related performance degradation, evaluation variability, and scaling behavior, while this paper distinguishes systematic errors from inconsistent failures through bias–variance analysis.
- Reasoning: Extended reasoning can degrade performance and increase output variance, but prior studies do not distinguish systematic errors from inconsistent failures.The paper’s ensembling analysis also connects to self-consistency work by framing aggregation as reducing error-incoherence.
- Evaluation variance: Evaluation research documents highly variant AI benchmarks and develops sensitivity and consistency metrics for characterizing failure modes.This paper connects such variability to bias and variance in a safety setting.
- Scaling behavior: Scaling studies report aligned representations and errors and improved long-horizon performance as models become larger and more capable.This paper complements those findings by measuring increased error-incoherence with longer reasoning and action across model families.
5 DISCUSSION AND WHAT OUR RESULTS DO NOT TELL US
The discussion identifies mechanisms and scope boundaries for increasing error-incoherence, including trajectory-level variance accumulation, reward misspecification, and limited applicability to open-ended goals.
- Mechanisms: The paper does not experimentally or theoretically determine the mechanisms behind increasing error-incoherence with trajectory length or, sometimes, model size.The authors present motivating observations rather than a tested mechanism.
- Mechanisms: As models’ effective state and action spaces expand, constraining generic dynamical systems to optimize a fixed loss becomes more difficult.The paper frames this as one possible reason capable models may become more incoherent.
- Mechanisms: Variance typically accumulates across trajectories unless an active correction mechanism such as ensembling is available.Irreversible real-world actions can make correcting noise introduced by model actions impossible or impractical.
- Reward misspecification: The experiments largely lack meaningful reward misspecification, so the results do not establish how poorly specified objectives would affect error decomposition.The authors speculate that specification bias could dominate as variance and mesa-level bias decline with capability.
- Open-ended goals: The analysis relies on settings with well-defined metrics, leaving hidden goals and complex incoherent behavior in open-ended tasks as an important unresolved area.The paper uses multiple-choice classification, coding unit tests, and objective functions rather than LLM judges to maintain defined and comparable metrics.
6 CONCLUSION
The paper proposes bias–variance decomposition to study AI failure and finds that longer reasoning and action sequences increase error-incoherence. Smarter models are not consistently more coherent, suggesting complex-task failures may be inconsistent rather than goal-directed.
- Longer sequences of reasoning and actions consistently increase model error-incoherence.
- Smarter AI models are not consistently more coherent across tasks.
- Advanced AI systems performing complex tasks may fail inconsistently rather than pursue a stable goal.
ETHICS STATEMENT
The research characterizes failure modes of increasingly capable AI systems to inform safer deployment. Its findings suggest that extended reasoning on complex tasks is associated with more incoherent failures, while the work does not directly prevent failures.
- The research aims to characterize failure modes of increasingly capable AI systems to inform safer deployment strategies.
- Complex tasks requiring extended reasoning are associated with more prevalent incoherent failures than systematic misalignment.
- The work offers empirical grounding for prioritizing safety interventions, but does not directly prevent AI failures.
REPRODUCIBILITY STATEMENT
The paper documents its bias–variance framework, classification assumptions, experimental setups, and evaluation procedures. It also records a limitation of the 0/1 decomposition and describes the datasets and sampling procedures used across experiments.
- The classification formulation assumes noiseless labels, with models producing probability distributions over class labels.
- The 0/1 decomposition cannot average bias and variance separately across questions without a multiplicative factor.
- For 0/1 loss, the paper reports absolute bias and variance errors in the Appendix but does not compute error-incoherence.
- The experiments use multiple-choice, safety, open-ended, and coding evaluations with repeated model samples and task-specific variance measures.
- Probability prompting asks models to estimate the correctness probability of each answer choice, and the reported behavior is qualitatively consistent with standard prompting.
B.4 SYNTHETIC TASKS
The paper describes synthetic quadratic-trajectory tasks, token-based transformer training, trajectory evaluation, and a survey measuring intelligence and coherence. Figure 8 compares frontier-model accuracy and error metrics across reasoning budgets and reasoning lengths.
- B.4 SYNTHETIC TASKS: The synthetic task uses a four-dimensional ill-conditioned quadratic function and 4,096-step trajectories generated by steepest descent.
- B.4 SYNTHETIC TASKS: Transformer models range from roughly 47 thousand to 5 million parameters and are trained with next-token prediction on trajectory data.
- B.4 SYNTHETIC TASKS: Evaluation samples starting positions and generates complete trajectories autoregressively, adding each decoded update to the current state.
- B.5 SURVEY ON INTELLIGENCE AND ERROR-INCOHERENCE: The survey asks subjects to rank entities by intelligence or coherence, with separate subjects evaluating the two attributes and randomized presentation order.
- B.5 SURVEY ON INTELLIGENCE AND ERROR-INCOHERENCE: Figure 8 shows inference-scaling improvements, lower accuracy for longer-reasoning questions, and similar qualitative behavior across error metrics.
C.1 GPQA MODEL PERFORMANCE OVERVIEW & DIFFERENT METRICS
These GPQA results document performance and error decomposition across model scales, reasoning-enabled variants, and alternative metrics. The supplementary analyses show qualitatively consistent bias–variance behavior and reduced error-incoherence under ensembling.
- Accuracy and overall error are summarized for frontier models, providing the performance baseline for the GPQA analyses.
- Alternative bias–variance decompositions show the same qualitative behavior as the main KL-error-incoherence analysis.
- Ensembling reduces variance and therefore lowers error-incoherence, while Laplace smoothing causes bias to decrease slightly with ensemble size.
- QWEN3 performance shows a multiplicative interaction between reinforcement learning and model scale, with a noticeable slope increase from instruct to thinking models.
C.2 SCALING LAWS WITH OTHER MODELS AND BENCHMARKS
The scaling-law analyses extend the study to GPQA and to GEMMA3 and LLAMA3. Across these settings, the relationship between model scale and error-incoherence is examined with benchmark- and model-family-specific setup differences.
- For QWEN3 on GPQA, reasoning length predicts error-incoherence more strongly than model size.
- GEMMA3 and LLAMA3 are evaluated alongside QWEN3 to test whether error-incoherence scaling findings generalize across model families.Unlike QWEN3, GEMMA3 and LLAMA3 are not native reasoning models and require chain-of-thought prompting.
C.3 REASONING VARIATION, ERROR CORRECTION, WAIT RATIOS
These analyses distinguish deliberate inference scaling from naturally variable reasoning and examine error correction, alternative metrics, model scaling, and qualitative response instability. Natural overthinking is associated with substantially higher error-incoherence despite similar task performance.
- Reasoning variation: Natural overthinking produces higher error-incoherence than longer reasoning budgets, while the two groups have similar accuracy or SWE-BENCH scores.
- Error correction and wait ratios: Wait-ratio analyses find no clear GPQA relationship and a shared positive MMLU relationship, while the paper does not further analyze reasoning structure.
- Different metrics: Across bias and variance metrics, errors increase with reasoning length, variance grows faster, and error-incoherence rises accordingly.
- Different metrics: Ensembling drastically reduces variance and lowers error-incoherence, although Laplace smoothing makes bias decrease slightly with ensemble size.
- Scaling analyses: For GPQA, harder question groups become more incoherent with model scale because variance scaling becomes shallower while bias slopes remain similar.
- Scaling analyses: Error-incoherence is independent of error magnitude, so models with lower overall error can have the same incoherence as higher-error models.
- Scaling analyses: Reasoning length has a stronger effect on error-incoherence than model size in both GPQA and MMLU.
- Scaling analyses: Across model families, the hardest questions generally show higher error-incoherence with model size, with LLAMA3 reported as the exception.
C.10 SURVEY RESULTS
The survey results and supplementary analyses examine whether perceived intelligence relates to error-incoherence across biological creatures, AI models, and human organizations. They also provide measurement checks, broader benchmark analyses, and related-work context.
- Survey results: Across biological creatures, AI models, and human organizations, subjects judged entities to have higher error-incoherence when they were judged more intelligent.
- Measurement and supplementary analyses: Bias and variance estimates stabilize at approximately 30 samples in the GPQA and O3-MINI sampling-efficiency analysis.
- Supplementary evaluations: Across MWE variations, error-incoherence generally grows with longer reasoning, although results vary and can be noisy.
- Supplementary evaluations: For SWE-BENCH, error-incoherence rises similarly when plotted against action rounds, output tokens, or reasoning length, while coverage error increases.
- Supplementary analyses: Model scale primarily reduces bias rather than variance in the reported loss-scaling analysis.
- Survey results: The survey separates the three entity groups while preserving the same positive relationship between judged intelligence and judged error-incoherence.
- Related work: Related work frames longer reasoning as a capability-scaling strategy, while other studies question whether scaling always improves behavior.
- Related work: Ensembling and verifier-based aggregation are presented as ways to marginalize individual samples and suppress error-incoherence, beyond improving accuracy.