PaperlayerView exact revision on arXiv

Source-linked AI summary

Language models fail at extended rule following

Tianxiang Dai, Jonathan Fan

arXiv:2605.02028v2cs.CL

TL;DR

Language models need to preserve exact state while repeatedly applying rules, but evidence for this capability remains limited. The paper evaluates 126 model variants with a counting-based procedural assay and finds broadly negative results, including plausible outputs without preserved rule-defined states.

  • Problem

    The study addresses whether models can reliably preserve state while applying simple procedural updates, despite strong performance on complex reasoning and long-context tasks.

  • Method

    The authors evaluate 126 language-model variants using SCC, a minimal assay of procedural state preservation and reliable procedure execution.

  • Results

    The findings are broadly negative: models can emit plausible numerical outputs while no longer preserving the rule-defined state.

  • Takeaways & Limitations

    Models may appear to execute procedures without carrying the variable required to preserve the procedure’s state.

  • Takeaways & Limitations

    Homogeneous counting is intentionally artificial and does not measure the full range of procedures required in natural tasks.

Abstract

from arXiv · show

Large language models are highly capable of answering difficult questions by retrieving, recombining, and attending to information in long contexts. For agentic tasks, an additional capability is required: the preservation of an exact state while repeatedly applying rules. We find that this reliability is absent across language models. To demonstrate, we query 126 leading model variants with the task of counting a long string of repeated characters, and we find they all cannot accurately count above a model-dependent, syntax-sensitive counting capacity threshold. Failures are abrupt and persist even with increasing model size, inference time computation, and external tool. Mechanistic probing indicates that models use a finite number of internal states to mimic counting as a rule and fail once these states are exhausted. Furthermore, such states are the basis for performing complex tasks beyond counting. These results indicate that fundamentally new model architectures are required for autonomous agents to achieve truly reliable rule following capabilities.

Results

Models count accurately only within a stable, model-specific regime before an abrupt collapse into large, structured errors, consistent with finite syntax-sensitive internal states. These states also constrain complex reasoning and agentic workflows, while standard benchmark performance weakly reflects procedural reliability.

  • Counting results: Models count accurately through a stable range, then undergo an abrupt structural collapse into large, unpredictable errors clustered around preferred integers.This pattern appears across model families and contradicts smoothly growing numerical error from continuous approximation.
  • Mechanistic analysis: Counting capacity is syntax-sensitive rather than a single abstract counter, with different characters and delimiters inducing distinct supporting feature trajectories.Changing the repeated character or delimiter preserves coarse progress but reorganizes the feature coalition underlying the state.
  • Agentic implications: Complex reasoning and coding disrupt counting more than a second independent count, and externalization does not reliably restore exact tracking or downstream task success.In gpt-5.4, exact marker counting fell to 15.8% and simulated file externalization to 15.0–16.7%; in claude-sonnet-4-6, downstream success reached 12.5% in the combined condition.
  • Mechanistic analysis: The collapse supports a finite internal state for counting: count-related representations are linearly readable during success but disappear when the model fails.Mechanistic analyses indicate organized count-like trajectories, late transfer to the final prompt state, and exhaustion of decoder-usable states.
  • Benchmark comparison: Counting capacity correlates only weakly to moderately with benchmark scores, so higher benchmark performance does not necessarily imply longer-horizon procedural state maintenance.The results indicate that conventional leaderboards are largely blind to fundamental procedural reliability.

Discussion

SCC isolates reliable rule execution and shows that even benchmark-leading LLMs can lose exact procedural state during extended counting. The findings imply that direct reliability measurement and architectural support for persistent, verifiable state are necessary.

  • Core finding: SCC finds that LLMs broadly fail to preserve procedural state and return correct final values, even when they excel at reasoning, coding, and long-context benchmarks.High aggregate benchmark scores therefore do not establish underlying procedural reliability.
  • Mechanism: Counting behavior indicates finite rule-like patterning: models produce exact answers within a supported regime, then emit plausible outputs without preserving rule-defined states.The procedure can appear to continue after the required variable has been lost.
  • Mechanism: Near the counting-capacity boundary, distributed state trajectories become decoder-unusable, so longer generation cannot reconstruct collapsed state and tools may externalize corrupted substitutes.Chain-of-Thought style generation and tool use can scaffold semantic reasoning but do not guarantee exact rule execution over long horizons (13,17–19,23).
  • Implications: For autonomous coding, planning, tool-use, and decision-support systems, state maintenance is locally reliable but globally brittle, enabling silent failure after thousands of apparently correct steps.Exact, independently verifiable state transfer is required for tool calls and external memory to mitigate the problem.
  • Limitations: Because counting is artificial, proprietary evaluations are partly unobservable, and mechanistic analyses cover only open-weight models, reliable rule execution should be measured directly.Behavioral dissociation across model families argues against inferring reliability from benchmarks, nominal context length, or inference-time computation.

Supplementary materials … Cross-benchmark alignment and paired analyses

The supplementary methods define the model coverage, counting prompts, adaptive SCC estimation, hierarchical rule-tracking assay, and cross-benchmark paired analyses used to test exact rule following. They evaluate 126 model variants and relate SCC to 47 public benchmarks and reasoning-augmented counterparts.

  • Supplementary materials: The supplementary materials provide the evaluation records and methodological details supporting the study’s model catalogue and analyses.
  • Model set and evaluation scope: The evaluation covered 126 proprietary and open-weight language-model variants spanning instruction tuning, reasoning augmentation, parameter counts, and context lengths.This scope tested whether SCC generalized across tokenizers, serving stacks, and model families; tool use was disabled.
  • Prompt construction and response parsing: Counting prompts used repeated lowercase items separated by comma-space delimiters, with targets defined as item counts rather than token counts.Responses were parsed for the last valid integer and compared with the exact target, while malformed or unparsable outputs counted as failures.
  • Adaptive estimation of Stable Counting Capacity: SCC estimates the largest length regime in which a model reliably preserves the exact tally, using randomized tiers, an nMAE < 0.05 stability threshold, and binary refinement after failure.Within-tier target variation prevents approximate or collapsed-number strategies from passing consistently, with an estimated guessing false-positive rate of approximately 0.025%.
  • Hierarchical rule tracking assay: The hierarchical assay tested equality-rule tracking by counting records whose KEY exactly matched the deepest PATH token amid nested structures and distractors.It required cumulative counting across records while ignoring negative distractors whose number scaled with the target match count.
  • Cross-benchmark alignment and paired analyses: SCC was cross-referenced with GPQA Diamond, ARC-AGI-2, SWE-Bench Verified, and OTIS Mock AIME, with correlations computed across 47 public benchmarks.The cited benchmark bibliography markers are (4,5,15,35,36).
  • Cross-benchmark alignment and paired analyses: Matched-pair analyses compared base models with reasoning-augmented counterparts using SCC boundaries and average total token consumption at the stable boundary.
  • Cross-benchmark alignment and paired analyses: Figure 5E plotted token-expenditure multipliers against ΔSCC to assess whether additional generation-time computation expanded stable capability.

Matched dual task counting controls … Residual stream projections and targeted causal interventions

The study tests exact counting under added reasoning and agent-operation demands, maps collapse boundaries and syntax effects, and probes the latent count representation with causal interventions. Mechanistic interventions identify a shift in causal count representation from intermediate sequence tokens to the final response prefix.

  • Matched dual task counting controls: Matched dual-task prompts pair marker counting with benchmark questions to measure interference between exact state tracking and complex reasoning.Target counts ranged from 32 to 96, with matched-length distractors, irrelevant code snippets, and an independent counting control using another marker.
  • Agent-operation and externalization assay: The agent-operation assay embeds marker counting in account updates, transaction-order preservation, net-total computation, and next-action selection across four task conditions.Conditions included an agent-only baseline, inserted marker counting, copy externalization, and copy-then-agent execution, scored for exact counts, copied content, and downstream fields.
  • Dense sweeps and motif perturbations: Dense count sweeps locate the transition from exact to failed outputs and identify incorrect-value attractors near the SCC boundary.The sweeps use actual counts around the boundary identified by a randomized ladder and aggregate incorrect predictions to characterize failure outputs.
  • Dense sweeps and motif perturbations: Varying repeated characters and delimiters compares relative SCC with tokenizer-specific compression to separate raw token volume from syntax-dependent counting trajectories.This tests whether changes in learned counting behavior track sequence syntax rather than token quantity alone.
  • Residual stream projections and targeted causal interventions: Causal patching across layers isolated a shift in count representation from intermediate sequence tokens, such as layer 31, to the final response prefix, such as layer 53.The study used donor activations, sequence-token replacement, and targeted final-token substitution to connect latent representations with generated counts.
  • Residual stream projections and targeted causal interventions: Targeted counter-projection clamping shifts failed hidden states along a learned progress direction while preserving orthogonal components, testing whether the coordinate supports state maintenance.Forty trials measured rescued exact greedy generation, changes in correct-logit margins, and downstream projection errors.

Sparse autoencoders and feature coalitions · Supplementary Note 1: Mathematical Bounds on Guessing Exploits · Supplementary Note 2: Prompt Templates and Generation Settings

The supplementary analyses identify syntax-sensitive sparse feature structure and establish bounds against successful guessing exploits. They also specify the exact prompting and deterministic generation settings used by the benchmark harness.

  • Sparse autoencoders and feature coalitions: Sparse autoencoder analyses use Gemmascope 2 to decompose residual representations into sparse feature dictionaries, ranking features by Pearson correlation with count within verified successful bounds.Feature rankings were recalculated under motif variants.
  • Sparse autoencoders and feature coalitions: The analyses measure Jaccard overlap among top-ranked feature sets and baseline-feature rank displacement under syntax perturbations.These comparisons are reported in Fig. 4, E to H, alongside the feature-dictionary analysis in Fig. 4D.
  • Supplementary Note 1: Mathematical Bounds on Guessing Exploits: The ±20% fuzzy boundary combined with relative MAPE prevents models from minimizing error through a generalized prior guess.This bound addresses architectures that lose sequence cohesion but retain approximate sequence depth.
  • Supplementary Note 1: Mathematical Bounds on Guessing Exploits: ≈10.1% is the lowest expected MAPE for guessing under the α=0.20 fuzzy boundary, while guessing the distribution center yields ≈10.2%.The optimal fixed guess is the geometric mean of the uniform distribution’s boundaries.
  • Supplementary Note 1: Mathematical Bounds on Guessing Exploits: At τ=5% and K=16, guessing reaches a Z-score of −3.48 and an approximately 0.025% false-positive rate.With a single trial, K=1, a guessing model has a 24.6% chance of passing randomly.
  • Supplementary Note 2: Prompt Templates and Generation Settings: The benchmark instructs models to count items exactly and return only one integer, using a fixed template asking for occurrences of the labeled symbol.The sequence is a generated repeated-token sequence with a chosen delimiter.
  • Supplementary Note 2: Prompt Templates and Generation Settings: Across APIs, the harness uses deterministic prompting with fixed templates, a search seed of 7, and output caps of 4096 initial and 32768 retry tokens.If output is capped without a parseable integer, the active token budget doubles until parsing succeeds or the retry ceiling is reached.

Supplementary Note 3: Full Model Catalogue

Supplementary Table S1 catalogs the evaluated models with developer or family, parameter metadata, nominal context window, architectural class, and measured Counting capacity. Architecture labels are inferred from parameter metadata, distinguishing MoE, Dense, and unavailable cases.

  • Catalogue contents: Table S1 provides the full model catalogue and records each model’s developer or family, parameter metadata, nominal context window, architecture class, and measured Counting capacity.The catalogue includes models from OpenAI, Qwen, Google, DeepSeek, Mistral, MiniMax, Anthropic, and Moonshot.
  • Metadata and architecture: Architecture labels mark models with distinct total and active parameter counts as MoE, single-parameter-count models as Dense, and unavailable cases as “–”.Nominal context-window and parameter metadata come from run manifests when available, while architecture is inferred from published parameter metadata.

Supplementary Note 4: Audit of Instruction-Following Failures

Beyond numerical errors, models exceeding stable sequence capacity often suffer severe instruction-following collapse, including formatting violations, token-limit failures, and unrelated or incomplete outputs. An audit found these failures across many model variants, even when parsers could recover valid integers.

  • Audit results: 501 of 9,797 trials (5.1%) violated the single-integer response constraint, spanning 44 model variants; parsers recovered valid integers in 305 cases (60.9%).Failures included extraneous text, formatting artifacts, or no number at all, and were counted from raw responses even when fallback parsing succeeded.
  • Failure modes: The most common deviations were unsolicited step-by-step explanations and blank or whitespace outputs, indicating attempts to bypass internal state loss through explicit generation.These categories were summarized in Supplementary Table S2.
  • Structural collapse: Structural state collapse caused regressions to pre-training priors or contextual disassociation, including unrelated reasoning, Python code, endless sequence echoes, and token-limit termination.Examples included deepseek-r1 producing unrelated calculus or algebra traces, gemini-3-flash-preview writing parsing code, and mistralai/devstral-2512 echoing the sequence indefinitely.

Supplementary Note 5: Heterogeneous Nested Key-Path Tracking Assay

The heterogeneous nested-tracking assay tests hierarchical depth tracking and multi-slot variable binding amid semantic distractors. Frontier models’ performance boundary collapses at the initial 32-match tier, indicating generalized fragility in higher-layer abstraction rather than simple-counting-specific failure.

  • Assay design: Each record requires matching its KEY token to the deepest PATH leaf while ignoring diverse distractors and alternating bracket geometries.Successful counting requires parallel hierarchical parsing, conditional equality evaluation, and cumulative summation across records.
  • Results: At the minimal tier of L = 32 target matches, performance collapses for the vast majority of frontier API model architectures.Models failing this initial accuracy threshold are categorically assigned Counting Capacity (CC) < 32; Supplementary Table S5 reports the assay results.
  • Results: The nested assay shows that structural variable binding ablates the state-maintenance capacity observed in the linear homogeneous assay.The result extends the observed capability cliff beyond one-dimensional accumulation and zero-entropy prompts.
  • Results: The abrupt capability cliff therefore reflects generalized fragility in higher-layer abstraction, not an anomaly specific to simple token counting.Scatter plots for the nested assay are provided in Supplementary Figure S8.

Supplementary Note 6: Anatomy of Latent Reasoning Breakdowns

Exposed reasoning traces show a sharp breakdown in sequence counting: models are universally correct at length 8 but often drift, round, or fail to terminate by length 96. Successful traces preserve the actual stopping boundary, whereas longer failed traces elaborate substitutes for the prompt rather than restoring the count.

  • Trace set and parsing: At L=96, only 5 of 30 records are correct, 24 fail, and 1 fails at the transport layer, compared with all 30 correct at L=8.At L=32, most errors are small local drifts; the most common coherent L=96 error is 100, appearing in 7 records.
  • Failure categories: Long failed traces commonly recount, copy or expand the prompt, impose a 10-by-10 schema, exhaust the budget, or derail into an unrelated task.These behaviors indicate counting-shaped text generation disconnected from the original sequence rather than reliable bounded counting.
  • Failure categories: At L=32, failures usually involve local item or comma-count drift; at L=96, models more often jump to rounded attractors, over-enumerate, or fail to terminate.The traces remain organized and can describe valid algorithms, but applying those algorithms to the actual sequence causes state drift or global collapse.

Supplementary Note 7: Matched Dual-Task Counting Controls

Matched dual-task controls show that complex reasoning causes earlier and larger counting errors than plain counting or length-matched irrelevant controls. The pattern supports finite, shared internal state representations that are consumed by semantic reasoning rather than raw token length.

  • Methods: The experiment paired homogeneous counting with matched BBH, CRUXEval-O, MATH-500, or MMLU-Pro tasks across target counts from 32 to 96, using six trials per count and category.Outputs required a strictly formatted JSON object containing both the parsed count and secondary-task solution.
  • Methods: Format-matched controls inserted syntactically valid irrelevant code or a secondary counting sequence at the real prompt’s estimated token length to separate cognitive load from context length.The mean-error summary plotted average parsed-count error against true count using a five-point centered smoothing window.
  • Results: Main Text Fig. 2F shows that MATH-500, BBH, and CRUXEval-O tasks induced earlier and larger undercounting than plain counting and length-matched irrelevant-text/code controls.Plain counting and irrelevant controls had minimal error up to sequence count 60, whereas complex reasoning tasks degraded accuracy sooner and more severely.
  • Results: Complex logical reasoning degraded counting more severely than the dual-counting control, indicating that interference depends on semantic execution rather than merely tracking another sequence.The dual-counting control was matched to the benchmark prompt while tracking ‘b’ markers.

Supplementary Note 8: Agentic Operation and Counting

The agent-operation assay tested whether models could preserve operational state while counting or exactly externalizing marker sequences. gpt-5.4 preserved surrounding transaction updates despite count/copy corruption, whereas claude-sonnet-4-6 also destabilized structured responses, ruling out simple externalization as a defense.

  • Supplementary Note 8: Agentic Operation and Counting: The assay separated counting or externalization failure from the surrounding deterministic task by requiring balance updates, review-identifier preservation, net-total computation, and action selection.Four conditions compared transaction updates alone, marker counting, copying into a file field, and copying followed by the transaction task.
  • Supplementary Note 8: Agentic Operation and Counting: gpt-5.4 preserved the transaction-update task while exact marker counting and copying failed at shorter lengths, leaving the surrounding agent state intact.The failures produced structured wrongness: copied lengths and reported counts drifted toward nearby or rounded values such as 200, 400, 500, and 1000.
  • Supplementary Note 8: Agentic Operation and Counting: claude-sonnet-4-6 showed stronger operational collapse: transaction performance remained mostly correct, but strict JSON adherence failed and externalization often became unparsable or refusal-like.Both wrong copied files and empty refusal-like completions break the operational pipeline.
  • Supplementary Note 8: Agentic Operation and Counting: Both models lost exact externalization, showing that copying the marker sequence is itself an exact state-preservation problem rather than a reliable escape from counting failure.For claude-sonnet-4-6, coupling externalization with the agent task also destabilized the whole structured response.

Supplementary Note 9: Additional Results on Gemma Model

Gemma counting capacity depends on the exact repeated-character motif and delimiter, while causal interventions show that extracted count directions control outputs only within the stable regime. Distributed middle-layer representations feed a causal response-prefix state, but low-dimensional control cannot restore collapsed high-dimensional features after stability fails.

  • Motif perturbation: Gemma’s CC limits varied by motif, from 11 for a space delimiter to 36 for the Chinese character “han,” despite mechanically identical counting tasks.Supplementary Table S11 defines the CC limit as the last exact count before the first observed failure.
  • Causal interventions: Sequence-token interventions along the Layer 31 count direction successfully modulated generated counts for target lengths up to 26, while final-token steering produced robust linear output shifts.These interventions tested whether controlled manipulation of learned one-dimensional residual-stream count states predictably changes decoded counts within the stable regime.
  • Causal interventions: Final-token donor patching was inert at early and middle layers but exerted significant overriding control at resid_post.layer53, whereas sequence-token patching controlled counts at resid_post.layer31 before saturation.The divergent depth profiles indicate that count information accumulates across distributed middle-layer sequence representations before consolidating into a causal response-prefix representation.
  • Causal interventions: The extracted count coordinate is behaviorally coupled to output within the stable regime, but cannot restore the collapsed high-dimensional feature coalition after the stability boundary is breached.Counter-direction clamping failures establish the limitation of low-dimensional control once the underlying state has collapsed.

Supplementary Note 10: Causal Counter-Direction Clamping Analysis

Counter-direction clamping in gemma-3-27b-it rejects the hypothesis that a 1D linearly decodable count state causally drives sequence generation. Clamping beyond Counting Capacity disrupts rather than rescues tracking, indicating distributed, nonlinear state maintenance.

  • Supplementary Note 10: Causal Counter-Direction Clamping Analysis: 0 exact-generation rescues occurred in 40 counter-direction patching trials, rejecting the 1D causal hypothesis for sequence counting.The intervention decreased the minimum correct-logit gap by an average of -0.256.
  • Supplementary Note 10: Causal Counter-Direction Clamping Analysis: Layer-53 projection error expanded by up to -875 times at L=27 when the 1D state was clamped at early layers.Clamping targeted layers 16, 31, 40, and 53 for counts 27–160 beyond Counting Capacity.
  • Supplementary Note 10: Causal Counter-Direction Clamping Analysis: Artificial translation along the 1D count axis pushes residual activations off the valid data manifold and causes latent-space collapse.The collapse reflects structural disintegration of a high-dimensional feature coalition rather than successful correction of the count state.

Supplementary Note 11: Qwen MoE Latent Tracking Replication

Replication in the sparse qwen3.5-35b-a3b MoE model recovered linearly readable count directions, degradation near a stable boundary, and causal final-token steering. These findings support bounded state tracking and abrupt structural collapse as generalized across transformer paradigms.

  • Supplementary Note 11: Qwen MoE Latent Tracking Replication: Despite discontinuous MoE token routing, robust one-dimensional count directions emerged across layers within the successful tracking regime.The analysis covered layers 15, 23, 31, and 39.
  • Supplementary Note 11: Qwen MoE Latent Tracking Replication: Applying final-token steering along the Layer-31 direction manipulated the decoded integer comparably within valid bounds.This behavioral intervention replicated causal manipulation of the tracked count variable in the MoE model.
  • Supplementary Note 11: Qwen MoE Latent Tracking Replication: The replication indicates that bounded, linearly readable state-tracking coordinates and their abrupt structural collapse generalize across modern transformer paradigms.The correct-logit margin showed parallel degradation as sequence lengths exceeded the stable boundary.
Loading 2605.02028v2…