PaperlayerView exact revision on arXiv

Source-linked AI summary

Language models fail at extended rule following

Tianxiang Dai, Jonathan Fan

arXiv:2605.02028v2cs.CL

TL;DR

Reliable rule following requires preserving state while repeatedly applying procedures, but whether language models can do this reliably remains an open question. The paper tests this with simple counting and finds that models broadly fail to preserve rule-defined states, even when they excel on other benchmarks.

  • Problem

    Whether language models can preserve state while applying simple procedures reliably remains an open question.

    Source: Discussion · p. 13

  • Method

    The paper uses a minimal assay asking whether a model can apply a procedural update, preserve the resulting state in context, and return the correct final value.

    Source: Discussion · p. 13

  • Results

    The answer is broadly negative, even for systems excelling at complex reasoning, coding, and long-context benchmarks; models can emit plausible numerical outputs after losing rule-defined states.

    Source: Discussion · p. 13

  • Takeaways & Limitations

    Reliable rule execution matters for coding, tool use, planning, and decision support, which must maintain task states across extended interactions.

    Source: Discussion · pp. 13–14

    Homogeneous counting is intentionally artificial and does not measure the full range of procedures required in natural tasks.

    Source: Discussion · p. 14

Abstract

from arXiv · show

Large language models are highly capable of answering difficult questions by retrieving, recombining, and attending to information in long contexts. For agentic tasks, an additional capability is required: the preservation of an exact state while repeatedly applying rules. We find that this reliability is absent across language models. To demonstrate, we query 126 leading model variants with the task of counting a long string of repeated characters, and we find they all cannot accurately count above a model-dependent, syntax-sensitive counting capacity threshold. Failures are abrupt and persist even with increasing model size, inference time computation, and external tool. Mechanistic probing indicates that models use a finite number of internal states to mimic counting as a rule and fail once these states are exhausted. Furthermore, such states are the basis for performing complex tasks beyond counting. These results indicate that fundamentally new model architectures are required for autonomous agents to achieve truly reliable rule following capabilities.

Results

The results characterize models’ procedural counting reliability, examining abrupt failure, syntax sensitivity, and count-related internal structure. They also relate bounded counting capacity to complex-task performance and benchmark scores, which only partly capture exact rule execution.

Model dynamics when counting fails

  • Models count accurately within a stable range, then abruptly collapse into large errors and preferred-number guesses, consistent with finite internal states.After losing count, outputs often cluster around values such as 500, 1000, or 2000; 5% of evaluated outputs also failed to provide a valid single-number response.Source: Model dynamics when counting fails · pp. 4–5
  • Changing the counted character or delimiter shifts counting capacity, indicating that models use syntax-dependent learned trajectories rather than one abstract state shared across equivalent symbols.Capacity sometimes changed despite little change in relative input token count.Source: Model dynamics when counting fails · pp. 5–6
  • In nested-depth tracking, even the highest-performing model maintained stability for only 416 true matches before structural collapse.The failure dynamics matched those of basic counting despite the task requiring richer structural state.Source: Model dynamics when counting fails · p. 5
  • Coupling counting with reasoning and coding increased counting errors more than the controls, while tracking a second independent count caused less interference.The authors interpret this as evidence that complex problem solving and procedural tracking compete for a limited internal resource.Source: Model dynamics when counting fails · p. 8
  • In gpt-5.4, exact marker counting was 15.8% and simulated file externalization was 15.0–16.7%, despite the surrounding agentic task remaining correct.The results indicate that externalizing the count did not reliably remove the tracking burden.Source: Model dynamics when counting fails · p. 8

Probing bounded state trajectories within models

  • During successful counting, a linearly readable count-related coordinate tracked the true count, then disappeared at the same point counting behavior failed; the first error occurred at 27 items.Source: Probing bounded state trajectories… · pp. 8–9
  • Teacher-forced analysis found that the correct-count logit margin decayed sharply near and beyond the capacity boundary and could become negative after collapse.The model could stop preferring the correct integer even under the correct answer prefix.Source: Probing bounded state trajectories… · p. 9
  • Count-correlated sparse-autoencoder features were structured and non-monotonic, and changing input syntax reorganized the supporting feature coalition.The results suggest related but distinct trajectories rather than one abstract counter shared across surface forms.Source: Probing bounded state trajectories… · p. 9
  • Length interpolation preserves a coarse linear trajectory while disrupting exact nonlinear token-by-token correspondence, controlling for sequence-length differences in patching.The study compares replacing only the start-of-turn token with replacing repeated-token states across the prompt.Source: Probing bounded state trajectories… · p. 9
  • At layer 51 of 62, final-token patching affects output only in late layers; full-sequence patching is stronger around middle layer 31.The results suggest per-token progress trajectories form in intermediate layers before count information transfers to the final prompt state for decoding.Source: Probing bounded state trajectories… · pp. 9, 11
  • Clamping scalar progress coordinates did not reliably rescue failed counts, indicating a richer causal representation; similar effects appeared in Qwen 3.5 35B-A3B.The findings suggest the phenomenon is not limited to the dense transformer used for mechanistic inspection.Source: Probing bounded state trajectories… · p. 11
  • Models appear to assign repeated items to finite count-like states, enabling exact decoding while organized, but guess plausibly once those states are exhausted or disrupted.After collapse, rule-defined count information is no longer available in decoder-usable form.Source: Probing bounded state trajectories… · p. 11

Comparing SCC with standard evaluations

  • Correlations between counting capacity and standard benchmark scores are weak to moderate overall, while post-release ARC-AGI-2 scores correlate nearly linearly with log(CC).The comparison covers GPQA Diamond, SWE-bench Verified, and ARC-AGI-2.Source: Comparing SCC with standard evaluations · pp. 11–12, Figure 5 · p. 12
  • ARC-AGI-2 score gains after its release reflect adaptation to a fixed abstract task format, not necessarily superior generalizable agentic analysis.Older models generally scored poorly, while models trained with knowledge of the benchmark’s format improved; benchmark familiarity can mask bounded exact rule execution.Source: Comparing SCC with standard evaluations · p. 12

Discussion

Accessing long contexts, solving benchmark tasks, and executing procedures reliably are distinct capabilities. SCC directly assays procedural rule execution and finds it broadly unreliable, even in systems strong on complex reasoning and long-context benchmarks.

Discussion

  • SCC’s direct assay finds procedural reliability broadly lacking, even in models that excel at complex reasoning, coding, and long-context benchmarks.High aggregate benchmark scores do not by themselves establish reliable procedure execution.Source: Discussion · p. 13
  • Reliable rule execution should be measured directly rather than inferred from benchmark performance, nominal context length, or inference-time computation.Standard benchmarks remain informative, but their high scores do not establish underlying procedural reliability.Source: Discussion · pp. 13–14
  • Counting behavior is consistent with finite rule-like patterning: models produce exact answers within a supported regime, then emit plausible outputs without preserving rule-defined states.Near the counting-capacity boundary, distributed state trajectories cease to be decoder-usable, so a model may appear to follow a procedure without carrying its required variable.Source: Discussion · p. 13
  • Because procedural state is lost rather than merely obscured, longer generation cannot reliably restore it, and tools may act on a corrupted substitute; Chain-of-Thought and tool use do not guarantee exact execution over long horizons.A tool can only externalize state the model can still represent, while scaffolding and external workspaces can improve semantic reasoning.Source: Discussion · p. 13
  • The structural causes of these limits remain unresolved: no single mechanism yet explains basic counting failures in deployed systems, and targeted ablations are needed to identify dominant causes across models.Potential contributors include routing, finite precision, positional encoding, attention sparsification, or lack of explicit recurrence.Source: Discussion · pp. 13–14
  • State maintenance is locally reliable but globally brittle: deployed models can match a requested procedure for thousands of steps while remaining vulnerable to sudden, silent failure.Autonomous coding, tool use, planning, and decision support require maintaining constraints, intermediate variables, commitments, and action preconditions across extended interactions.Source: Discussion · p. 14
  • Tool calls and external memory are useful mitigations only when transferred state is exact and independently verifiable; improving reliability requires architectural support for persistent variables, explicit states, recurrence, and verifiable traces.Increasing model size, extra training, or generated tokens alone is insufficient; external memory is also among the proposed architectural supports.Source: Discussion · p. 14
  • The study’s limitations include artificial homogeneous counting, limited observability of proprietary evaluations, and mechanistic analyses confined to open-weight models that do not prove all models fail identically.The counting metric does not span the full range of natural procedures, and proprietary preprocessing, hidden prompts, and reasoning-token accounting are not fully observable.Source: Discussion · p. 14

Supplementary materials … Cross-benchmark alignment and paired analyses

The supplementary materials describe the model set, counting and rule-tracking assays, and analyses relating Stable Counting Capacity (SCC) to public benchmarks and reasoning-augmented models.

Supplementary materials

Model set and evaluation scope

  • The evaluation covered 126 proprietary and open-weight model variants, including instruction-tuned and reasoning-augmented systems across broad parameter and context-length ranges, with tool use disabled.The range was chosen to test whether SCC generalizes across tokenizers, serving stacks and model families.Source: Model set and evaluation scope · p. 31

Prompt construction and response parsing

  • Counting prompts used repeated lowercase characters separated by commas and spaces, targeted item counts rather than token counts, and treated responses without a parsable integer as failures.Responses were parsed for the last valid integer and compared with the exact target.Source: Prompt construction and response parsing · p. 31

Adaptive estimation of Stable Counting Capacity

  • SCC estimation expanded tiers with nMAE below 0.05 and binary-refined tiers above that threshold; randomized target counts limited guessing-based false positives to approximately 0.025%.SCC estimates the largest length regime over which a model reliably preserves the exact tally.Source: Adaptive estimation of Stable Counting… · p. 32

Hierarchical rule tracking assay

  • The hierarchical assay counted records only when the KEY token matched the deepest token in a nested PATH, requiring cumulative counting while ignoring distractors.Prompts included negative distractor records, with their number scaled relative to the target match count.Source: Hierarchical rule tracking assay · p. 32

Cross-benchmark alignment and paired analyses

  • Correlations were calculated across intersecting model subsets for 47 public benchmarks, including GPQA Diamond, ARC-AGI-2, SWE-Bench Verified and OTIS Mock (4,5,15,35,36).Source: Cross-benchmark alignment and paired… · p. 33
  • Matched-pair analyses compared base models with reasoning-augmented counterparts using SCC boundaries and token consumption at the stable boundary, plotting expenditure multipliers against ΔSCC in Fig. 5E.The analysis evaluated whether additional generation-time computation expanded stable capability.Source: Cross-benchmark alignment and paired… · p. 33

Matched dual task counting controls … Residual stream projections and targeted causal interventions

These sections describe matched dual-task, agent-workflow, dense-sweep, and mechanistic tests of exact counting. Causal interventions locate a shift in count representation from intermediate sequence tokens to the final response prefix.

Matched dual task counting controls

  • Matched prompts paired marker counts from 32 to 96 with benchmark-style questions, using length-matched code or independent-count distractors to isolate reasoning load from sequence length.Source: Matched dual task counting controls · p. 33

Agent-operation and externalization assay

  • The agent assay tracked six signed transactions across accounts A–C, flagged identifiers, net total, and next action, comparing agent-only, count-insertion, copy-only, and copy-then-agent conditions.Source: Agent-operation and externalization… · pp. 33–34
  • Using a seed-17 jittered ladder of 50 to 1000 markers, the assay scored JSON validity, exact counts and copies, and downstream-field correctness, with six samples per agent-bearing condition.Source: Agent-operation and externalization… · p. 34

Dense sweeps and motif perturbations

  • Dense actual-count sweeps around randomized-ladder SCC boundaries mapped exact-to-failure transitions and aggregated incorrect predictions into failed-output attractors (Fig. 2, D and E).Source: Dense sweeps and motif perturbations · p. 34
  • The study varied repeated characters and delimiters, comparing relative SCC with tokenizer-specific compression rates to distinguish token-volume from syntax-dependent trajectory effects (Fig. 2F).Source: Dense sweeps and motif perturbations · p. 34

Residual stream projections and targeted causal interventions

  • Residual-stream analyses fit one-dimensional sequence-position projections across lengths and layers in Gemma 3 27B-it and Qwen 3.5 35B-A3B, and used teacher-forced logit gaps to assess count preference.Source: Residual stream projections and… · p. 34
  • Causal interventions isolated a shift in count representation from intermediate sequence tokens, such as layer 31, to the final response prefix, such as layer 53.Source: Residual stream projections and… · p. 35
  • They tested whether a linearly decoded progress coordinate sufficed by shifting failed prompts along the learned counter direction while leaving orthogonal components unchanged, across 40 trials.Source: Residual stream projections and… · p. 35

Sparse autoencoders and feature coalitions · Supplementary Note 1: Mathematical Bounds on Guessing Exploits · Supplementary Note 2: Prompt Templates and Generation Settings

Sparse-autoencoder analyses examined count-correlated features and their stability under motif and syntax changes. Supplementary notes bound guessing performance under the fuzzy evaluation threshold and specify the benchmark prompts and generation settings.

Sparse autoencoders and feature coalitions

  • Gemmascope 2 decomposed residual representations into sparse feature dictionaries, which were ranked by Pearson correlation with count and tested for stability across motif and syntax perturbations.The analyses compared top-feature sets using Jaccard overlap and measured baseline-feature rank displacement under syntax changes.Source: Sparse autoencoders and feature… · p. 35

Supplementary Note 1: Mathematical Bounds on Guessing Exploits

  • With α = 0.20, the minimum relative error for a guess without token tracking is ≈10.1%, versus ≈10.2% for guessing the distribution center.The ±20% fuzzy boundary combined with relative MAE prevents a model from artificially minimizing error through a generalized prior guess.Source: Supplementary Note 1: Mathematical… · p. 36
  • At a 5% passing threshold, a guessing model has a 24.6% chance of passing one trial, but a 0.025% false-positive rate at 16 trials.For 16 trials, the guessing model's Z-score at the threshold is −3.48.Source: Supplementary Note 1: Mathematical… · p. 36

Supplementary Note 2: Prompt Templates and Generation Settings

  • The system instruction required an exact count and an integer-only response, while the user prompt asked how many occurrences of the labeled item appeared in the sequence.The prompt used {item_label} and {sequence} placeholders.Source: Supplementary Note 2: Prompt Templates… · pp. 36–37
  • For Google models, the preferred API path sent the system instruction separately and used the user prompt as the sole content block.If that path was unavailable, the harness used an inline Google prompt.Source: Supplementary Note 2: Prompt Templates… · p. 37
  • The shared harness used deterministic prompting, RNG seed 7, output caps of 4,096 and 32,768 tokens, and adaptive search beginning at length 32.Adaptive search used a 0.2 jitter fraction and 16 samples per length.Source: Supplementary Note 2: Prompt Templates… · p. 37
  • Adaptive search used 15 parallel trials per batch, 5% tolerance, and 10% steps; unparseable output after truncation triggered doubled token budgets up to 32,768 tokens.Retry logic was identical across provider-specific paths.Source: Supplementary Note 2: Prompt Templates… · p. 37

Supplementary Note 3: Full Model Catalogue

Supplementary Table S1 presents the full model catalogue and its measured Counting results, alongside model and context metadata. It also classifies architectures using published parameter metadata when available.

  • Table S1 catalogs models with their developer or family, parameter metadata, nominal context window, architecture class, and measured Counting results.Source: Table S1 · p. 38, Supplementary Note 3: Full Model… · p. 38
  • Architecture labels are inferred from published parameter metadata: distinct total and active parameter counts indicate MoE, a single count indicates Dense, and unavailable cases are marked “–”.Source: Table S1 · p. 38

Supplementary Note 4: Audit of Instruction-Following Failures

Beyond numerical miscounts, models above stable sequence capacity often lose instruction following, producing non-single-number responses, reasoning-loop failures, or unrelated outputs.

Supplementary Note 4: Audit of Instruction-Following Failures

  • Across 9,797 trials, 501 responses (5.1%) violated single-integer formatting, though the parser recovered integers in 305 (60.9%) of those cases.These failures affected 44 model variants: 286 cases from closed-source APIs and 215 from open-weight models. Responses were classified from raw text, even when the parser extracted a fallback integer.Source: Supplementary Note 4: Audit of… · pp. 41–42
  • The most common deviation was unsolicited step-by-step counting or explanation, followed closely by blank or whitespace outputs.The audit also included responses with formatting artifacts or no number at all.Source: Supplementary Note 4: Audit of… · pp. 41–42
  • During adaptive ladder expansion, reasoning-optimized and coding models could exceed maximum output-token limits and overthink into infinite latent reasoning loops; gpt-5.2-codex had 17 null failures in 129 trials at L=256.Source: Supplementary Note 4: Audit of… · p. 42
  • After structural state collapse, models could regress to pretraining priors, hallucinate unrelated instruction sequences, or become contextually disassociated.Examples include deepseek-r1 generating an unprompted trace for unrelated calculus or algebra, gemini-3-flash-preview writing Python parsing code, and mistralai/devstral-2512 echoing the sequence indefinitely.Source: Supplementary Note 4: Audit of… · p. 43

Supplementary Note 5: Heterogeneous Nested Key-Path Tracking Assay

In a heterogeneous nested-key-path assay, most tested frontier models failed at the smallest target tier, and structural variable binding reduced the state-maintenance capacity observed in linear counting.

Supplementary Note 5: Heterogeneous Nested Key-Path Tracking Assay

  • The assay tests hierarchical depth tracking and multi-slot variable binding across semantic distractors, probing generalization of higher-layer abstraction.A record matches only when its KEY token equals the deepest leaf token nested inside its PATH.Source: Supplementary Note 5: Heterogeneous… · p. 45
  • Random distractors and cycling bracket geometries require parsing nested structure, checking each record’s KEY against its deepest PATH leaf, and summing matches.These features prevent reliance on one-dimensional token distances or homogeneous attention-sink dynamics.Source: Supplementary Note 5: Heterogeneous… · p. 45
  • The adaptive ladder treated L as the target match count, sampled x_k from U(0.8L, 1.2L), and shuffled positives with proportional negative distractors.The negative-record count was max(2, ⌈0.5 × x_k⌉ + 2).Source: Supplementary Note 5: Heterogeneous… · pp. 45–46
  • Models failing the minimal initial tier were categorically assigned a Counting Capacity (CC) of < 32.The assay bypassed binary search for these models because they lacked a stable lower bound for interpolation.Source: Supplementary Note 5: Heterogeneous… · p. 46
  • At L=32 matches, the vast majority of frontier API models failed the initial tier, and structural variable binding ablated linear-assay state-maintenance capacity (Table S5).The result supports a generalized fragility in higher-layer abstraction rather than a failure specific to simple token counting.Source: Table S5 · p. 46, Supplementary Note 5: Heterogeneous… · p. 46

Supplementary Note 6: Anatomy of Latent Reasoning Breakdowns

The exposed reasoning traces examine how models break down when counting longer sequences, despite being able to describe valid counting algorithms. They document failures including drift, familiar-schema substitution, boundary errors, miscounts, malformed or derailed output, and failure to answer.

Outcome transition

Rounded-attractor failures

  • At L=96, 100 was the most common coherent wrong answer, appearing in 7 records; 12 of 13 wrong-integer records mentioned 100, consistent with a rounded attractor after exact state was lost.The analysis describes this as replacing the sequence boundary with a salient rounded value.Source: Supplementary Note 6: Anatomy of… · p. 47

Reasoning length and outcome

  • At L=96, median exposed reasoning was 1,885 words for wrong-integer responses and 4,292 words for responses with no final integer; correct responses were not the longest.Four of the five correct records exposed no separate reasoning fragments, and longer scratchpads did not restore counts after prompt-derived state collapsed.Source: Supplementary Note 6: Anatomy of… · pp. 47–48, Table S7 · p. 48

Failure traces and interpretation

  • Long exposed traces commonly repeat recounting, copy the prompt, imagine grouping, or exhaust the token budget, and may return no meaningful estimate.These traces are behavioral evidence, not guaranteed-faithful records of internal computation; some failures spend thousands of exposed tokens before an incorrect or absent answer.Source: Supplementary Note 6: Anatomy of… · pp. 47–48

Boundary binding

  • Successful L=96 traces preserve the terminal remainder, whereas failed traces may substitute a 10-by-10 schema yielding 100 or lose the boundary and return 93, 98, or 120.Successful groupings include nine tens plus six, six groups of sixteen, or 95 separators plus one; the decisive feature is anchoring the final partial group, not base-10 grouping itself.Source: Supplementary Note 6: Anatomy of… · p. 48

C04: Early local count drift or loop

  • At length 32, some models drift to a wrong local count or loop despite applying a valid counting rule, and such failures precede length-96 collapse.The shared pattern is local state drift, not task misunderstanding.Source: Supplementary Note 6: Anatomy of… · pp. 51–52

C05: Rounded 100 attractor

  • At length 96, a common coherent failure is counting 100 by imposing a familiar 10-by-10 schema and continuing past the list’s endpoint.This rounded-base-10 attractor differs qualitatively from a small off-by-one error.Source: Supplementary Note 6: Anatomy of… · pp. 52–53

C06: Near-boundary coherent drift

  • At length 96, coherent near-boundary errors include answers of 98 and 93; these are real attempted counts but still have the wrong terminal boundary.These cases show subtler state-tracking errors than the 100 attractor.Source: Supplementary Note 6: Anatomy of… · p. 53

C07: Enumeration overflow or large overcount

  • At length 96, one model answers 120 after lengthy explicit enumeration that continues beyond the true sequence boundary, showing that more reasoning need not improve state tracking.The trace is long and explicit, but its count boundary drifts beyond the prompt.Source: Supplementary Note 6: Anatomy of… · pp. 53–54

C08: Repetitive reasoning loop, no answer

  • Some models produce no final integer because repetitive verification loops do not convert a repeated counting plan into a terminated answer.The same model’s no-answer loop occurs at both lengths 32 and 96, so this failure is not limited to the longer boundary.Source: Supplementary Note 6: Anatomy of… · pp. 54–55

C09: Prompt-copy or runaway expansion

C10: Malformed visible generation or incidental parse

  • In C10, visible generation degenerates into copied sequence fragments or punctuation, leaving no answer or an incidental, meaningless parsed number.Examples include parser returns of 10 and -1 that are not meaningful estimates.Source: Supplementary Note 6: Anatomy of… · pp. 55–56

C11: Task derailment after reasoning limit

  • In C11, reasoning reaches its length boundary and the visible answer switches from the counting task to an unrelated algebra problem.The representative L = 96 record has parsed answer 14.Source: Supplementary Note 6: Anatomy of… · p. 56

Overall pattern

  • Models generally describe valid counting algorithms but fail when applying them to the real sequence.Some traces spend thousands of tokens trying to verify the algorithm but still cannot individuate the list’s terminal characters.Source: Supplementary Note 6: Anatomy of… · p. 56

Length-dependent breakdowns

  • At L = 32, failures are usually small miscounts; at L = 96, many models jump to attractors such as 100 or expand the sequence and fail to terminate.Some models elaborate a rounded or copied substitute rather than reconstructing the missing count once per-item state is no longer decoder-usable.Source: Supplementary Note 6: Anatomy of… · pp. 56–57

Supplementary Note 7: Matched Dual-Task Counting Controls

Matched dual-task tests show that complex reasoning interferes with exact counting more than length-matched irrelevant text or code, supporting the finding that shared internal state tracking is consumed by reasoning rather than raw context length.

Supplementary Note 7: Matched Dual-Task Counting Controls

  • The evaluation tested gpt-5.4-mini on counts from 32 to 96 with matched secondary tasks drawn from BBH, CRUXEval-O, MATH-500, and MMLU-Pro.Prompts required a JSON response with both the parsed count and secondary-task solution; MMLU-Pro mathematics and computer-science categories were excluded.Source: Supplementary Note 7: Matched… · p. 57
  • Format-matched controls inserted irrelevant valid code or a secondary sequence of ‘b’ markers sized to match each benchmark prompt’s estimated token length.These controls were designed to separate active semantic execution from raw context-length degradation.Source: Supplementary Note 7: Matched… · p. 57
  • Plain counting and length-matched irrelevant-text/code controls had minimal error through count 60, whereas MATH-500, BBH, and CRUXEval-O tasks caused earlier, larger systematic undercounting.Main Text Fig. 2F plots mean parsed-count error by true count, smoothed with a five-point centered window.Source: Supplementary Note 7: Matched… · p. 57
  • Dual-counting degraded accuracy less severely than complex logical reasoning, consistent with finite shared structural representations being consumed by reasoning independently of raw token length.Supplementary Fig. S11 shows raw parsed counts for all conditions.Source: Supplementary Note 7: Matched… · p. 57

Supplementary Note 8: Agentic Operation and Counting

The agent-operation assay tested whether models could preserve a count-bearing state while completing a deterministic transaction task. Both models lost exact counting or copying, but the effects on their surrounding responses differed.

Supplementary Note 8: Agentic Operation and Counting

  • The assay paired a three-account, six-transaction update task with marker counting, exact sequence copying, or both to distinguish count and copy failures from operational-task failures.The task also required preserving review-transaction identifiers, computing the net total, and choosing one of three next actions.Source: Supplementary Note 8: Agentic… · p. 58
  • gpt-5.4 preserved outer JSON and nearly solved transaction updates, yet exact marker counting and copying failed at shorter lengths while the surrounding operational state remained intact.After the exact-copy boundary, copied sequence lengths and reported counts drifted toward values such as 200, 400, 500, and 1000.Source: Table S9 · p. 58, Supplementary Note 8: Agentic… · pp. 58–59
  • claude-sonnet-4-6 mostly maintained transaction accuracy in count-insert trials but often violated JSON-only formatting, then produced unparsable or empty refusal-like responses after its last successful copied sequence.Both wrong copied files and empty refusals break the operational pipeline.Source: Table S9 · p. 58, Supplementary Note 8: Agentic… · p. 59
  • Exact externalization did not bypass the state-preservation problem: both models lost exact copying, and coupling it with the agent task destabilized claude-sonnet-4-6’s structured response.The threshold summary treats copy-only rows as instability sweeps because they used one trial per marker level, not stable threshold estimates.Source: Table S10 · p. 59, Supplementary Note 8: Agentic… · pp. 59–60

Supplementary Note 9: Additional Results on Gemma Model

Gemma’s counting behavior depended on the exact repeated-character motif and delimiter, while causal interventions showed that count directions controlled outputs within the stable regime. The intervention profiles also indicate that count information is accumulated in sequence representations and consolidated into a causal response-prefix representation, but low-dimensional control cannot restore a collapsed state.

Additional Results on Gemma Model

  • Delimiter substitutions also changed CC limits: a space yielded 11, | yielded 16, and section yielded 18.The task remained mechanically identical despite the changed character or delimiter.Source: Supplementary Note 9: Additional… · p. 60
  • Repeated-character motifs had different CC limits: b counted exactly to 26, whereas han counted exactly to 36.CC limit is the last exact count before the first observed failure for each repeated-character motif.Source: Table S11 · p. 60, Supplementary Note 9: Additional… · p. 60
  • Sequence-token linear patching along the Layer 31 count direction modulated generated counts for target lengths up to 26.The direction was fitted from successful homogeneous-counting sequences.Source: Supplementary Note 9: Additional… · p. 61
  • Adding or subtracting the learned count direction at the final prompt token before Layer 31 decoding yielded robust linear shifts in output value.This final-token steering intervention manipulated the learned count direction directly.Source: Supplementary Note 9: Additional… · p. 61
  • Final-token donor patching was inert at layers 16, 31, and 40 but exerted significant overriding control at resid_post.layer53.This targeted intervention replaced the final assistant-prefix token with the corresponding donor state.Source: Supplementary Note 9: Additional… · p. 61
  • Sequence-token donor patching made resid_post.layer31 track donor counts through the middle range before saturating, while late layers had no decoded effect.The interventions indicate that exact count information accumulates across distributed sequence representations before being consolidated into a causal response-prefix representation at generation onset.Source: Supplementary Note 9: Additional… · p. 61
  • The extracted count coordinate controlled behavior within the stable regime but could not restore a collapsed high-dimensional feature coalition beyond the stability boundary.The supplementary note attributes this limitation to counter-direction clamping failures.Source: Supplementary Note 9: Additional… · p. 61

Supplementary Note 10: Causal Counter-Direction Clamping Analysis

Counter-direction clamping of a linearly decodable count axis did not rescue over-capacity sequences and instead disrupted their activations. The findings reject the 1D causal hypothesis and point to state maintenance by a distributed, nonlinear feature coalition.

Supplementary Note 10: Causal Counter-Direction Clamping Analysis

  • On gemma-3-27b-it, researchers clamped failed sequences at layers 16, 31, 40, and 53 toward exact-prefix extrapolated coordinates for target counts 27 to 160, beyond counting capacity.The intervention shifted residual states along the 1D progress axis.Source: Supplementary Note 10: Causal… · p. 62
  • 0 of 40 targeted patching trials rescued exact generation, and clamping decreased the minimum correct-logit gap by an average of -0.256.The analysis reports these results as rejecting the 1D causal hypothesis.Source: Supplementary Note 10: Causal… · p. 62
  • Early-layer clamping magnified subsequent projection error at Layer 53, with error expansion up to -875 times at L=27.This was reported for clamping the 1D state at early layers such as Layer 16 and 31.Source: Supplementary Note 10: Causal… · p. 62
  • Artificial translation along the 1D count axis pushed activations off the valid data manifold, consistent with state maintenance relying on a highly distributed, nonlinear feature coalition.The analysis describes latent-space collapse as structural disintegration of that coalition, which cannot be causally rescued by rectifying its 1D linear projection.Source: Supplementary Note 10: Causal… · p. 62

Supplementary Note 11: Qwen MoE Latent Tracking Replication

The Qwen MoE replication found linearly readable count directions and bounded steering effects, alongside degradation beyond a stable sequence-length boundary. These findings support bounded state tracking and collapse beyond dense Gemma models.

Supplementary Note 11: Qwen MoE Latent Tracking Replication

  • The Qwen replication extends bounded, linearly readable state tracking and abrupt structural collapse beyond dense Gemma to a sparse MoE architecture.The note characterizes this as a generalized phenomenon across modern transformer paradigms.Source: Supplementary Note 11: Qwen MoE Latent… · p. 62

Supplementary Figures

  • Adaptive-search traces show stable regions and failure boundaries across closed and open models; in the heterogeneous nested assay, bounded capability appeared only in gpt-5.4.The heterogeneous assay caption describes early, catastrophic collapse across the majority of evaluated models.Source: Figure S1 · p. 64, Figure S4 · p. 67, Figure S8 · p. 70
  • Clamping the state failed to improve the decoder’s preference for the true count and often reduced the logit margin; earlier-layer patches caused severe out-of-distribution corruption, including -875.The clamping results indicate that the abstract state is distributed and nonlinear at its capacity limit.Source: Figure S10 · p. 71, Supplementary Figures · p. 71, Figure S9 · p. 71
  • Additional Gemma controls show that count directions produce interpretable count shifts in some interventions, while other interventions move the state off-distribution.The panels include sequence-token patching, final-token steering and donor patching, and counter-direction clamping at multiple layers.Source: Figure S12 · p. 73, Supplementary Figures · p. 73

Supplementary Note 11: Qwen MoE Latent Tracking Replication

Supplementary Figures

  • Benchmark-correlation panels indicate that Counting Capacity captures capability axes orthogonal to standard task leaderboards and aligns weakly with their metrics.The panels map pairwise correlations between Counting Capacity and established benchmarks and summarize rank correlations across public benchmark metrics.Source: Figure S14 · p. 75, Supplementary Figures · pp. 75, 77, Figure S15 · p. 76

Related papers

Loading 2605.02028v2…