Long-Horizon State Tracking in LLMs: Executing MD5 through a Deep Sequence of Dependent Tool Calls

Motivation

• Long deterministic procedures challenge LLMs because dependent-step errors compound and quality degrades with long contexts, prompting the use of hand-authored controllers.The introduction attributes error compounding and self-conditioning to Sinha et al. 2025, and context-length degradation to Liu et al. 2024 and Modarressi et al. 2025.Source: §1 · p. 1

Benchmark

• The benchmark makes long-horizon exactness measurable with a from-scratch MD5 implementation, a canonical 196-call trace, and bit-level digest checks.The procedure maintains four 32-bit words across 64 rounds, with each step depending on the previous state.Source: §1 · p. 2

Driver/worker setup

• In the strongest configuration, a driver and worker LLM compute the hash from scratch, while the CPU implementation verifies values but never supplies them.Each primitive operation is replaced by a worker LLM that performs the bitwise functions, modular addition, or rotation.Source: §1 · pp. 1–2

Results

• A ∼5.5B-active-parameter gpt-oss-120b carries state across all 196 calls and returns the correct digest on a majority of completed runs.The model runs at temperature 0 with a short fixed prompt; the inputs are non-memorized.Source: §1 · p. 2

Success conditions

• Success depends on keeping the driver’s reasoning in context and majority-voting the arithmetic worker, without changing model weights.The paper separates driver state tracking from worker arithmetic and characterizes residual failures by driver state, worker arithmetic, and serving.Source: §1 · p. 2

• At 196 dependent steps, p=0.99 implies ~14% end-to-end success and p=0.95 implies ~4 × 10−5 (Sinha et al. 2025).Sinha et al. also report that models degrade when conditioning on their own prior outputs, while larger models and chain-of-thought extend but do not remove the horizon limit.Source: §2 · p. 2
• Long-context retrieval and reasoning degrade through positional and length effects, with many models below half their short-context score by ~32k tokens (Liu et al. 2024; Modarressi et al. 2025).Function-calling accuracy also degrades specifically in long contexts (Adhikary, Contractor et al. 2025).Source: §2 · p. 2
• MD5 maps a message to a 128-bit digest through 64 rounds per block, using four 32-bit state words and modular addition and rotation.In the harness, one block requires 196 tool calls: two setup calls, three per round, and two finalization calls.Source: §2 · p. 2
• The 196-step MD5 procedure is strictly dependent and cannot be guessed or interpolated, making it a clean probe of sustained exact execution.Its 32-bit modular additions with carries and bit rotations also stress arithmetic on which LLMs are brittle (Lee et al. 2024); reasoning-in-context, self-consistency, and a driver/worker split are the conditions the study investigates.Source: §2 · p. 2

Long-horizon execution and long context

• Prior work reports compounding errors (Sinha et al. 2025) and context or tool degradation (Liu et al. 2024; Modarressi et al. 2025; Adhikary, Contractor et al. 2025); the 196-step transcript exercises both.The cited work identifies a positional “lost in the middle” effect, degradation past roughly 32k tokens, and tool-calling degradation in long contexts.Source: §3 · pp. 2–3

Reasoning traces and exact arithmetic

• Scratchpads (Nye et al. 2022) and chain-of-thought externalize intermediate work; the worker path addresses magnitude-sensitive arithmetic (Lee et al. 2024) with primitive-level self-consistency (Wang et al. 2023).The paper describes its reasoning-channel result as a serving-time analogue of scratchpads and chain-of-thought.Source: §3 · p. 3

Tool-using agents and self-managed memory

• ReAct (Yao et al. 2023) and Toolformer (Schick et al. 2023) leave workflow state implicit in context; self-editing memory such as MemGPT (Packer et al. 2023) motivates a bit-checkable test of exact state preservation.The MD5 harness tests whether self-management preserves exact state across a long horizon.Source: §3 · p. 3

4.1 Models and configuration

• Both driver and worker use gpt-oss-120b, an openweight mixture-of-experts model with ∼117B total and ∼5.1–5.5B active parameters per token.All endpoints serve the model with natively MXFP4-quantized MoE expert weights (∼4.25 bits).Source: §4.1 · p. 3
• The driver uses temperature 0, a short fixed prompt, and context caching; in swap mode, the worker uses temperature 0.7 with 3-sample majority voting.Runs use multiple non-memorized single-block inputs; endpoints and dates are logged because quantized serving at temperature 0 is not bit-deterministic, so results are treated as distributions and reported with N.Source: §4.1 · p. 3

4.2 Verification and metrics

• Call-by-call ground-truth alignment identifies state divergences, while injected first divergences yield counterfactual digests that separate a fresh error from downstream propagation.Live checks flag the first state slip by round index, avoiding misalignment from benign duplicate lookups.Source: §4.2 · p. 3
• Worker-computed primitives are checked against CPU twins for measurement, not correction; wrong results can be cascaded or corrected in place to distinguish end-to-end damage from failure attribution.Metrics include exact-digest success, first-divergence round, calls-to-completion versus 196, per-operation worker pass rate with and without voting, and latency; requests, reasoning, and outcomes are logged as structured JSONL and indexed in SQLite.Source: §4.2 · pp. 3–4

5.1 State-carrying driver

• The seven exact CPU primitives and stateful driver complete the canonical 196-call sequence and return a verified correct digest on multiple distinct inputs.Failed traces turn red when the driver supplies a mis-carried word; common errors include a rotation slip, dropped or duplicated calls, and early stops.Source: §5.1 · p. 4

5.2 Removing the arithmetic oracle

• With three-sample majority voting, a driver and stateless LLM workers compute the hash across 196 calls without an exact-arithmetic oracle in the loop.The CPU twin records pass/fail results but does not supply operation values; voting suppresses worker arithmetic slips.Source: §5.2 · p. 4, Figure 2 · p. 4

5.3 What makes it work

• Keeping the model’s reasoning channel in context enables state tracking across 196 calls; stripping it or moving identical text to the visible message causes early failure.Among 105 logged sessions, 71 produced no digest, and nearly half of those halted within the first 24 calls.Source: §5.3 · pp. 4–5
• Increasing worker samples further reduces residual arithmetic errors, while endpoint choice also affects whether multi-turn runs continue correctly and how quickly they proceed.The reasoning-channel input behavior varies across serving routes; a fast reasoning endpoint can loop, stop early, or rarely emit a digest.Source: §5.1 · p. 4, §5.3 · p. 5
• The 40 →48 skip recurs in seven long failing runs and none of the successful runs, making it a position-locked candidate model fingerprint rather than a random slip.Some runs also skip rounds 24–31; the authors characterize the pattern as a candidate diagnostic, not a universal behavior.Source: §5.3 · p. 5, §5.4 · p. 5, Table 2 · p. 5

5.4 Failure analysis

• Failures partition into driver state, worker arithmetic, and serving context errors; driver rotation and call-sequencing mistakes dominate, while worker slips are usually corrected by voting.Correcting worker operations alone does not rescue runs in which the driver mis-carries state, showing the error sources are independent.Source: §5.2 · p. 4, §5.4 · p. 5, Table 2 · p. 5

Practical reach

• LLMs may drive workflows such as accounting and compliance when context is managed, reasoning stays in context, and arithmetic is made reliable by voting.The authors contrast this with the prevailing assumption that LLMs should be confined to leaf tasks under a hand-built controller.Source: §6 · p. 5

Toward a self-managing agent

• The same-model driver/worker split is scaffolding toward one agent that carries long-horizon state, computes its own primitives, and curates its context.The driver owns state and sequencing while the worker handles bounded, verifiable computation; collapsing them into one agent is proposed as a next step.Source: §6 · p. 5

Self-editing context

• A self-editing context tool could write and read durable scratch, compress transcripts, and drop unneeded material, with the task checking state bit by bit after each edit.The proposal connects to MemGPT (Packer et al. 2023) and frames a ground-truthed testbed for exact state management; the authors expect scratch-memory use to improve robustness as transcripts grow and state becomes larger and messier.Source: §6 · pp. 5–6

7 Conclusion

• With ∼5.5B active parameters, the model computed an MD5 digest across 196 dependent tool calls, including arithmetic from scratch with a self-consistent worker replacing every primitive.The conclusion presents this as a positive answer to whether an LLM can carry out a deterministic algorithm step by step.Source: §7 · p. 6
• Success turned on two context-level levers, not changes to model weights: keeping the model’s reasoning in context and voting to make arithmetic reliable.Source: §7 · p. 6
• Residual failures were few and mechanical, with the 40 →48 skip providing a reproducible signature of the model.Source: §7 · p. 6
• The conclusion argues that models can drive long, exact workflows when given the needed context, rather than requiring hand-built controllers to take over.It frames this as a foundation for agents that may eventually manage that context themselves.Source: §7 · p. 6