PaperlayerView exact revision on arXiv

Source-linked AI summary

Staying on Task: Testing the Foundations of Long-Horizon Agent Reliability

Jeffrey Willette, Krishna C. Puvvada, Boris Ginsburg

arXiv:2609.38712v1cs.AI

TL;DR

Benchmarks of long-horizon workflows measure whether tasks succeed, but do not isolate sustained execution failures. Long-Transduction tests repeated context-dependent operations while varying context length, local difficulty, and input format. Across models, accuracy falls substantially as context grows, and the authors conclude that nominal capacity does not ensure reliable exhaustive execution.

  • Problem

    Existing benchmarks measure overall task success but do not isolate sustained-execution failures, such as losing alignment while processing a long ledger.

    Source: §1 · p. 1

  • Method

    Long-Transduction evaluates repeated arithmetic, sorting, lookup, and table operations with controlled variation in task difficulty, input format, and context length.

    Source: §2 · p. 2

  • Results

    62.8% relative decrease in performance averaged over models when scaling context from 4K to 128K nominal tokens.

    Source: §4 · p. 3

  • Takeaways & Limitations

    Nominal context capacity does not certify reliable completion; the authors recommend stable IDs, chunked outputs, and smaller, less complex task units for current long workflows.

    Source: §5 · p. 4

    The synthetic deterministic benchmark does not test planning, tool selection, multi-turn interaction, recovery, permissions, human escalation, or direct safety behavior.

    Source: E Limitations · p. 8

Abstract

from arXiv · show

Long-horizon agentic workflows require models to sustain repeated state-dependent actions all while the context grows, sub-task complexity changes, and new data arrives. Each situation represents an independent axis along which an agent may fail. An agent reconciling a long ledger, for example, must repeatedly read its state, update the correct record, and preserve alignment across thousands of outputs. A model may accept the entire ledger yet lose its place or stop applying the operation consistently as generation proceeds. We introduce Long-Transduction, a controlled diagnostic that tests a model's ability to stay on task during long generation while continuously reading, mutating, and outputting input-context dependent operations such as arithmetic, sorting, variable lookups, and table transformations. Long-Transduction evaluation independently varies local task complexity, input data formatting, and context length isolate failures along each axis. We evaluate seven open-weight models, finding a 62.8\% decrease when scaling context length from 4-128K, a 36.5\% decrease when varying input format, and a 39.9\% decrease by increasing local task complexity. Together, these failures represent critical liabilities in long-horizon agentic workflows.

1 Long-horizon Execution Needs Its Own Diagnostic

Long-horizon benchmarks measure task success but do not isolate whether a model can sustain correct state-dependent execution. Long-Transduction tests this repeated read, mutate, and output loop with item-level scoring and controlled variation.

1 Long-horizon Execution Needs Its Own Diagnostic

  • Existing benchmarks measure overall success across real-world questions, web interactions, code repair, and tool-mediated goals, but do not isolate sustained-execution failures.A ledger can be wrong because of retrieval, computation, omission, or losing alignment with the correct record.Source: §1 · p. 1
  • Long-Transduction makes long-horizon retrieval and transformation a long stateful output trajectory, using elementary operations so every required item can be scored.This complements long-context benchmarks that produce answers short relative to their input context.Source: §1 · p. 1
  • The diagnostic independently varies context length, local difficulty, and input format, with per-record and position-resolved scoring across seven models.The evaluation shows that supported context length does not certify reliable completion.Source: §1 · p. 1

2 Long-transduction

Long-Transduction tests repeated input-dependent operations across task families, formats, difficulty settings, and context horizons. Its scoring measures exact record accuracy while weighting task-difficulty cells equally.

2 Long-transduction

  • Four task families test line-by-line arithmetic, UUID sorting, variable lookup, and table transformations, with difficulty varied within each family.Difficulty changes operand or item counts, dictionary size, or table-transformation complexity.Source: §2 · p. 2
  • Three formats vary position tracking: ordered records with numeric IDs, shuffled IDs requiring ordered output, and unnumbered records tracked by context position.Without IDs, an omission can shift subsequent output positions.Source: §2 · p. 2
  • The evaluation comprises 1,440 documents per model across task-format-difficulty combinations, six context budgets, and five document samples.Nominal input-plus-output horizons range from 4K to 128K tokens; the largest tier averages 65,374 input and 61,240 required output tokens.Source: §2 · p. 2
  • Item accuracy is the fraction of required output records exactly correct; records are averaged within documents, then task-difficulty cells receive equal weight.Documents, rather than individual items, are the independent scored units unless otherwise specified.Source: §2 · p. 2

3 Open-weight Model Evaluation

The evaluation compares open-weight model checkpoints on a common task grid, with one checkpoint constrained by its context limit. Figures describe performance across context horizons, tasks, and input formats.

3 Open-weight Model Evaluation

  • Eight open-weight checkpoints are tested, with seven spanning the full grid; Olmo stops at 32K nominal tokens because of its context limit.Closed-weight models are omitted from quantitative comparisons after pilot runs in which they often declined, terminated early, or requested clarification.Source: §3 · p. 2
  • Figure 2a reports accuracy by context length averaged across tasks, formats, and difficulties; Figure 2b shows individual task performance at 128K.Olmo ends at 32K in the figure because of its context limit.Source: Figure 2 · p. 3
  • Figure 3 compares input-format performance at 128K nominal tokens, averaged over four difficulty settings.The caption notes that removing row IDs removes a stable lookup key and that ordered transformations without IDs score worse than shuffled numbered rows.Source: Figure 3 · p. 3

4 Results

The results show reliability losses across context length, input format, local task complexity, and item-by-item generation. These patterns indicate that nominal context capacity alone does not ensure reliable exhaustive execution.

4 Results

  • 62.8% relative decrease in performance averaged over all models when scaling context from 4K to 128K.DeepSeek overall accuracy falls from 0.909 at 4K to 0.554 at 128K.Source: §4 · p. 3
  • At 128K, exact completion is 41/240 for DeepSeek (17.1%), 11/240 for Qwen-122B (4.6%), and 2/240 for Qwen-35B (0.8%).At the same horizon, DeepSeek reaches 0.961 on arithmetic but 0.262 on table transformation; Qwen-122B reaches 0.952 on arithmetic, 0.233 on UUID sorting, and 0.170 on table accuracy.Source: §4 · p. 3
  • Format changes reduce average performance by 16.2% from ‘Ordered + IDs’ to ‘Ordered, No IDs’; the no-ID variant reduces UUID sorting by 64.3% and variable lookup by 66.4%.Position tracking therefore depends on the task being performed.Source: §4 · pp. 3–4
  • Every non-degenerate model declines as local work increases from easiest to hardest, and Figure 4 shows overall accuracy decreasing with complexity at 128K.The Figure 4 metrics are averaged over input format; Olmo data extend only to 32K.Source: §4 · p. 4, Figure 4 · p. 4
  • For UUID sorting, self-consistency is significantly higher than per-position accuracy, indicating that models can follow the operation while failing to read the correct input from context.Figure 5 examines scored items throughout generation for the ‘Ordered, no IDs’ format, averaged over context lengths and difficulties.Source: §4 · p. 4, Figure 5 · p. 4

5 Conclusion

The conclusion identifies context length, input format, and local difficulty as failure sources for long-horizon agents. It recommends stable IDs, checkpointing output chunks with IDs, and splitting complex work into smaller tasks.

5 Conclusion

  • Long-horizon workflows should use stable input IDs, checkpoint output chunks with IDs, and split tasks into small, less complex units.The paper also calls for future models to mitigate these failure modes during training.Source: §5 · p. 4

A Benchmark details and examples

The benchmark tests arithmetic, UUID sorting, variable lookup, and CSV table transformations, with difficulty and input structure varied across tasks.

A Benchmark details and examples

  • Arithmetic expressions use ordered, shuffled, or unnumbered records, with difficulty set by 2, 4, 8, or 16 terms per summand.Shuffled records require ascending output indices; unnumbered records require tracking position without an index.Source: A Benchmark details and examples · p. 6
  • UUID sorting requires sorting each hexadecimal-identifier list, with 2, 4, 8, or 16 items per list and ordered, shuffled, or no-ID formats.A record is correct only when the emitted identifiers exactly match the expected sorted sequence.Source: A Benchmark details and examples · p. 6
  • Variable lookup resolves expressions against a definition pool, whose size ranges from 8 to 256 independently of output-record count.It uses ordered, shuffled, and no-ID variants.Source: A Benchmark details and examples · p. 6
  • CSV tasks test row/column permutations or key-value resolution, varying permutation extent and the number of adjective and noun keys.Permutation inputs use homogeneous four-digit integers or variable-length UUID fragments; key-value resolution preserves CSV structure while replacing expressions with resolved phrases.Source: A Benchmark details and examples · pp. 6–7

B Scoring

Scoring checks exact output records, averages correctness across documents and settings, and distinguishes record accuracy from complete-document success.

B Scoring

  • A required output record is correct only when it exactly matches its expected answer.Records include output rows or CSV cells.Source: B Scoring · p. 7
  • Record correctness is averaged within each document and then equally across tasks, difficulty, format, or context length.For numbered streams, IDs match outputs to expected records; for unnumbered streams, non-empty lines are matched ordinally, so an omission can shift later matches.Source: B Scoring · p. 7
  • Exact document completion is one only when every required record is correct; its 128K rate averages this indicator over 240 documents.Source: B Scoring · p. 7

C Additional results

Figure 8 reports results for every task and input format at the 128K context length.

C Additional results

D Reproducibility and Release

The reproducibility section specifies the benchmark grid and inference settings, and promises release of the generator, verifier, and scripts upon acceptance.

D Reproducibility and Release

  • The frozen grid spans 4 tasks, 3 formats, 4 difficulty settings, 6 input-plus-output horizons, and 5 samples, totaling 1,440 documents.Source: D Reproducibility and Release · p. 7
  • Seven primary models cover the identical grid, while Olmo is limited to four tiers through 32K by its context limit.At the largest tier, preparation estimates mean prompt and required-output lengths of 65,374 and 61,240 tokens, respectively.Source: D Reproducibility and Release · p. 8
  • Inference uses greedy sampling at zero temperature, with explicit reasoning disabled when supported and otherwise set to the lowest available effort.Outputs may use up to 1.5 times the target input budget; the reported response-usage records contain zero reasoning tokens.Source: D Reproducibility and Release · p. 8
  • The authors will release a fully seeded data generator, exact verifier, and evaluation scripts upon acceptance.Source: D Reproducibility and Release · p. 8

E Limitations

LongTransduction is a synthetic, deterministic diagnostic rather than a realistic enterprise environment, and it omits several agent capabilities and safety behaviors. Its positional scoring also makes omissions cascade, measuring format fragility alongside local competence.

E Limitations

  • The synthetic, deterministic setting excludes planning, tool selection, multi-turn interaction, recovery, permissions, human escalation, and direct safety behavior, and is not a realistic enterprise environment.The records resemble enterprise data-processing primitives, but the setting is described as mimicking fundamental primitives rather than an enterprise environment.Source: E Limitations · p. 8
  • Ordinal scoring of unnumbered tasks makes omissions cascade by design, so the benchmark measures format fragility as well as local competence.Source: E Limitations · p. 8
  • At 128K, Figures 6–8 show local-difficulty curves and task/model performance; Figure 7 groups by format and task, while Figure 8 contrasts absolute performance with difficulty sensitivity.Olmo* has no 128K point in Figures 6 and 7, and Figure 8 marks its missing cells with gray dashes.Source: Figure 6 · p. 9, Figure 7 · p. 10, Figure 8 · p. 11
  • Figure 9a corresponds to Figure 6, while Figure 9b corresponds to Figure 3.Source: Figure 9 · p. 11

Related papers

Loading 2609.38712v1…