PaperlayerView exact revision on arXiv

Source-linked AI summary

MiRL: Error-Targeted Reinforcement Learning for Correcting Terminal Agent Misbehaviors

Runchu Tian, Zaid Khan, Zun Wang, Justin Chih-Yao Chen, Mohit Bansal

arXiv:2610.35697v1cs.LGcs.AIcs.CL

TL;DR

Terminal agents tackle long-horizon shell tasks, but existing error-targeted methods focus on single-turn problems and failure analyses are mainly used for evaluation and debugging. MiRL profiles recurring misbehaviors and trains from their prefixes; it reduces measured misbehaviors and matches or exceeds TMax-4B on two benchmarks using fewer rollouts.

  • Problem

    Existing error-targeted methods are designed for single-turn tasks, while terminal-agent failure analyses are mainly used to evaluate and debug rather than train agents.

    Source: §1 · pp. 1–2

  • Method

    MiRL profiles policy errors to find recurring misbehaviors, then trains from their prefixes and rewards rollouts only when tests pass and the targeted behavior is avoided.

    Source: §1 · p. 2

  • Results

    On unseen prefixes, MiRL reduces measured misbehaviors by 43% to 57% on average; it scores 51.8% vs. 47.4% on Terminal-Bench Lite and 21.3% vs. 20.6% on Terminal-Bench 2.1, using 35% of TMax-4B’s rollouts.

    Source: §1 · p. 2, §4.3 · p. 6

  • Takeaways & Limitations

    The results suggest error-targeted learning can work on long-horizon terminal tasks when training targets generalizable misbehaviors where they occur.

    Source: §5 · p. 9

    The study trains and evaluates only one model backbone, trains each recipe once, relies on external LLMs for profiling, and discards prefixes from non-deterministic environments.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Error-targeted learning gives a legible way to improve a model: identify specific weaknesses of the model, train the model to fix these weaknesses, and check whether the weaknesses are fixed. We study how to apply error-targeted learning to terminal agents, which solve long-horizon tasks specified in natural language by interacting with a command shell. Existing error-targeted methods are designed for single-turn tasks, and existing analyses of agent failures are used to evaluate and debug agents rather than to train them. We propose Misbehavior-driven RL (MiRL), which first profiles the errors of a policy to find generalizable misbehaviors: kinds of wrong steps that recur across many environments. MiRL then trains the policy with RL, starting rollouts at the steps where these misbehaviors occur and rewarding a rollout only if it passes the tests and avoids the misbehavior. On Qwen3.5-4B, eight misbehaviors account for 52.1% of wrong steps. After training with MiRL, the policy repeats the measured misbehaviors 43% to 57% less often on average on unseen prefixes. MiRL outperforms TMax-4B on Terminal-Bench Lite (51.8% vs. 47.4%) and matches it on Terminal-Bench 2.1 (21.3% vs. 20.6%), while using 35% of its rollouts. These results show that error-targeted learning can be applied to realistic, long-horizon terminal tasks.

1 Introduction

The section motivates error-targeted learning for terminal agents, where recurring wrong steps must be identified and addressed within long-horizon tasks. MiRL profiles those errors, trains from misbehavior prefixes, and improves measured behavior and benchmark performance.

1 Introduction

  • Prior agent-failure analyses mainly evaluate and debug agents, while terminal-agent training commonly scales the number of environments.The authors state that, to their knowledge, error-targeted learning had not been applied to terminal agents.Source: §1 · p. 1
  • Single-turn error-targeted methods do not directly address terminal agents’ long-horizon tasks, where one or two wrong steps can cause failure and task-specific errors may not generalize.Starting training from scratch also spends rollouts on steps the agent already performs correctly and may not reach the misbehavior.Source: §1 · p. 2
  • Eight error types cover 52.1% of wrong steps and appear in 75.4% of environments; MiRL trains from their prefixes and rewards test-passing rollouts that avoid the targeted behavior.Prefixes are retained only when replaying their commands in a fresh container reproduces every recorded output.Source: §1 · p. 2
  • MiRL scores 51.8% vs. 47.4% on Terminal-Bench Lite and 21.3% vs. 20.6% on Terminal-Bench 2.1 against TMax-4B, using 35% of its rollouts.On held-out prefixes, it also reduces measured misbehaviors by 43% to 57% on average.Source: §1 · p. 2

2 Related Work

Related work addresses terminal-agent data efficiency, failure analysis, and more targeted reinforcement-learning signals. MiRL differs by using recurring policy errors to choose rollout starts and shape rewards.

Terminal agent training and data efficiency

  • Terminal-agent data-efficiency work improves task or trajectory quality or selects learnable examples, whereas MiRL targets recurring policy errors.Examples include Terminal-Lego (Yang et al., 2026) and Terminal-World (Cheng et al., 2026).Source: §2 · pp. 2–3

Analyzing agent failures

  • Failure taxonomies and step-localization methods diagnose or debug agents; MiRL instead uses the identified misbehavior prefixes to drive training.The cited approaches include MAST, AgentErrorTaxonomy (Zhu et al., 2026b), Who&When (Zhang et al., 2025), and AgenTracer.Source: §2 · p. 3

RL from selected states and step-level supervision

3 Methodology

MiRL profiles policy errors to identify recurring misbehaviors, then trains from the prefixes where they occur. Its workflow localizes and groups errors, reconstructs prefix states, and rewards successful rollouts that avoid the targeted behavior.

3 Methodology

  • MiRL combines error profiling with RL from misbehavior prefixes, adding a behavior reward so rollouts must pass tests and avoid the targeted misbehavior.Error profiling runs the policy on environments, records and groups its error steps, and identifies recurring types.Source: §3 · p. 3

3.1 Preliminaries

  • A terminal environment provides a natural-language instruction, a shell-command container, and tests of the final container state; an error step is one the localizer marks problematic given its prefix.A generalizable misbehavior is an error type recurring across many environments rather than being tied to very few.Source: §3.1 · pp. 3–4

3.2 Error Localization

  • The error localizer flags a policy step likely to lead to failure, and a teacher supplies a corrected step that is checked before execution so the trajectory can continue.A rollout is discarded if the teacher cannot provide an accepted correction within three tries.Source: §3.2 · p. 4

3.3 Error Taxonomy Construction

  • An annotator proposes error types across batches, retains types recurring across batches, labels error steps, and keeps types covering a fixed share; a few types cover about half of error steps and recur across environments.The authors treat those recurring types as generalizable misbehaviors.Source: §3.3 · p. 4

3.4 Rollout from Misbehavior Prefixes

  • MiRL starts rollouts from recorded misbehavior prefixes, then reconstructs their container state by replaying commands in a fresh container and discards prefixes whose outputs differ.This rerun check addresses flaky environments where commands can produce different outputs; saving container snapshots at every step would require substantial storage.Source: §3.4 · pp. 4–5

3.5 Misbehavior-Aware Reward Design

  • The behavior judge checks the first m post-prefix steps, and the combined reward r = ro ∧ re is positive only when tests pass and the targeted misbehavior is avoided.The outcome reward alone checks whether the tests pass and cannot indicate whether the rollout avoids the misbehavior.Source: §3.5 · p. 5

4 Experiments

Experiments profile recurring errors across terminal environments, then test MiRL’s task performance, misbehavior reduction, and resource use. MiRL improves on same-scale baselines and uses fewer rollouts than TMax-4B, while ablations show that prefix starts and behavior rewards contribute to performance.

4.2 Misbehavior Analysis

  • The eight most frequent error types cover 52.1% of 12,746 profiled error steps, and at least one occurs in 75.4% of the 2,017 environments.The types appear in 6.9% to 46.7% of environments, while remaining errors form a long tail of environment-specific types.Source: §4.2 · p. 6

4.3 Main Results

  • MiRL scores 51.8% on TB-Lite and 21.3% on TB-2.1, versus TMax-4B’s 47.4% and 20.6%, respectively.It gains 4.4 points on TB-Lite and matches TMax-4B on TB-2.1.Source: Figure 5 · p. 6, §4.3 · p. 6
  • Against same-environment baselines, MiRL exceeds the best by 4.2 points on TB-Lite and 2.2 points on TB-2.1.The comparison covers TMax-189, PivotRL, and RaR+, each trained on 189 environments.Source: §4.3 · p. 6

4.4 Ablations

  • Starting rollouts at misbehavior prefixes adds 7.9 points on TB-Lite and 1.5 points on TB-2.1; adding the behavior reward adds 6.5 and 3.3 points, respectively.Environment selection alone lowers scores as the selected environments are harder, and increases groups with no learning signal from 27.3% to 42.9%.Source: §4.4 · p. 7

4.5 Analysis

  • On unseen prefixes, MiRL lowers three of four misbehaviors under Gemini 3.8 Flash and Qwen3.8-27B; false claims about tests drop from 26.7% to 8.1% and 29.8% to 11.7%, respectively.Source: §4.5 · p. 7, Figure 7 · p. 7
  • Up to its best checkpoint, MiRL uses 35% of TMax-4B’s rollouts, 39% of its GPU hours, and 56% of its total cost while scoring 4.4 points higher on TB-Lite and matching it on TB-2.1.Against TMax-189, MiRL uses about the same rollouts and GPU hours but scores 8.4 points higher on TB-Lite.Source: Figure 9 · p. 8, §4.5 · p. 8
  • MiRL’s TB-2.1 score rises from 16.1% to 22.1% as the environment count increases from 0.25× to 2×; half the environments match TMax-4B on TB-Lite, 48.0% versus 47.4%.The 2× setting uses 378 environments, including the original 189.Source: Figure 11 · p. 9, §4.5 · p. 9
  • With 16 TB-2.1 seeds, MiRL retains the highest mean score at 21.14%, versus 20.22% for TMax-4B and 19.66% for RaR+.Mean standard error shrinks as the number of seeds grows, reaching 0.62 at 16 seeds.Source: Figure 12 · p. 9, §4.5 · p. 9

5 Conclusion

MiRL applies error-targeted learning to terminal agents and shows it can work when training targets generalizable misbehaviors where they occur. The authors note limitations in model coverage, training-run replication, profiling data, and reward design.

5 Conclusion

  • MiRL reduces targeted misbehaviors, matches or outperforms baselines on two benchmarks, and uses fewer training resources than TMax-4B.The authors conclude that error-targeted learning can work for long-horizon terminal tasks when training targets generalizable misbehaviors where they occur.Source: §5 · p. 9
  • Error profiling locates and corrects a policy’s error steps, then groups them into error types to identify generalizable misbehaviors.Source: §5 · p. 9
  • MiRL starts RL rollouts from misbehavior prefixes and rewards them only when they pass tests and avoid the misbehavior.Source: §5 · p. 9

Limitations

  • The study uses one model backbone, leaving the generality of its gains across model choices unclear.The authors also report training each recipe once, reliance on external LLMs for profiling, possible prefix-selection bias, and limited reward-combination exploration.Source: Limitations · p. 10

A Configuration Details

This configuration section specifies prompts and settings for judging terminal-agent behavior and profiling recurring, training-relevant errors. It covers detecting named failure modes, generating and scoring task rubrics, and judging command-response equivalence by execution stage and primary action type.

A Configuration Details

  • The error-localizer prompt judges one proposed step from the agent’s context, marking as bad steps that raise failure risk, including impossible steps, but not merely slow or different approaches.The prompt asks for a short justification and a good-or-bad verdict for the single bash command.Source: A Configuration Details · p. 12
  • Error profiling asks annotators to identify recurring, training-useful behaviors, favoring types that transfer across examples and occur often enough to matter.Types should describe what the agent did rather than the task or tool, and be checkable from one example alone.Source: A Configuration Details · pp. 12–13
  • The RL settings follow TMax except where noted, and all Gemini 3.8 Flash calls use the medium thinking level.Table 1 lists the recipes’ hyperparameters.Source: A Configuration Details · p. 12, Table 1 · p. 13
  • In refinement, annotators assign examples to matching existing types and propose a new type only when several examples share a recurring pattern not already covered.The instructions also allow multiple matching types, require strict assignment, and prohibit inventing or renaming types during fixed-list labeling.Source: A Configuration Details · pp. 14–15
  • The failure-mode prompt asks whether a named mode occurs in any of the agent’s next m steps, focusing only on that mode and ending with a yes/no answer.It supplies the task, the agent’s steps, and the mode’s name, definition, and examples.Source: A Configuration Details · p. 15

Step 2: Check EXECUTION STAGE

Step 3: Check ACTION TYPE

  • Responses with different primary action types—such as installing versus searching or edge-case versus stress testing—are not equivalent.The action categories include diagnostic, install, search, modify, backup/restore, and edge-case or stress testing.Source: Step 3: Check ACTION TYPE · p. 16

Step 4: Check SCOPE PARITY

The judge checks whether candidate actions match the reference in scope and functional outcome, allowing implementation differences when the task result is equivalent.

Step 4: Check SCOPE PARITY

  • Scope parity compares the number of files or directories examined for exploration and the number of tests or validation checks for verification.Examples treat examining 2 rather than 5 files, testing 2 rather than 6 cases, or verifying 1 rather than 3 outputs as mismatches.Source: Step 4: Check SCOPE PARITY · p. 16

Step 5: Check FUNCTIONAL EQUIVALENCE

  • Different tools, command flags, installation methods, algorithms, or intermediate filenames are allowed when they achieve the same result.Cosmetic differences such as variable names and comments are also allowed.Source: Step 5: Check FUNCTIONAL EQUIVALENCE · pp. 16–17
  • Functional equivalence focuses on task output and functional outcome rather than implementation details.For code, checks include approach, outputs for the same input, key functionality, and critical bugs; for commands, they include operation type, outcome, and purpose in the workflow.Source: Step 5: Check FUNCTIONAL EQUIVALENCE · pp. 16–17
  • Different execution stages, action types, significantly different scope, missing code, or missing required artifacts count as non-equivalent.The checks distinguish, for example, write-only from write-and-execute actions and examining from restoring.Source: Step 5: Check FUNCTIONAL EQUIVALENCE · p. 17

B.1 Statistics

Profiling finds that eight recurring misbehaviors account for just over half of the error steps, with each appearing across multiple environments.

B.1 Statistics

  • 52.1% of the 12,746 profiled error steps are covered by the eight most frequent misbehaviors.The table reports each misbehavior’s share of error steps and environments; steps and environments may carry multiple labels.Source: §B.1 · p. 17, Table 2 · p. 17
  • The three most frequent misbehaviors each cover more than 11% of error steps and appear in more than a third of environments.Even the least frequent listed misbehavior, “Prints a binary file as text”, appears in 140 of 2,017 environments (6.9%).Source: §B.1 · p. 17, Table 2 · p. 17

B.2 Examples of Misbehaviors

The figures provide one compact example for each misbehavior, showing the local context and the error step identified by the error localizer.

B.2 Examples of Misbehaviors

  • Figures 13 and 14 show one example error step for each misbehavior, selected to make the error visible with a short prefix and command.Each box includes the annotator’s definition, task start, final prefix step, error command in red, and localizer explanation.Source: §B.2 · p. 17

B.3 Long-Tail Error Steps

The long-tail error steps are unlabeled or belong to rare types, and often depend on details specific to one environment, making their fixes unlikely to transfer.

B.3 Long-Tail Error Steps

  • The remaining error steps are unlabeled or fall into rare types, each covering less than 1% of error steps.Table 3 gives examples tied to details of individual environments.Source: §B.3 · p. 17, Table 3 · p. 17
  • Fixes for these errors depend on environment-specific details, such as a log format or version rule, and are therefore unlikely to help on other environments.Source: §B.3 · p. 17, Table 3 · p. 17

C Cost Calculation Details

The cost calculation counts resources through each recipe’s best checkpoint, includes preprocessing, estimates unavailable TMax totals, and prices policy tokens separately from GPU hours.

C Cost Calculation Details

  • Resources are counted through each recipe’s best checkpoint, including preprocessing stages such as MiRL error profiling, PivotRL step profiling, and RaR+ rubric writing.The authors describe this as fairer to recipes that train longer than needed; most values come from their logs.Source: C Cost Calculation Details · p. 17
  • TMax-4B’s unreported totals are estimated from its settings and curves, yielding 75.9K rollouts and 2,370 environments.The rollout estimate uses SBG/(1 − d), with d derived as 32.5% from the TMax paper’s Figure 11; all-pass groups are counted as zero.Source: C Cost Calculation Details · p. 18
  • TMax-4B token and GPU-hour estimates use per-step tokens, rollout length, and throughput from the TMax-189 run, except output tokens per step, taken from the TMax paper’s Figure 6.Source: C Cost Calculation Details · p. 18
  • Total cost converts policy tokens to dollars at the Table 4 Tinker prices, adjusts uncached input tokens, and adds the model API fee while reporting GPU hours separately.GPU hours are excluded from total cost because the policy is priced as a per-token service rather than a rented GPU cluster.Source: C Cost Calculation Details · p. 18

D Artifacts, Licenses, and Use of AI Assistants

The paper lists licenses and software versions for its research artifacts, describes its release terms, and discloses use of Claude Code for scripting and formatting.

D Artifacts, Licenses, and Use of AI Assistants

Related papers

Loading 2610.35697v1…