PaperlayerView exact revision on arXiv

Source-linked AI summary

AndroidFlux: Evaluating and Reward Modeling Failure Recovery Capabilities in Mobile Use Multimodal Agents

Gyubeum Lim, Yemo Koo, Ghunwoo Jang, Zsolt Kira

arXiv:2610.34384v1cs.LGcs.AIcs.CLcs.CV

TL;DR

Clean-start success does not isolate mobile agents’ ability to recover from intermediate errors, a capability needed in changing environments. AndroidFlux evaluates agents and recovery-action preferences from controlled checkpoints around first errors; recovery only partially correlates with clean-start success, and reasoning helps consistently only for models trained with GUI-domain RL.

  • Problem

    Clean-start task success does not isolate whether mobile agents can recover from intermediate errors in changing environments.

    Source: §1 · p. 2

  • Method

    AndroidFlux restores AndroidWorld trajectories to controlled checkpoints around first errors, evaluates recovery and outcome-grounded action preferences, and releases an offline RL dataset.

    Source: §1 · p. 2, §3 · p. 3, §2 · p. 3

  • Results

    Recovery performance only partially correlates with clean-start success, and explicit reasoning improves recovery consistently only for models trained with GUI-domain RL.

    Source: §1 · p. 2, §5.1 · p. 5

  • Takeaways & Limitations

    The results indicate that clean-start benchmarks alone are insufficient to assess recovery robustness, while the released offline RL dataset improves both recovery and clean-start success.

    Source: §1 · p. 2

    AndroidFlux covers only AndroidWorld and replay- or snapshot-reconstructable first recoverable errors, not interacting error chains or other GUI environments.

    Source: §6 · p. 7

Abstract

from arXiv · show

Mobile-use multimodal agents are typically evaluated by task success from a clean initial state, which does not isolate their ability to recover from intermediate errors. Because mobile environments change dynamically, a general mobile-use agent must complete tasks robustly even from unexpected states and in unseen environments. This requires knowing when and where an agent makes errors, and whether it can recover from errors and subsequent states. To evaluate this capability, we introduce AndroidFlux, a failure-recovery benchmark built on AndroidWorld. AndroidFlux restores recorded trajectories to controlled checkpoints before, at, and after the first recoverable error, so that different agents continue from the same device state and history. On these checkpoints, a complementary reward-modeling benchmark evaluates recovery-action preferences using downstream rollout outcomes. We further release an offline reinforcement learning (RL) dataset that improves both error recovery and overall task success. Our evaluations show that recovery performance only partially correlates with clean-start success, and that explicit reasoning helps only models trained with GUI-domain specific RL.

1 Introduction

Clean-start task success does not reveal intermediate errors or recovery actions, despite the need for robust behavior in changing mobile environments. AndroidFlux evaluates recovery from controlled failure checkpoints and finds that recovery performance only partially correlates with clean-start success.

1 Introduction

  • Clean-start benchmarks score whole episodes and can hide intermediate failures and which actions helped or hurt.Failure states arise only by chance during clean-start rollouts, making them costly to observe and difficult to isolate.Source: §1 · p. 2
  • Robust mobile agents must recover from unexpected states in dynamic environments, including when taking over tasks from another agent.A stronger agent may inherit mistakes and a device state produced by a smaller agent.Source: §1 · p. 2
  • AndroidFlux evaluates recovery from controlled checkpoints around policy-induced first errors, and its results show that recovery performance only partially correlates with clean-start success.The benchmark also evaluates reward models’ recovery-action preferences and provides an offline RL dataset that improves recovery and clean-start success.Source: §1 · p. 2

2 Related Work

Prior mobile GUI benchmarks generally use clean-start episodes and terminal outcomes, while AndroidFlux targets mobile recovery at controlled checkpoints. The paper also distinguishes its outcome-grounded recovery preferences from earlier GUI reward-modeling approaches.

Mobile/GUI benchmarks

  • Most interactive GUI benchmarks start episodes from the home screen and return one terminal outcome, whereas AndroidFlux resumes mobile agents before, at, and after a first error.GUI-RobustEval concurrently studied post-error recovery for desktop agents [Bu et al., 2026].Source: §2 · p. 2
  • AndroidFlux evaluates whether a reward model can identify the recovery action more likely to succeed, in addition to whether an agent recovers.The action preferences are grounded in downstream outcomes.Source: §2 · p. 3

Reward modeling

  • UI-Genie uses perturbed correct actions as negatives, unlike recovery modeling from actions a policy actually took after breaking a state.The sources of policy-induced errors had not been scaled across policies, and RL with such rewards remained largely unexplored in online and offline settings.Source: §2 · p. 3

3 Problem Formulation

The formulation treats mobile interaction as history-dependent and resumes target policies from replayed or restored checkpoints around a first recoverable error. It also defines recovery-action reward modeling as ranking candidate actions by their likely downstream recovery outcome.

3 Problem Formulation

  • Mobile interaction is modeled as a finite-horizon POMDP with agent-specific context, because screenshots omit device information needed to reconstruct state.Recorded responses are re-rendered in each target policy’s context format rather than copying another model’s prompt format.Source: §3 · p. 3
  • Replay re-executes the recorded action prefix under the same task seed, with emulator snapshots used when replay is not reproducible.The checkpoint is the last replayed step supplied in context, and the target’s first new action follows it.Source: §3 · p. 3
  • The first error is the earliest action that fails to advance the task; later actions occur in a state already affected by that error.The benchmark considers only recoverable first errors, and subsequent actions are treated as recovery actions.Source: §3 · p. 3
  • Changing the checkpoint relative to the first error separates the cost of a committed error from the cost of inheriting another agent’s history.A target policy resumes from restored state and reconstructed context, and the terminal state is scored by the verifier.Source: §3 · p. 3
  • Recovery-action reward models score candidate next actions at the error checkpoint and are evaluated on whether their ordering matches outcome-labeled preferences.Each action pair shares the task, state, and history.Source: §3 · p. 3

4 AndroidFlux: Error Recovery & Recovery Action Reward Modeling Benchmark

AndroidFlux combines controlled recovery evaluation, recovery-action preference modeling, and an offline RL dataset, all built around policy-induced failures and restored mobile states.

4 AndroidFlux: Error Recovery & Recovery Action Reward Modeling Benchmark

  • AndroidFlux benchmarks whether policies repair broken states and whether reward models identify actions that can repair them, using AndroidWorld tasks.The benchmark compares agents on matched restored states and histories.Source: §4 · p. 3

4.1 Error Recovery Benchmark

  • The recovery benchmark includes 80 human-verified cases from six source models and 312 LLM-verified replay cases across 104 tasks and 21 source models.Of the human-verified cases, 65 were restored by replay and 15 by emulator snapshots.Source: §4.1 · p. 4

4.1.1 Annotation of the First Error

  • The first error is the earliest trajectory error unaffected by a preceding error, labeled by human review or a two-stage detector–verifier pipeline.The pipeline is tuned for precision over recall and reaches 90.8% step-level precision on the annotated set; errors are classified as Stagnation, Direction, or Execution.Source: §4.1.1 · p. 4

4.2 Recovery Action Reward Modeling Benchmark

  • The reward-modeling benchmark has 100 recovery states across 26 AndroidWorld task types, pairing next actions by differences in downstream rollout success.A panel of twelve agents evaluates action outcomes, and pairs are retained only when their success-count margin is at least two.Source: §4.2 · p. 4

4.3 Dataset: Offline RL Training Dataset for Error Recovery

  • The offline RL dataset contains 8,499 policy-induced contexts sampled from pre-error, first-error, and successful-state slices.Contexts are labeled for whether the goal is already satisfied; histories do not extend past the first error.Source: §4.3 · p. 4

5 Main Results

AndroidFlux evaluates recovery from controlled checkpoints around first errors and examines recovery-action preferences and offline RL. Results vary with agent architecture, failure timing, training context, and whether reasoning was trained for GUI tasks.

5.1 Failure Recovery Across Agent Architectures

  • MobileRun with Gemini Flash 3.0 raises recovery from 27.5 to 67.5 at tall on the human-verified set; GUI experts are generally more robust around the first-error checkpoint than general-purpose VLMs.Mobile-Agent-v3-7B reaches 35.4 at trep, compared with 24.6 for GUI-Owl-7B.Source: §5.1 · pp. 4–5
  • Explicit CoT does not uniformly improve recovery: GUI-domain reasoning RL variants improve, whereas reasoning-enabled Qwen3-VL-32B and Gemini Flash 3.0 perform worse at t.The comparisons use matched instruct and reasoning variants within model families, with Gemini Flash 3.0 as a same-model thinking on/off control.Source: Table 3 · p. 5, Figure 2 · p. 5, §5.1 · p. 5

5.2 Rewinding vs. Injection by Failure Timing

  • For early failures, rewinding gains +8.76pp, while for late failures injection gains +5.86pp; the gap between remedies grows by 12.22pp.The direction holds for 17 of 18 models individually and across task-difficulty quartiles and trajectory-length terciles.Source: §5.2 · pp. 5–6, Table 4 · p. 6, Figure 3 · p. 6

5.3 Recovery-Action Reward Modeling

  • BT outperforms same-data SFT on UI-Genie (0.904 vs. 0.878), while GUI experts’ likelihoods reach 0.58–0.60 on AndroidFlux without reward-model training.Combining five agents’ preferences with BT-256K ranks 87 of 100 AndroidFlux pairs correctly, versus 60 for the best single scorer.Source: Table 5 · p. 6, §5.3 · p. 6

5.4 Offline Reinforcement Learning

  • The UI 40% / t 35% / t−1 25% mixture improves both clean-start execution and error recovery over UI-Genie 100% and the base policy.Training only at the first-error checkpoint yields the strongest recovery but degrades clean-start performance.Source: Table 6 · p. 7, §5.4 · p. 7
  • Every run using the curated offline RL dataset outperforms UI-Genie 100% in mean success, while UI-Genie-only training provides little overall benefit and reduces first-error recovery.Reward rises similarly across mixtures, but GPRS reward is not a direct measure of policy quality.Source: §5.4 · p. 7, Figure 4 · p. 7

6 Limitations and Conclusion

AndroidFlux evaluates recovery from policy-induced first errors and reward models’ ranking of recovery actions, but its coverage is limited to AndroidWorld and reconstructible first errors. The results show recovery is distinct from clean-start success and varies with the source policy and induced error state.

6 Limitations and Conclusion

  • AndroidFlux evaluates recovery from policy-induced first errors and reward models’ ranking of recovery actions, but covers only AndroidWorld and first errors reconstructible by replay or snapshots.The authors identify chains of interacting errors and other GUI environments as open coverage gaps.Source: §6 · p. 7

A AndroidWorld & AndroidFlux Action Space

C.1 Human Verified Set

  • The human-verified set contains 80 cases selected from six models’ trajectories, after four of AndroidWorld’s 116 tasks were excluded for environment issues.65 error states were reproduced by replay; the remaining 15 were restored from emulator snapshots.Source: §C.1 · pp. 9–10, Table 10 · p. 10
  • Pre-error checkpoints t−3 and t−5 mix offset starts with clean starts when errors occur too early, and only 60.0% and 32.3% of cases admit those offsets.All cases admit t−1; the authors report aggregate scores alongside feasible-case scores where relevant.Source: §C.2 · p. 10, Table 11 · p. 10

C.3 LLM Verified Set

  • The LLM-verified set’s source-model coverage is deliberately unbalanced because the detector–verifier surfaces more usable first errors from weaker or more verbose policies.The set contains 312 cases over 104 tasks from 21 source models.Source: §C.3 · p. 11, Table 13 · p. 11

C.4 Human-Verified Set and LLM-Verified Set

  • Across verified sets, performance degrades from clean or pre-error states when agents resume with an annotated first error, while a single recovery action can alter downstream recoverability.Some Qwen3-VL models benefit from inherited correct prefixes at pre-error checkpoints.Source: §C.4 · p. 11, Table 14 · p. 12, Table 15 · p. 12

C.5 Cross-Agent Recovery Matrix

  • Recovery depends jointly on the source agent and its first-error state: agentic frameworks recover best across sources, while execution errors are more recoverable than direction-heavy profiles.Agentic frameworks reach 55.6% on Qwen3-VL-4B errors, versus 36.5% for general agents and 33.3% for expert agents.Source: §C.5 · p. 13, Table 16 · p. 13

C.6 Reproducibility

The evaluation repeats configurations under matched settings, but emulator and serving behavior still introduce nondeterminism. Replay restores most human-verified error states; snapshots handle cases that cannot be reproduced from recorded actions alone.

C.6 Reproducibility

  • Identical Docker images, emulator snapshots, app versions, task lists, and fixed seeds do not eliminate nondeterminism from asynchronous rendering, residual device state, clock-sensitive verifiers, and batched serving.Floating-point reduction order can vary with batch composition, preventing bitwise reproducibility even under greedy decoding.Source: §C.6 · p. 14, Table 17 · p. 14

C.6.1 Replay and Snapshot

  • Of 80 human-verified cases, 65 were restored by replaying actions through the first error, while 15 required saved emulator snapshots.Replay is considered successful when it reaches the same erroneous state observed in the original trajectory.Source: §C.6.1 · p. 14
  • The 15 snapshot cases could not be reproduced by isolated replay, in part because preceding tasks had changed residual state such as recency-ordered app positions.The emulator was manually reconstructed to match the original error state, with history adjusted only for state-dependent replay mismatches.Source: §C.6.1 · p. 14

C.6.2 Recovery Trajectory Generation

Recovery trajectories resume from a reconstructed first-error state and give the target policy the corresponding rendered history without telling it recovery is required. The protocol scores task completion and compares recovery actions through downstream outcomes.

C.6.2 Recovery Trajectory Generation

  • At checkpoint t, the target receives the restored screen and policy-native context, with no indication that the prefix was replayed, erroneous, or authored by another model.Recognizing the broken state is part of the benchmark’s evaluation.Source: §C.6.2 · p. 14
  • After the injected or restored state, the target acts autonomously until a terminal action or its remaining step budget expires, and earns success only if the programmatic verifier accepts task completion.Partial repair without completing the task is scored as failure; the same fixed state and prefix are given to every evaluated agent.Source: §C.6.2 · p. 15
  • Recovery runs repeat each task–target pair four times, and replay-restored spread is 2–5 recovered tasks per model, comparable in absolute terms to clean-start variation.The repeated-run table includes 65 replay-restored and 15 snapshot cases.Source: §C.6.2 · p. 15, Table 18 · p. 15
  • Rewinding gain compares success at t−1 with t, while injection gain compares t+1+ with t+1−; the headline contrast subtracts the earliest-bucket delta from the latest-bucket delta.The buckets group errors by steps already taken: ≤3, 4–6, and ≥7.Source: §C.6.2 · p. 15
  • The headline contrast’s 95% interval is [+4.1, +20.8], but its bootstrap p-value is not reported because it depends on only 6 to 13 of 6,000 resamples.The interval changes by less than 0.25 percentage points across random seeds.Source: §C.6.2 · pp. 15–16
  • Across robustness checks, the contrast is positive for 17 of 18 models, with Qwen3-VL-32B-Instruct the exception at −9.30pp; the two-sided sign test gives p = 0.0001.All nine latest-minus-earliest subgroup contrasts are positive.Source: §C.6.2 · p. 16, Table 20 · p. 16

C.7 First Error and Recovery Across Error Categories

First errors are classified by which part of the goal–action–transition sequence breaks, with repetition distinguishing stagnation from execution failure. Error-class distributions vary across source models, while generalized context construction re-renders histories for each target policy.

C.7 First Error and Recovery Across Error Categories

  • The taxonomy distinguishes errors by intended goal direction, chosen action, and observed transition, applying the fixed priority stagnation → direction → execution.Figure 5 illustrates one human-verified example per class: repeated swipe, wrong route, and wrong committed value.Source: §C.7 · pp. 16–17, Figure 5 · p. 18
  • Stagnation is established repetition or oscillation without visible progress, direction errors execute successfully but move away from the goal, and execution errors realize the intended action incorrectly.A first unsuccessful attempt is not stagnation; a first task-critical action with no effect is execution, while a repeated failed action is stagnation.Source: §C.7 · p. 17
  • Direction errors are the largest human-verified class overall at 43.75%, while stagnation is largest in the LLM-verified set at 47.11% and execution smallest at 17.31%.The LLM labels collapse detector predictions, and the sets differ in model composition, so per-model rows are the appropriate comparison.Source: §C.7 · p. 17, Table 21 · p. 18, Table 22 · p. 18

C.8 Generalized History and Context

  • A schema-neutral record is re-rendered in each target agent’s native format, while fields absent from the source are synthesized and the live accessibility tree is supplied at resume time.The recorded tree is omitted because it would describe the source trajectory’s screen rather than the target’s restored screen.Source: §C.8 · p. 19, Table 23 · p. 19
  • Synthesized histories omit explanations of purpose and any mention of errors, mistakes, replay, or recovery, so the target is not told that something went wrong.Usable source summaries are preferred only after sanitization; otherwise action-type templates supply neutral descriptions.Source: §C.8 · p. 19
  • Figure 6 batches up to ten screenshots with per-step text and carries each batch’s screen/action/result summary verbatim into the next batch’s history.This incrementally summarizes long trajectories instead of rereading them in full.Source: Figure 6 · p. 20

C.9 Detector–Verifier First Error Detection Framework

A two-stage detector–verifier pipeline identifies first errors in Android GUI trajectories, favoring precision because false labels would alter the states used in downstream experiments.

C.9 Detector–Verifier First Error Detection Framework

  • A detector proposes clustered error candidates from ten-step screenshot-and-action windows, then a verifier checks each candidate against the resulting UI state and returns one first-error step.Both stages use Gemini Flash 3.0 and are tuned toward precision because false labels change the state used in downstream experiments.Source: §C.9 · p. 21
  • Across six source models, verifier precision rises from 87.2 to 90.8 while recall stays essentially unchanged, indicating removal of false candidates rather than a precision–recall trade-off.Verifier precision ranges from 80.0 to 94.9 by source model, while recall ranges from 37.8 to 70.1.Source: §C.9 · p. 21, Table 24 · p. 21

C.10 Detector–Verifier Framework Prompt

  • Figures 7 and 8 show the first-stage detector prompt (Parts 1–2) for evaluating Android GUI trajectory steps; Figures 9 and 10 show the cluster verifier prompt (Parts 1–2) for first-counted-error refinement.Source: Figure 7 · p. 22, Figure 8 · p. 23, Figure 9 · p. 24

D AndroidFlux: Recovery Action Reward Modeling Benchmark

AndroidFlux’s recovery-action benchmark grounds action preferences in downstream environment outcomes, while its offline RL corpus supplies checkpointed contexts around first errors and successful task completion.

D AndroidFlux: Recovery Action Reward Modeling Benchmark

  • The benchmark builds outcome-grounded action preferences by restoring first-error checkpoints, collecting candidate actions, rolling out each action with a fixed panel, and pairing candidates by outcome margin.The panel scores each candidate by how many agents reach the task goal; preferences do not rely on local heuristics or an LLM judge.Source: D AndroidFlux: Recovery Action Reward… · p. 26
  • Of 100 restored checkpoints, 72 contain detected errors and 28 are clean-path controls; Direction is the most frequent detected-error category at 43.06%.Stagnation, Direction, and Execution are all represented in double digits, so no single failure mode dominates the error-path cases.Source: D AndroidFlux: Recovery Action Reward… · p. 26, Table 25 · p. 26
  • The released 100 preference pairs have a mean margin of 2.86 panel successes and a maximum margin of 8/12, with half at the two-success release threshold.All five candidate generators appear at least 17 times on each side, and no single generator dominates either role.Source: D AndroidFlux: Recovery Action Reward… · p. 27, Table 27 · p. 27

D.1 Reward Model Training

  • Three UI-Genie reward-model baselines use either pairwise Bradley–Terry training on 64K or 128K pairs or generative classification training on 64K pairs.The 128K set contains the 64K set; all examples use a context variant retaining at most five prior screenshots.Source: §D.1 · p. 28, Table 28 · p. 28

E.1 AndroidFlux Offline RL Dataset for error recovery

  • The offline RL corpus contains 8,499 checkpointed contexts from four positions: t−n, t−1, t, and successful trajectories, with goals checked against source trajectories.All twenty unseen-test task types are excluded; each context is annotated for whether its state already satisfies the goal.Source: §E.1 · p. 29
  • A positive termination label marks a restored state already satisfying the goal, making termination the correct next action.There are 212 positive prompts, or 2.49% of the corpus.Source: §E.1 · p. 29

E.2 Training Details

The training setup combines generic GUI and AndroidFlux checkpoint contexts for offline RL, with reward-model-based group shaping and a guard against premature termination.

E.2 Training Details

  • The policy is initialized from Qwen3-VL-4B-Instruct and fine-tuned with LoRA while freezing the vision tower and vision–language aligner.Its input includes the goal, latest ten history steps, and screenshot; it produces brief reasoning and an action followed by one tool call.Source: §E.2 · p. 30
  • Group Preference Reward Shaping converts reward-model scores within eight sampled completions into a centered advantage, while unparseable outputs receive zero but remain opponents in the group.The reward model is a frozen UI-Genie Bradley–Terry model trained on 128K pairs.Source: §E.2 · p. 30
  • A terminate action receives zero shaped reward unless the current context is labeled as already satisfying the goal, preventing reward-model ranking alone from rewarding premature termination.The guard changes shaped policy reward, not the underlying reward-model score.Source: §E.2 · p. 30
  • All mixtures use 2,400 prompts while varying source distribution across generic UI-Genie contexts and AndroidFlux contexts at t and t−1.The t-only set repeats 320 of its 2,080 available prompts; no reward-model or rollout-based filtering is applied.Source: §E.2 · p. 31
  • AndroidFlux mixtures are interpreted through their consistent improvement over UI-only training, not as individually significant pairwise differences or an optimized mixture ratio.Each distribution used one seed and 65 tasks per condition; observed seed variation was 3.1 points on clean evaluation and 6.1 points on error-t evaluation.Source: §E.2 · p. 31

F Qualitative: Succeeded and Failed Error Recovery Trajectory

The figures present examples of recovery trajectories after a first error, separated into successful and failed outcomes.

F Qualitative: Succeeded and Failed Error Recovery Trajectory

  • Figure 11 presents successful recovery trajectories beginning from the first error at t.The caption specifies GPT-5.6 Terra with think-on(medium) on the LLM-verified set.Source: Figure 11 · p. 32
  • The paired figures contrast success and failure under the same stated model, reasoning setting, and verified-set scope.The captions identify the trajectory categories but do not state what distinguishes their actions.Source: Figure 11 · p. 32, Figure 12 · p. 33
  • Figure 12 presents failed recovery trajectories beginning from the first error at t.The caption specifies GPT-5.6 Terra with think-on(medium) on the LLM-verified set.Source: Figure 12 · p. 33

Related papers

Loading 2610.34384v1…