PaperlayerView exact revision on arXiv

Source-linked AI summary

Absorbed in Inertia: Activation Analysis for Computer-Use Agents

Giulio Segalini, Zhi Wen Soi, Jérémie Decouchant, Lydia Chen

arXiv:2609.37176v1cs.AI

TL;DR

Computer-use agents can repeat ineffective actions even after acknowledging failure, and the paper examines whether this inertia is reflected in model activations. It analyzes activation states and introduces R3, which resets context to find a screen-changing action before restoring prior history. R3 reduces measured inertia by 17–55% across models, with model-dependent effects on task score.

  • Problem

    Computer-use agents may repeat an unchanged screen–action pair despite explicitly recognizing in their reasoning that the approach has failed.

    Source: §1 · p. 1

  • Method

    The paper analyzes pre-action activations and introduces R3, which resets context, searches for a screen-changing action, then restores earlier history with that action appended.

    Source: §1 · pp. 1–2

  • Results

    R3 reduces measured inertia by 17–55% across models, with model-dependent effects on task score.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Changing context can interrupt recurrence, whereas linear steering does not reliably reduce it; R3 allows reconsideration from fresh context before restoring interaction history.

    Source: §5 · p. 9

    EvoCUA shows no execution improvement; search exhaustion terminates some trajectories and accounts for part of the recurrence reduction.

    Source: §4.2 · p. 9

Abstract

from arXiv · show

Computer-use agents have become increasingly capable of executing tasks on live desktops through natural-language instructions, based on trajectories of screenshots, actions, and reasoning. We discover that they can stealthily exhibit inertia, in which they repeat fruitless actions despite recognizing that these actions are ineffective. We hypothesize that inertia is reflected in the agent's internal state, i.e., the activation values of the agent's underlying model, and propose a protocol to measure the relationship between the two. Extensive analysis of high-dimensional activation states shows that inertia corresponds to an absorbing region of activation space, where activation values become stale across actions and even after attempts to steer them. We conjecture that drastically changing the agents' activations by re-initializing them is necessary to escape inertia. Specifically, we propose R$^3$ (Reset, Reroute, Restore), which temporarily resets the agent's context trajectory to escape the absorbing region and then restores the historical context to effectively complete the task. Our approach yields 17-55% lower measured inertia across models relative to unmodified agents. These results suggest that changing the context can interrupt recurrence more effectively than directly steering the resulting activations. Our code is available at https://anonymous.4open.science/r/vlm-agent-defense-D076

1 INTRODUCTION

The paper defines inertia as screen–action recurrence that continues despite an explicit acknowledgment of failure, and finds that pre-action activations distinguish recurrence and inertia. It compares activation steering with context reset and introduces R3 to restore useful history after finding a screen-changing action.

1 INTRODUCTION

  • Inertia is repeated screen–action recurrence despite acknowledged failure, combining an unchanged visible state with explicit failure acknowledgment in the agent’s reasoning.Such recurrence is more common in unsuccessful than successful trajectories and can consume interaction budget after the agent recognizes its approach is ineffective.Source: §1 · p. 1
  • Pre-action activations distinguish screen–action recurrence from other actions and, among recurrences, distinguish cases with and without failure acknowledgment.These findings are consistent with inertia occupying an absorbing region of activation space.Source: §1 · p. 1
  • Neither linear activation steering nor a prompt-level failure notice consistently reduces recurrence.The figure schematically contrasts steering that can preserve recurrence with reset recomputation that can lead to a different action.Source: §1 · p. 1, Figure 1 · p. 2
  • At recorded recurrent states, steering produces the recurrent action in 69–86% of samples, compared with at most 4% after context reset.A reset can discard useful task-progress information, motivating R3’s restoration step.Source: §1 · p. 2
  • R3 reduces measured inertia by 17–55% across models, with model-dependent effects on task score.The method resets context when recurrent actions leave the screen unchanged, searches for a screen-changing action, then restores earlier history with the new action appended.Source: §1 · p. 2

2 INERTIA AS AN ABSORBING REGION

The paper operationalizes inertia as detected screen–action recurrence paired with explicit failure acknowledgment, then examines recurrence patterns and pre-action activations. Across the tested models, recurrence concentrates in unsuccessful trajectories, while activation analyses show persistence during recurrence and information about recurrence and acknowledgment on held-out tasks.

2 INERTIA AS AN ABSORBING REGION

  • The section frames inertia as an absorbing-region hypothesis and examines how it appears in the model’s internal activations.The activation is sampled after reasoning so that reasoning is incorporated into the model state, but before the action is generated.Source: §2 · p. 2, §2.2 · p. 3

2.1 CHARACTERIZING INERTIA

  • The detector flags repeated actions that leave the screen unchanged or screen–action cycles that return to their starting screen.Repeated actions that reveal new screens, such as useful scrolling, do not count as recurrence.Source: §2.1 · p. 2
  • Inertia is defined as screen–action recurrence together with explicit failure acknowledgment in the preceding reasoning, assessed separately by a blind LLM judge.The judge sees the task and reasoning but not screenshots, executed actions, or detector outputs.Source: §2.1 · p. 3
  • Exact screen–action recurrence is used instead of judging action utility, which remains an open problem and can be difficult to infer from diverse trajectories.The detector identifies recurrence mechanically without assigning broader step-level utility labels.Source: §2.1 · pp. 2–3
  • Recurrence occurs in up to 66% of unsuccessful trajectories, and 55–89% of recurrences include failure acknowledgment for three of the four models.EvoCUA has the lowest measured inertia among screen–action recurrences.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 INERTIA IN ACTIVATION SPACE

  • During the longest recurrent runs, successive pre-action activation vectors change little, unlike the variation across actions before recurrence.The activations are examined immediately before action generation, after reasoning has been produced.Source: §2.2 · pp. 3–4, Figure 3 · p. 4
  • The recurrence direction separates actions with and without screen–action recurrence with validation AUC of 0.72–0.95, while the inertia direction reaches 0.82–0.90.Figure 4 reports mean AUC under nested task validation with equal task weights and 95% bootstrap intervals.Source: §2.2 · p. 4, Figure 4 · p. 4
  • Held-out-task readouts distinguish recurrence and failure acknowledgment, suggesting that acknowledgment remains encoded even when pre-action activations change little across recurrent actions.Activation persistence measures state change; readouts assess whether the state carries information about recurrence and acknowledgment.Source: §2.2 · p. 4

3 ACTIVATION EDITING TO ESCAPE INERTIA

This section tests whether changing pre-action activations can interrupt inertia, contrasting direct steering with recomputing activations from altered context. Because steering is inconsistent, it introduces R3's reset-reroute-restore procedure.

3.1 WHY NOT JUST STEERING?

  • Steering directions contrast recurrence with non-recurrence and inertia with non-recurrence after removing the recurrence-aligned component; negative steering strength subtracts both directions.The directions are task-balanced mean differences, normalized to unit ℓ2 norm, and applied at every action.Source: §3.1 · p. 5
  • Steering changed inertia by a 33.8% reduction to a 44.0% increase and recurrence by a 5.6% reduction to a 39.0% increase; every observed reduction had a confidence interval including zero.The tested steering configuration did not consistently reduce either measure.Source: §3.1 · p. 5

3.2 R3: RESET, REROUTE, RESTORE

  • R3 triggers after k consecutive repetitions of the same screen–action pair without a screen change, then resets context to the task instruction and current screen.Only single, non-wait actions count toward the trigger.Source: §3.2 · p. 6
  • From the reset context, Reroute proposes actions using only the task and current screen, accepting the first screen-changing action without judging task-level progress.It retries for at most N attempts; if none changes the screen, the trajectory terminates.Source: §3.2 · p. 6
  • Restore reconstructs the history preceding the recurrent sequence and appends the accepted Reroute interaction, while excluding the recurrence and rejected attempts from context.Subsequent activations are computed from the restored history and accepted transition.Source: §3.2 · p. 6

4 EVALUATION

The evaluation tests R3 against other interventions on complete OpenComputer trajectories and examines how reset changes proposed actions, activation neighborhoods, and execution outcomes. R3 reduces inertia across all four models, while task-score and execution effects vary by model.

4.1 REDUCING INERTIA IN FULL TRAJECTORIES

  • Inertia falls 33.9%, 16.5%, 39.2%, and 54.9% under R3 for UI-TARS, GUI-Owl, OpenCUA, and EvoCUA, respectively; recurrence also falls 17–43%.R3 has the largest point-estimate reduction in screen–action recurrence for all four models.Source: §4.1 · p. 7, Table 1 · p. 7
  • Task-score changes under R3 are +0.60, +1.20, −2.50, and −0.90 points for UI-TARS, GUI-Owl, OpenCUA, and EvoCUA, with neither negative change significant.Escaping a recurrent action does not guarantee task completion.Source: §4.1 · p. 7

4.2 HOW RESET BREAKS INERTIA

  • At recorded trigger states, reset repeats the triggering action in 0–4% of proposals, versus 69–86% with steering; selective reset still repeats it in 61–74%.Selective reset removes later duplicate attempts but keeps the first traversal and adds a prompt note naming the repeated action.Source: §4.2 · p. 8, Table 2 · p. 8
  • After reset, every model has fewer nearby activation states associated with action reuse, even when reference activations are matched by trajectory position.GUI-Owl stays low over recorded follow-up; the share gradually rises again for UI-TARS and OpenCUA.Source: §4.2 · p. 8, Figure 6 · p. 8
  • For UI-TARS, GUI-Owl, and OpenCUA, R3 changes the screen without an observed return in 69%, 72%, and 73% of triggers, versus 44%, 29%, and 22% unmodified.Some trajectories still return within the recorded follow-up.Source: §4.2 · p. 9, Figure 7 · p. 9
  • EvoCUA does not improve during execution: reroute exhausts its search budget in 19.59% of trajectories, which then terminate.Restoring trigger history and resuming after exhaustion increases exact re-entries without a clear task-score improvement.Source: §4.1 · pp. 7–8, §4.2 · p. 9

5 CONCLUSION

The conclusion links acknowledged failure to persistent repeated actions and interprets activation analyses as consistent with inertia occupying an absorbing region. It presents context change through R3 as a way to interrupt recurrence, while noting model-dependent task-score effects.

5 CONCLUSION

  • Pre-action activations distinguish recurrence from other actions and distinguish acknowledged from unacknowledged failures among recurrences, consistent with an absorbing activation region.The conclusion describes recovery as requiring a more drastic change in the agent’s current state, internally and externally.Source: §5 · p. 9
  • Across four models, R3 reduces measured inertia by 17–55%, while task-score effects vary by model.R3 temporarily resets context so the agent can reconsider its next action before the earlier interaction history is restored.Source: §5 · p. 9
  • Changing context can interrupt recurrence, whereas linear activation steering does not reliably reduce it.R3 changes context before restoring the earlier interaction history.Source: §5 · p. 9

AI USE STATEMENT

The paper describes generative-AI assistance, situates exact screen–action recurrence among related work, and specifies how recurrence and acknowledgment are measured in the analyzed corpus.

AI USE STATEMENT

  • Generative AI supported hypothesis and methodology development, experiment critique, implementation, result interpretation, figure and code work, manuscript drafting, and literature-related tasks.The authors state that it was not used to generate synthetic datasets or formulate mathematical claims, among other listed exclusions, and take responsibility for the final manuscript and artifacts.Source: AI USE STATEMENT · p. 10

A RELATED WORK

  • Unlike broader judgments about information state or action utility, exact recurrence is assigned directly from screens and actions and is examined alongside expressed failure acknowledgment.The paper distinguishes this measure from prior labels and evaluation approaches, including overacting, step-level rewards, progress prediction, and action-effect verification.Source: A RELATED WORK · p. 14

B.1 CORPUS AND ELIGIBILITY

  • The analysis uses OSWorld and OpenComputer trajectories, measuring recurrence among eligible trajectories by outcome and acknowledgment among recurrent actions.Acknowledgment percentages are action-based, so long repeated sequences contribute more than short ones.Source: §B.1 · p. 14, Table 3 · p. 15

B.2 DETECTOR IMPLEMENTATION AND IDENTITY SENSITIVITY

The detector identifies repeated screen–action patterns using pixel and action identity rules, including bounded cycle detection. Its exact matching choices make some changes unobservable and limit which recurrences it can detect.

B.2 DETECTOR IMPLEMENTATION AND IDENTITY SENSITIVITY

  • Screens match when pixels outside a masked desktop-clock region are identical; actions match when commands, coordinates, keys, and text are identical.Masking prevents clock updates from separating otherwise identical screens, but makes changes inside the masked region unobservable.Source: §B.2 · p. 15
  • The cycle detector searches for the shortest screen-continuous return up to eight transitions, groups signatures up to cyclic rotation, and counts later matching traversals as recurrence.First traversals do not count as recurrence; an unchanged-screen rule also marks repeated screen–action keys that leave the screen unchanged.Source: §B.2 · pp. 15–16
  • A detected recurrence does not establish that hidden application state stayed fixed or that the action was unnecessary.The eight-transition search bound also limits the cycles this implementation can detect.Source: §B.2 · p. 15
  • The detector’s recurrence reductions remain negative at every tested grid size, but OpenCUA’s intervals include zero at all three relaxed coordinate grids.The other three models retain intervals below zero throughout; these checks concern recurrence, not acknowledgment.Source: §B.2 · p. 16, Table 4 · p. 16

B.3 FAILURE-ACKNOWLEDGMENT PROTOCOL AND VALIDATION

The protocol labels whether pre-action reasoning acknowledges the agent’s own failures, then compares those labels with recurrence and validates judge agreement. Its measures capture expressed acknowledgment and co-occurrence, not latent awareness or whether an action contradicts the stated plan.

Failure-acknowledgment protocol

  • The judge labels failure acknowledgment from the agent’s reasoning, including loop language, blamed unsuccessful approaches, and announced retries, while excluding task-required repetition and neutral continuation.It receives the task instruction and reasoning, but not screenshots, executed actions, or the detector’s recurrence record.Source: §B.3 · pp. 16–17

Scope

  • The labels record co-occurring acknowledgment and recurrence, not whether the emitted action contradicts the reasoning’s plan.The judge labels expressed acknowledgment rather than latent awareness.Source: §B.3 · p. 17

Human validation

  • Human checks found 40/40 agreement with Judge 1 in one sample and included negative labels and OpenCUA in a separate 96-action check, but neither estimates population accuracy.Both checks use one annotator; the first sample is selected largely from joint positives, and the second uses a balanced design.Source: §B.3 · p. 18, Table 6 · p. 18

Agreement and task weighting

  • Table 8 reports task-macro acknowledgment and Cohen’s κ on actions assessed by both judges; these measure acknowledgment rates and inter-judge consistency, not accuracy against a human reference.Tasks without judged screen–action recurrences are excluded from the macro rate.Source: §B.3 · pp. 18–19, Table 8 · p. 19

Reasoning exposure

  • At recurrent actions, exposed reasoning length varies substantially by model, and shorter text may indicate less expressed acknowledgment without implying less internal failure recognition.Reported mean lengths range from 42.9 to 238.3 words across the listed model–benchmark combinations.Source: §B.3 · p. 19

B.4 REASONING AND EXECUTED ACTIONS: EXAMPLES

Examples show agents repeating screen–action behavior after acknowledging failure, sometimes even while announcing a change of approach. The following analyses specify activation extraction and contrasts, then describe validation projections and direction selection.

B.4 REASONING AND EXECUTED ACTIONS: EXAMPLES

  • In the examples, agents repeat unchanged-screen actions after identifying failure, including a three-action Generate–Tone–Cancel cycle and a click repeated despite an announced change of approach.The cycle returns to its earlier screen, while another example repeats a click on cell A1 after saying the approach should change.Source: §B.4 · pp. 19, 21, Figure 8 · p. 20

C.1 EXTRACTION SITES AND TASK SPLIT

  • Residual activations are extracted at each model’s reasoning–action boundary and paired with the same action’s label; OSWorld tasks are split by application category, with all actions from a task kept together.Fitting tasks estimate directions and reference projections; held-out validation tasks assess directions and select intervention layers.Source: §C.1 · p. 21, Table 9 · p. 21

C.2 STEERING-DIRECTION CONSTRUCTION

  • The primary steering direction contrasts recurrent with eligible non-recurrent actions using equally weighted within-task mean differences; a second direction isolates acknowledged recurrence and removes its component shared with the primary direction.The second contrast excludes recurrent actions without valid positive acknowledgment from its positive class.Source: §C.2 · p. 22, Table 10 · p. 22

C.3 VALIDATION

  • Validation projects activations onto the fitted direction and an orthogonal training-data principal component, while validation AUC ranks layers for steering and is measured on the same tasks used to select them.AUC averages within-task comparisons; direction stability is assessed across task splits against within-task label-shuffled fits.Source: §C.3 · p. 22, Figure 10 · p. 23, Table 11 · p. 23

C.4 INERTIA READOUT

The inertia readout separates acknowledged from unacknowledged recurrent actions, and its validation AUC is high across four models. The section also qualifies neighborhood analyses, specifies intervention behavior, and notes that reported parameter choices use the evaluation comparisons.

C.4 INERTIA READOUT

  • Validation task-macro AUCs for the inertia readout are 0.899 for UI-TARS, 0.818 for GUI-Owl, 0.899 for OpenCUA, and 0.869 for EvoCUA across tasks from both benchmarks.These are validation values on the same tasks used to select each model’s layer, not test values.Source: §C.4 · p. 23, Figure 11 · p. 24
  • On OpenComputer, inertia-readout AUCs range from 0.776 for OpenCUA to 0.976 for UI-TARS, while EvoCUA’s direction is not shown to be stable against label-shuffled fits.GUI-Owl’s AUC is 0.809; its split-half cosine is 0.707 against a label-shuffled maximum of 0.323.Source: §C.4 · pp. 23–24
  • On OSWorld, the registered split cannot evaluate OpenCUA or EvoCUA, so seeded splits are used; OpenCUA’s seeded direction is not shown to be stable.Registered-split validation AUCs are 0.836 for UI-TARS and 0.951 for GUI-Owl; seeded-split values are 0.822, 0.834, 0.949, and 0.747 in model order.Source: §C.4 · p. 24

C.5 ACTIVATION NEIGHBORHOODS

  • The neighborhood analysis uses available, uniquely matched captures rather than a random sample, and EvoCUA’s curve ends too early to show persistence after the trigger.Hash-only matching can select against repeated responses, and the query states from the two policies are not paired counterfactual activations.Source: §C.5 · p. 24
  • At the first R3 trigger, most neighbors for UI-TARS, GUI-Owl, and EvoCUA come from the first two actions of reference tasks, consistent with fresh-context states resembling task starts.The corresponding share for OpenCUA is 28.0%; matching action index does not match prompt length, so the shift does not isolate recurrence-specific representation changes.Source: §C.5 · p. 25, Table 12 · p. 25

D.1 POLICY SPECIFICATIONS

  • The reset controllers trigger on consecutive non-wait single actions with the same screen–action key that leave the screen unchanged, unlike the measurement detector’s multi-action-cycle coverage.Linear steering instead adds its direction update at every reasoning–action boundary, regardless of detector or judge labels.Source: §D.1 · p. 25
  • R3 saves history, tries up to three proposals from fresh contexts, and restores the pre-trigger history with a screen-changing accepted attempt appended.Screen change—not task progress—is the acceptance criterion; the environment is not restored between attempts, and failed search ends execution.Source: §D.1 · p. 26

D.2 PARAMETER SETTINGS

  • Steering strengths are selected on OSWorld validation tasks, while R3 thresholds use the reported OpenComputer comparisons rather than a separate held-out selection set.The R3 selection and reporting reuse the same tasks, which can favor selected estimates; an interval containing zero does not establish equivalent task performance.Source: §D.2 · pp. 26–27, Table 13 · p. 26

D.3 FULL RESULTS

The full results report relative reductions in task-macro rates for R3, alongside analyses of control variants, persistence and return, intervention costs, and replay outcomes.

D.3 FULL RESULTS

  • Relative reductions in the reported task-macro rates were 16.87%, 38.02%, 42.69%, and 33.78%, respectively.The paired-policy comparison uses task-level bootstrap intervals; the reported rates are not pooled across models or benchmarks.Source: §D.3 · p. 27, Table 14 · p. 27
  • R3 point estimates reduce inertia in all four models, but only the UI-TARS interval excludes zero; recurrence-action reductions have intervals below zero in all four.A lower rate of the joint inertia label may also reflect less expressed acknowledgment without fewer repeated actions.Source: §D.3 · pp. 27–28, Table 16 · p. 28

D.4 THRESHOLD SENSITIVITY AND CONTROLS

  • In soundness controls, only EvoCUA’s re-entry interval excludes zero when search exhaustion is followed by continued execution rather than stopping.The control restores trigger history and removes failed proposal turns without rolling back the environment.Source: §D.4 · pp. 28–29, Table 17 · p. 29

D.5 EXECUTED PERSISTENCE AND RETURN

  • Persistence counts repeated executions of the same action from the same screen that leave it unchanged, while return requires that action to recur after a screen change.Return rates are conditional on screen changes, and the policies can reach different states and have different remaining action horizons, so they are not comparisons from identical starting states.Source: §D.5 · p. 29, Table 18 · p. 30, Table 19 · p. 30

D.6 COMPUTATIONAL RESOURCES AND INTERVENTION COSTS

  • R3 executed 121, 93, 23, and 140 search actions; mean paired elapsed-time changes were +67, −93, −992, and −572 seconds in model order.These times include environment and serving time and do not isolate compute cost.Source: §D.6 · pp. 30–31, Table 21 · p. 31

E.1 REPLAY PROTOCOL

  • Replay proposals were not executed; a proposal escape means a parser-valid action differs from the repeated action, not that the screen changed or task progress occurred.Replay used recorded OpenComputer states with reset and steering conditions that differ in whether history and reasoning are retained.Source: §E.1 · p. 31

E.2 PROPOSAL OUTCOMES

Replay proposal outcomes compare context reset and activation steering, with selective-reset results also reported; proposal escapes do not establish executed progress.

E.2 PROPOSAL OUTCOMES

  • Context reset proposal escapes were 96%, 98%, 96%, and 100% for UI-TARS, GUI-Owl, OpenCUA, and EvoCUA, respectively.The figure’s percentages count proposal escapes, not executed progress.Source: §E.2 · p. 31, Figure 12 · p. 32
  • Composite steering reached 28%, 17%, 14%, and 15% proposal escapes, while matched-norm recurrence-only steering reached 26%, 18%, 16%, and 15%.Activation movement did not consistently change the proposed action.Source: §E.2 · pp. 31–32, Figure 12 · p. 32
  • For EvoCUA, steering left replay outcomes unchanged, with 84.8% repeated proposals in the unmodified and both steered conditions.The figure distinguishes composite steering, matched-norm recurrence-only steering, and reset proposals.Source: §E.2 · p. 32, Figure 12 · p. 32
  • Selective-reset repeated-proposal rates were 70.2%, 60.7%, 62.5%, and 73.9%, respectively.Replay used the trajectory policy’s explicit failure note and progress guard.Source: §E.2 · p. 32, Table 23 · p. 32

F LIMITATIONS

The paper identifies limits in its recurrence proxy, causal interpretation, and evaluation scope.

F LIMITATIONS

  • Exact recurrence detects repeated screen–action transitions, not whether hidden application state made progress.Equivalent screen content or action effects and model-specific coordinate sensitivity can cause misses.Source: F LIMITATIONS · p. 32
  • Reset effects do not establish causal mediation by fitted activation directions, and history removal is not isolated from resampling and termination.Search attempts execute without rollback and may have irreversible effects.Source: F LIMITATIONS · p. 32
  • Full-task mitigation was evaluated on OpenComputer, so transfer to other desktop or mobile environments is not established.Task-score intervals also do not establish improved completion or non-inferiority.Source: F LIMITATIONS · p. 32

Related papers

Loading 2609.37176v1…