PaperlayerView exact revision on arXiv

Source-linked AI summary

PANDO: Efficient Multimodal AI Agents via Online Skill Distillation

Yubo Li, Yidi Miao, Yuntian Shen, Yuxin Liu

arXiv:2605.24785v2cs.AI

TL;DR

Web agents often improve by spending more inference-time computation, motivating whether accumulated experience can instead make them more efficient. PANDO uses single-rollout online skill distillation with structured skills and efficiency-oriented routing, compression, and prompting, reaching 58.3% success on 910 VWA tasks while using fewer tokens than evaluated baselines. The paper also reports trajectory-level metrics and scope limitations for interpreting this result.

  • Problem

    The paper asks whether web agents can become more efficient as they accumulate experience instead of relying on rollout search, verifier passes, offline discovery, and other added inference computation.

  • Method

    PANDO performs single-rollout online skill distillation through a structured Skill Library combined with reflection, confidence-based demotion, hierarchical routing, visual compression, and cache-aware prompting.

  • Results

    58.3% success on all 910 VWA tasks outperforms SGV and the WALT reproduction while using fewer tokens than every evaluated baseline.

  • Takeaways & Limitations

    PANDO’s results support reporting success together with trajectory-level efficiency measures such as repetition, step overhead, and prompt-cache utilization.

  • Takeaways & Limitations

    Empirical claims are limited to VWA, while desktop tasks require new rules and adversarial task ordering could increase online-learning cold-start cost.

Abstract

from arXiv · show

Recent advances in multimodal web agents often rely on increased inference-time computation, including rollout search, verifier passes, offline skill discovery, and specialist model stacks. This raises a central question: can a web agent become more efficient as it accumulates experience, rather than more expensive? We first analyze trajectories from VisualWebArena and identify three recurring sources of inefficiency: repeat-action loops, hidden discovery costs, and low prompt-cache reuse. We then introduce PANDO, a single-rollout online skill-distillation framework that maintains a structured Skill Library and combines progress reflection, confidence-based skill demotion, hierarchical routing, visual compression, and cache-aware prompting. On the full set of 910 VisualWebArena tasks, PANDO achieves a 58.3% success rate, outperforming SGV (54.0%) and our WALT reproduction (45.2%), while using 58% fewer tokens than SGV and 61% fewer tokens than WALT, without any pre-evaluation discovery budget. A 300-task ablation further shows that rules and routines provide most of the success gains, while routing, compression, and cache-aware prompting convert the larger skill library into lower marginal token cost. Finally, we introduce three trajectory-level efficiency metrics -- Action Repetition Rate, Step Overhead Ratio, and Prompt Cache Utilization -- to make efficiency visible beyond terminal success.

1 Introduction

PANDO frames web-agent efficiency as a trajectory-level problem: agents spend computation on repeated actions, discovery, planning, reflection, and poorly reusable prompts. It addresses this by accumulating online skills and reports higher success with lower token use than the cited baselines.

  • Motivation: 34–42% of image-annotated failures involve repeat-action loops, while text/caption methods have prompt-cache utilization below 11%.The trajectory audit also identifies off-benchmark tool discovery as a recurring source of wasted work.
  • Contribution: PANDO maintains a structured Skill Library that grows online during the evaluation stream, requiring no pre-evaluation discovery budget.Rules target repeated failures, routines replace multi-step browser subgoals, and learning includes skill admission, merging, confidence updates, and demotion.
  • Results: 58.3% SR on all 910 VWA tasks exceeds SGV by 4.3 percentage points and the WALT reproduction by 13.1 percentage points.The contribution summary reports fewer tokens than every baseline.
  • Results: A VWA-300 ablation attributes most success gains to skill components and most token reductions to routing, compression, and cache-aware prompting.These components make the larger skill library cheaper to invoke.

2 Related Work

Related work spans multimodal agents, serving and routing efficiency, test-time reasoning, and skill acquisition. PANDO differs by targeting trajectory-level inefficiencies and amortizing reusable experience across tasks.

  • Multimodal and computer-use agents: Computer-use benchmarks differ in action space, including DOM selection, offline demonstration matching, free-form pyautogui, and mobile gestures with function calls.These grounding differences shape what successful interaction means across benchmarks.
  • Efficiency analyses of agents and LLMs: Trajectory-level analyses measure step inflation, latency allocation, partial progress, cached-token share, hidden discovery spend, and marginal tokens per successful task.The cited work presents raw token count as only one accounting unit.
  • Efficiency analyses of agents and LLMs: Existing routing, caching, and screenshot-pruning methods reduce call, token, or visual-processing costs but do not detect cross-step repetition or amortize discovery across tasks.The comparison identifies trajectory-level coverage as the missing dimension.
  • Skill libraries and tool acquisition: Skill-acquisition methods vary by representation and lifecycle, from offline discovery to online induction during or across tasks.Offline methods pay pre-evaluation discovery costs, while some online methods lack skill deprecation.

3 A Cost Decomposition for Comparing Lifelong Agent Methods

The paper decomposes agent cost into rollout execution, verification, induction, and amortized pre-evaluation discovery, enabling descriptive comparison across systems. It emphasizes that omitted off-benchmark costs can make published comparisons unreliable.

  • Cost identity: Per-task cost is decomposed as Ctask(τ; π) = Nrollout(τ) Cexec(τ) + Cverify(τ) + Cinduce(τ), with pre-evaluation cost handled separately.The decomposition distinguishes rollout count, execution, verification, induction, and amortized discovery terms.
  • Comparison caveat: The identity is exact, but published VWA results often omit Cpre/|B| when discovery occurs off-benchmark, making cross-study comparisons unreliable.The paper marks missing terms as unreported rather than treating them as zero.
  • Published operating points: SGV keeps one rollout but adds a two-pass verifier, yielding approximately ρSGV ≈ 2.2 relative to the bare baseline.The cited mechanism gives Cverify(τ)≈1.2 Cexec(τ).
  • Published operating points: WALT pays an offline per-website discovery budget, while its published 52.9% VWA headline reports only post-discovery inference cost.The aggregate discovery term remains unquantified in the cited literature.
  • Online induction: ASI provides a precedent for inducing parameterized Python skills online during the test-query stream and admitting them after rewrite-and-test verification.This places online skill induction outside both rollout/verifier scaling and pre-evaluation discovery.
  • Scope: The decomposition is bookkeeping rather than a Pareto-frontier result, an optimal-investment prescription, or a guarantee that low-cost terms can be combined.The authors reserve those questions for empirical evaluation.

4 The PANDO Framework

PANDO separates planning, execution, reflection, and learning around a structured, executable Skill Library. Its lifecycle combines reusable rules and routines with verification, demotion, hierarchical routing, visual compression, and cache-aware prompting.

  • Architecture: PANDO uses strong models sparingly for planning and reflection, while a cheaper Actor handles frequent grounding and deterministic skills replace repeated action chains.The design directly maps rules to repeat loops, routines to recurring subgoals, and demotion to stale skills.
  • Skills: The Skill Library partitions into pattern-triggered rules and parameterized routines with structured metadata, keyword retrieval, confidence statistics, and executable semantics.Retrieval is literal keyword containment rather than embedding search.
  • Learning: Successful sub-trajectories become skill candidates only when they have reusable templates, verified selector patterns, and no matching demotion entry.Confidence updates, polarity-pair merging, and persistent demotion prevent monotonically accumulating stale skills.
  • Execution economy: The Reflector verifies URL, DOM, and screenshot changes, while unmatched subgoals fall through from retrieved routines to the Actor.Hierarchical routing reserves expensive reasoning for novel planning and reflection.
  • Execution economy: Visual compression reduces the dominant Actor term, and stable-prefix prompt layout raises cache utilization as the library grows.These mechanisms are designed to reduce the marginal invocation cost of accumulated skills.

5 Experimental Setup

PANDO is evaluated across all 910 VisualWebArena tasks using a shared trajectory ledger that measures success, computational cost, repetition, overhead, caching, and skill reuse.

  • Evaluation benchmark: PANDO is evaluated on all 910 VisualWebArena tasks across Classifieds, Shopping, and Reddit.Tasks are shuffled once with fixed seed 42 to interleave domains during online learning.
  • Core accounting: Success rate is the fraction of tasks with successful terminal evaluator verdicts, while Steps counts non-evaluator LLM calls, routine invocations, and browser actions.Steps track events where latency and tokens accrue under the 50-step VWA budget.
  • Core accounting: Tokens measure mean prompt, completion, and reasoning tokens per task, while Time measures wall-clock seconds from reset to terminal verdict.
  • Efficiency metrics: ARR, SOR, Prompt Cache Utilization, and stream-wise skill hit distinguish loop avoidance, failed-task overhead, prompt reuse, and skill firing.Prompt Cache Utilization is defined as U = Qcached/Qprompt.

6 Results and Analysis

Across the full VisualWebArena benchmark, PANDO combines higher success with lower token use and improved trajectory efficiency. Ablations and stream-wise analyses attribute the gains mainly to learned rules and routines, while routing, compression, caching, and skill hygiene reduce ongoing cost.

  • Main results: 58.3% SR makes PANDO the best automated result, exceeding SGV at 54.0% and reproduced WALT at 45.2% while using 115K tokens per task.PANDO uses fewer tokens than every baseline and strictly Pareto-dominates them in the token–success plane.
  • Ablation: Skill-learning components raise SR from 38.6% to 57.3% and reduce steps from 15.2 to 9.8 on the stratified VWA-300 subset.The subset result remains aligned with the full run: 59.0% versus 58.3% SR and 117K versus 115K tokens.
  • Ablation: Routing, compression, and cache-aware prompting add 1.7 pp SR while reducing tokens from 147K to 117K and increasing cache utilization from 69.3% to 72.0%.The ablation separates competence supplied by the library from marginal-cost reductions supplied by prompt-structure optimizations.
  • Learning dynamics: By task 910, the library grows from 12 seed routines to 47 induced routines, with 32 active after 15 demotions and 11 polarity-pair merges.Average steps fall from unstable early runs near 30 steps/task to 8.5 steps/task at the end of the rolling curve; cache rises from about 60% to 73%.
  • Stream-wise economics: Over the final 310 tasks, average steps fall to 8.9, tokens to 103K, cache reaches 76.0%, and skill-hit rate reaches 58.4%.These stream-wise values recover the full-run averages when weighted by block size.
  • Skill utility: Retrieved skills correspond to 70.6% SR versus 50.4% without a skill hit, while routine-backed subgoals use 3.7 fewer browser actions and 41K fewer tokens than matched fallbacks.Demotion also blocks 36 rediscovery attempts after 15 induced routines are blacklisted for repeated failure.
  • Token economics: PANDO’s efficiency gains coincide with lower marginal inference load through stable prompt prefixes, cached tokens, and shorter action chains.This distinguishes the system from a high-SR method with an unreported additional compute bill.

7 Limitations and Conclusion

PANDO’s empirical scope is limited to VisualWebArena, with broader desktop-task transfer and adversarial ordering left as open boundaries. The conclusion argues that online skill reuse can improve web-agent efficiency and that benchmarks should report resource metrics alongside success.

  • Limitations: PANDO’s empirical claims are restricted to VisualWebArena, while OSWorld-style tasks require new rules for misclicks, window focus, and multi-application coordination.
  • Limitations: Adversarial task ordering could increase cold-start cost despite stability under scrambled and 16-worker shared-library variants.
  • Limitations: Polarity-pair induction is syntactic, leaving broader program-equivalence discovery for future work.
  • Conclusion: PANDO reaches 58.3% VWA success while using fewer tokens than every evaluated baseline.
  • Conclusion: The paper recommends reporting success rate with raw tokens, cached-token share, hidden discovery spend, latency, and tokens per successful task.

A Additional PANDO Method Details

PANDO uses a structured, auditable skill library whose rules and routines are retrieved deterministically, merged across polarity pairs, and pruned when brittle. Its execution loop combines sparse reflection, model routing, visual compression, and cache-stable prompting, while diagnostics expose trajectory-level efficiency.

  • Skill representation and retrieval: The active skill library stores rules and parameterized routines with metadata, literal trigger keywords, confidence statistics, and executable semantics.
  • Skill representation and retrieval: Literal keyword retrieval keeps skill selection deterministic, auditable, and stable under prompt caching rather than using embedding search.
  • Skill representation and retrieval: Polarity-pair merging combines directionally opposite routines into one parameterized routine, doubling reuse probability for extremum tasks while reducing prompt churn.
  • Learning and demotion: Skills maintain pass/fail confidence estimates and can be demoted into a blacklist to prevent rediscover-and-refail cycles.
  • Reflection: The Reflector fires periodically or after errors, compares multiple environment signals, and uses progress evidence to update confidence or trigger replanning.
  • Routing, compression, and cache: PANDO routes high-capability models sparingly, uses deterministic routines for repeated chains, and places stable prompt content before volatile observations to improve cache reuse.
  • Accounting and diagnostics: The trajectory ledger counts LLM calls, routines, and primitive browser actions as steps, enabling ARR and related efficiency metrics from shared event records.
  • Accounting and diagnostics: By task 310, one polarity-aware routine had 47 passes and 3 failures, completing a query in 1 skill call plus 2 primitive actions versus 6 baseline actions.

I.1 Main Results: Per-Method Token and Dollar Cost

Across the 910-task VisualWebArena benchmark, PANDO combines the highest reported success with the lowest token and dollar costs, and its learned library becomes cheaper to use as evaluation proceeds.

  • Main results: PANDO defines a new cost–success Pareto point: every baseline has both lower success and higher per-task cost, including WALT at headline and amortized costs.
  • Main results: PANDO is 86% cheaper per task than WALT’s headline figure and 77% cheaper than SGV while posting higher success in both comparisons.
  • Ablation and cost: Routing, visual compression, and cache-aware prompting account for most cost reduction from the $0.258 baseline to full PANDO at $0.085, while skill rows provide larger success gains.
  • Learning curve: PANDO’s per-task cost falls from $0.164 on task 1 to $0.062 on task 910, a 62% reduction as the routine library and cache stabilize.
  • Learning curve: PANDO spends $77.4 across 910 tasks versus $149.2 for a fixed-library variant run at task-1 cost.
  • Token composition: PANDO has the lowest total token load at 115K per task, despite Planner and Reflector tokens dominating its own composition.

J Per-Domain VWA Results

The supplied per-domain passages identify Classifieds as the domain where PANDO’s success-rate lead is largest, associating that advantage with concentrated polarity-pair induction.

  • Per-domain results: PANDO’s largest per-sub-site success-rate lead occurs on Classifieds, where polarity-pair induction concentrates.

K Residual Failure Analysis

PANDO’s residual failures are concentrated in grounding and task-specification issues rather than repeat-action loops. Robustness checks show stable success across task-order and parallel-execution variants, with paired-bootstrap intervals capturing observed ordering variation.

  • Residual failure categories: 37.5% of audited residual failures were grounding errors, followed by underspecified tasks at 18.7%.Other categories included polarity variants (15.3%), skill-library coverage gaps (13.7%), unmatched loops (9.0%), and miscellaneous failures (5.8%).
  • Robustness: 57.9% SR under scrambled task order and 58.1% SR with shared-library parallelism show small deviations from the main run.The scrambled run changed SR by −0.4 pp, while the parallel variant changed it by −0.2 pp.
  • Uncertainty: 0.4 pp was PANDO’s maximum cross-ordering spread, contained within the paired-bootstrap half-width of ±1.6 pp.The intervals use per-task verdicts and paired resampling for method comparisons.
  • Statistical comparison: +4.3 pp was PANDO’s lead over SGV, with a 95% CI of [+2.0, +6.6] and McNemar p<0.001.The comparison is reported for the full VWA-910 evaluation.

N Backbone-Controlled Comparison

Backbone-controlled analyses indicate that PANDO’s routing and skill-learning effects persist across model substitutions, while cold-start performance reflects different training-cost and asymptotic-success trade-offs. The cost claim remains partly dependent on model pricing and modality choices.

  • Routing-attributable lift: +19.9 pp is PANDO’s routing-attributable lift over its 38.4% same-backbone baseline, more than twice SGV’s +9 pp lift.The comparison does not eliminate backbone confounding, but the within-paper lifts bound how much it can explain.
  • Cold start and accumulation: 56.7% cold-start SR for SGV-on-Opus exceeds PANDO’s 50.5% in the first 100 tasks because PANDO is still bootstrapping its library.Across the stream, PANDO block averages rise from 50.5% on tasks 1–100 to 61.0% on tasks 601–910.
  • Backbone swaps: 50.3% SR for PANDO-on-Gemini retains most of the routing lift over the approximately 45% Gemini-Flash backbone-only baseline.The remaining 4.4 pp gap to Opus-backboned PANDO is attributed in the passage to the reported multimodal capability gap.
  • Scope of claims: The $0.085 per-task cost depends on Opus 4.6 and GPT-5.2 prices and their modality split, unlike savings attributed to cache reuse and skill compression.The passage separates cost dependence from the success-rate claim’s stated routing-lift condition.

O Reproducibility

The paper documents implementation details, artifacts, robustness settings, release safeguards, and checklist-supported scope statements. Reproducibility is supported by trajectory logging and release plans, while the evaluation remains bounded by benchmark, task-stream, and statistical limitations.

  • Experimental disclosure: The paper reports seeds, API rate limits, software versions, and sequential-versus-parallel wall-clock characteristics for reproducibility.These details are listed alongside scripts that regenerate reported figures from manuscript tables and trajectory ledgers.
  • Artifacts: Every efficiency metric can be recomputed from logged trajectories containing calls, actions, routines, reflections, cache counters, and evaluator outputs.The release includes the tracker, prompt templates, plotting scripts, skill schemas, and anonymized VWA trajectories.
  • Responsible release: The release excludes credentials, private site states, and policy-bypassing traces while keeping rules and routines inspectable, reviewable, disableable, and blacklistable.This is presented as a safeguard for potentially undesirable browser automation.
  • Scope and limitations: VWA-only evaluation, dependence on a trusted task stream, syntactic polarity-pair induction, and release constraints define the stated limitations.The paper identifies transfer beyond VWA as a next step because OSWorld-style tasks add failures and actions not exercised by VWA.
  • Statistical limitation: The main results are single full-benchmark runs rather than repeated independent trials with confidence intervals.Scrambled-order and parallel variants are reported as robustness checks, not substitutes for full statistical error bars.
Loading 2605.24785v2…