PaperlayerView exact revision on arXiv

Source-linked AI summary

AdvSim2Real : Training Web Agents Against Adaptive Prompt Injection in a Web World Model

Sarim Hashmi, Mukul Ranjan, Kshitij Mishra, Mikhail Kuznetsov, Praneeth Vepakomma, Nils Lukas

arXiv:2610.08773v1cs.CLcs.AIcs.LG

TL;DR

Web agents must use page data and controls while resisting third-party instructions, and existing approaches leave either adaptive attacks or changing tasks insufficiently addressed. AdvSim2Real co-evolves a curriculum, an injection adversary, and an agent in a frozen web world model. On 150 tasks, it improves completion under clean and attacked conditions, including a 33.6% relative gain against an unseen adversary.

  • Problem

    Web pages mix task-critical data and controls with third-party instructions, while fixed-injection defenses and fixed-task adversarial training leave adaptive attacks or task freshness unaddressed.

    Source: §1 · p. 2

  • Method

    AdvSim2Real co-evolves a task curriculum, an injection adversary, and a web agent inside a frozen web world model.

    Source: §1 · p. 2, §1.1 · p. 3

  • Results

    33.6% relative gain in completion under unseen Kimi-K3; on 150 tasks, clean completion rose from 74.89% to 81.33% and completion under three learned adversaries from 48.07% to 57.48%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The capability gain carried over to a real browser, where strict success on the submitted form rose from 25.56% to 44.44% without a world-model call.

    Source: §1 · p. 3

    The evaluation does not isolate a single component, so the authors attribute gains to the checkpoint sequence as a whole rather than to any one mechanism.

    Source: §6 · p. 9

Abstract

from arXiv · show

Web agents complete user requests by reading and acting on pages that third parties write, so an instruction planted on a page can redirect the agent away from the user's goal. The agent cannot simply ignore the page, because the page also holds the values and controls the task requires. Current defenses fine-tune the agent on injections fixed before training, and attackers that adapt to the trained model bypass them. Adversarial training lets the attacker adapt but keeps the tasks fixed, so a task stops teaching once the agent solves it. We introduce AdvSim2Real, which co-evolves a task curriculum, an injection adversary, and the agent inside a frozen web world model. The curriculum is rewarded for tasks the agent solves about half of the time, and the adversary only for a success flip, an injection that turns a judged success into a failure. Training in the simulator makes a 4B agent both more capable and more robust: its completion rises with and without attacks, holds against a frontier-model adversary it never trained against, and its capability gain carries over to a real browser. On 150 web tasks, AdvSim2Real raises completion under this unseen adversary by 33.6\% relative to the base agent.

1 INTRODUCTION

Web pages combine task-critical data and controls with third-party instructions that can redirect agents, while existing defenses and adversarial training leave gaps in adaptation or task freshness. AdvSim2Real co-evolves tasks, an injection adversary, and an agent in a frozen web world model, improving completion on its benchmark and transferring capability to a real browser.

1 INTRODUCTION

  • Third-party pages mix task-critical values and controls with planted instructions, so agents must resist competing instructions without ignoring the page.Indirect prompt injection succeeds in up to 86% of cases on a web-agent security benchmark.Source: §1 · p. 2
  • Prior defenses train on fixed injections, while adaptive-attacker methods keep tasks fixed, so solved tasks stop teaching and new tasks require executable sites.The paper positions its approach as addressing both constraints.Source: §1 · p. 2
  • 33.6% relative gain in completion under unseen Kimi-K3; clean completion rose from 74.89% to 81.33%, and completion under three learned adversaries from 48.07% to 57.48%.The real-browser strict success rate also rose from 25.56% to 44.44%.Source: §1 · p. 3

1.1 CONTRIBUTIONS

  • ADVSIM2REAL co-evolves a task curriculum, an injection adversary, and a web agent in a frozen world model, with tasks and attacks adapting to the current agent.The curriculum targets tasks solved about half the time; the adversary earns reward only when an injection flips an accepted clean run into failure.Source: §1 · p. 2, §1.1 · p. 3
  • The success-flip reward credits the adversary only when replaying an accepted clean run with its injection makes the continuation fail.Failures the agent makes without an injection earn no adversary credit.Source: §1.1 · p. 3
  • The released benchmark contains 150 form-filling tasks in five skill strata, with protected fields, forbidden controls, reactive attacks, and deterministic browser checks.It includes checkpoints and trajectories.Source: §1.1 · p. 3
  • Without the curriculum stage, the final agent loses 4.67 clean points but only 0.44 points of attacked completion.This comparison is reported in Table 7.Source: §1.1 · p. 3

2 RELATED WORK

Related work addresses adaptive attacks, task generation, and web-world-model training, but the paper identifies no prior approach that adapts tasks to the current agent while co-training an attacker inside a web world model.

Defenses trained on fixed injections

  • Defenses trained on fixed injections can be bypassed by attackers that optimize against the trained model, motivating adversarial training with a learning attacker.Examples include RETA, ARLAS, CoER, GPT-Red, Self-RedTeam, DMAST, WARD, and ToolHazard.Source: §2 · p. 3

Adversarial training with a learning attacker

  • Prior adaptive-attack methods co-train attackers and agents but keep training tasks fixed, unlike ADVSIM2REAL’s task co-evolution.The related-work comparison also notes that these methods do not train inside a web world model.Source: §2 · p. 3

Adaptive task generation

  • Adaptive task-generation methods address the limits of fixed task sets, using failed attempts, curricula, synthesized variants, or intermediate-success rewards.ADVSIM2REAL uses an uncertainty reward like R-Zero and Agent0, but estimates success from judged executor completions rather than self-consistency.Source: §2 · p. 4

Learning inside a web world model

  • Web-world-model approaches train or evaluate agents using simulated page transitions, but prior work does not combine task co-evolution and attacker training inside one frozen web model.ADVSIM2REAL freezes WebWorld’s pretrained transition model so rounds and checkpoints share the same simulator.Source: §2 · p. 4

3 ENVIRONMENT, BENCHMARK, AND THREAT MODEL

The study evaluates web-task completion in a frozen world model and tests transfer in Chromium, using a benchmark with explicit form requirements and an attacker limited to page injections. Attacked results rely on world-model verdicts and do not establish that tasks remain feasible after injection.

Environment

  • The frozen world model predicts pages from the interaction history and browser actions, while a frozen judge grades whether trajectories complete the authorized goal.Trajectories end at a final message or after 12 actions; transfer uses the same pages in a real Chromium browser.Source: §3 · p. 4

Benchmark

  • The 150-task benchmark spans five skill strata and requires correct final fields, unchanged protected fields, avoidance of forbidden controls, and review before commit where required.A deterministic browser check verifies these requirements; evaluation tasks differ from curriculum-proposed training tasks.Source: §3 · p. 4

Threat model

  • The attacker can request one page injection after an executor action but cannot change the authorized goal, system instructions, parameters, or judge rubric.At evaluation it observes the current page and selected action, and may wait or inject; Kimi-K3 was not used in training.Source: §3 · p. 4
  • Injections are rendered only as plausible page content, but the world model can omit requested content or remove needed controls, and attacked outcomes are judged inside that model.The judge reads the same untrusted observations as the executor.Source: §3 · p. 5

Metrics

  • Completion measures the percentage of judged trajectories that succeed; the learned-adversary score weights Adv v1–v3 equally.The reported mean and sample standard deviation use evaluation seeds, not independent training runs.Source: §3 · p. 5
  • The evaluation measures task completion but neither attacker-objective success nor whether a task remains feasible after an injection.These boundaries qualify interpretation of attacked completion rates.Source: §3 · p. 5

4 METHOD

ADVSIM2REAL alternates curriculum-executor training with adversary-executor training inside a frozen world model. Its curriculum targets tasks near the executor’s competence frontier, while its adversary is rewarded for injected success flips and the executor trains on fresh rollouts.

Stage 1: Curriculum-executor co-evolution

  • Stage 1 co-trains the curriculum and executor, then Stage 2 freezes the curriculum and alternates adversary and executor updates across three rounds.The executor trains on new attacks, attacks from earlier rounds, and clean tasks; the adversary starts from the base model.Source: §4 · p. 5

Stage 2: Adversary-executor co-evolution

  • The executor trains on fresh trajectories from a mixture of current attacks, historical attacks, and clean tasks, with rewards based on judged outcomes.Stored trajectories and verdicts are discarded when assessing the mixture; the update uses new rollouts.Source: §4 · pp. 5, 7

Stage 1: Curriculum-executor co-evolution

  • The curriculum is rewarded for tasks the executor solves about half the time, and the executor trains on fresh trajectories for newly proposed tasks.Tasks solved always or never produce zero group-relative advantage for the executor update.Source: §4 · p. 5, Figure 3 · p. 6
  • The curriculum reward peaks at a judged success rate of one-half and vanishes at zero or one; zero observed successes do not establish impossibility.Difficulty pressure emerges from the update loop rather than a fixed target, and the reward includes a repetition penalty.Source: §4 · p. 6

Stage 2: Adversary-executor co-evolution

  • Stage 2 rewards the adversary only when an injection turns a good clean run into a bad judged continuation, excluding failures that occur without an attack.Clean controls are saved for tasks the executor solves at least half the time, and tasks with no good controls are dropped.Source: §4 · p. 6
  • The adversary proposes an injection, rendering marker, transition index, and optional target action; malformed proposals receive −1.The reward measures judged disruption but does not verify task feasibility or that the target action was taken.Source: §4 · pp. 6–7

Policy optimization

  • The policy updates use separate low-rank adapters, group-normalized advantages, and a penalty toward the adapter-disabled base model.The method omits likelihood ratios, clipping, and importance correction for curriculum replay.Source: §4 · p. 7

5 EXPERIMENTS

Experiments evaluate Qwen3.5-4B checkpoints on 150 web tasks, including clean Chromium runs and attacks from both learned and unseen adversaries. Training improves completion under attack, while capability-stage gains also transfer to the browser.

5 EXPERIMENTS

  • Evaluation uses three rollout seeds per checkpoint and compares the Qwen3.5-4B base, three checkpoints from each stage, and hosted Qwen3.5-9B.Checkpoints are named for the round that produced them.Source: §5 · p. 7
  • Stage 1 raises clean completion by 4.44 points, while the attack mean rises from 48.07% to 54.74% after its first round.The executor sees no injections during Stage 1.Source: §5 · p. 8
  • 44.44% strict browser success at Capability iter 3 compares with 25.56% for Base; correct final field values rise from 52.50% to 73.24%.The clean Chromium runs cover the 150 tasks; the deterministic check verifies submitted forms, and the field count covers 746 target values per seed.Source: §5 · p. 8, Table 2 · p. 8
  • Stage 2 raises attacked completion by 3.14 points, ending at 57.48%, while clean completion ends 2.00 points above Capability iter 3.The first Stage-2 round lowers both metrics; later rounds recover and exceed the starting point.Source: §5 · p. 8
  • 30.72% completion under Kimi-K3 for Robust iter 3 is 7.72 points above Base's 23.00%.Kimi-K3 took no part in training; the table reports completion under this frontier-model adversary.Source: Table 3 · p. 8, §5 · pp. 8–9
  • Removing Stage 1 leaves attacked completion only 0.44 points lower but clean completion 4.67 points lower than the full pipeline.The ablation jointly removes executor and curriculum initialization and does not match compute, so it does not isolate the curriculum alone.Source: §5 · p. 9

6 DISCUSSION AND LIMITATIONS

Endpoint gains are reported, but the authors cannot attribute them to individual components. The discussion also bounds the results by world-model judgment and by differences between training and evaluation attacks.

6 DISCUSSION AND LIMITATIONS

  • Endpoint gains are 6.44 clean points and 9.41 attacked points, but first-round declines and no component isolation limit attribution.The authors say they attribute the gains to the checkpoint sequence as a whole.Source: §6 · p. 9
  • Three designed mechanisms remain unisolated: curriculum rewards vanish at judged rates of 0 or 1, adversaries get no credit for the executor’s own failures, and historical attacks stay in training.Source: §6 · p. 9
  • World-model judgments can accept values the executor never entered or incorrect calculations, so judged robustness is not yet executable task completion; only capability checkpoints were browser-verified.Source: §6 · p. 9
  • Training adversaries propose from the initial page, whereas evaluation adversaries react to trajectories; matching a checkpoint matches the attacker, not the attack.Source: §6 · p. 9

7 CONCLUSION

ADVSIM2REAL improves completion on the benchmark, including against an unseen adversary, while the reported robustness figures remain judgments within one world model. The authors release artifacts for evaluation and identify limits on real-world testing and reproducibility.

7 CONCLUSION

  • On 150 web tasks, clean completion rises from 74.89% to 81.33%, learned-adversary completion from 48.07% to 57.48%, and unseen Kimi-K3 completion by 33.6% relative to the base agent.Source: §7 · p. 9
  • A 23.85-point clean-to-attacked gap remains, and every robustness number is a model judgment inside one web world model.Source: §7 · p. 10
  • Generative AI assistants supported experimental design, audits, interpretation, literature work, drafting, and consistency checks; the authors say they reviewed the assisted content and take responsibility.Source: §7 · p. 10
  • The released adversaries were trained and evaluated on synthetic pages; the authors have not tested them against deployed agents or real websites.External grading and Kimi-K3 evaluation send generated task and trajectory text to hosted providers, requiring safeguards for private data.Source: §7 · p. 10
  • The authors release code, benchmark tasks, checkpoint results, and trajectories, but attacked continuations do not share model-server random state and Kimi-K3’s served version may change.The hosted 9B reference is available only as a recorded evaluation log.Source: §7 · p. 10

A BENCHMARK AND TRAJECTORY ANALYSIS

The appendix describes the benchmark’s task composition and analyzes saved episodes across checkpoints. Its trajectory statistics distinguish injection-requested episodes from episodes without a request and document how attacks affect behavior.

A BENCHMARK AND TRAJECTORY ANALYSIS

A.1 COMPOSITION

  • The 150 tasks span ten templates and five skill strata, with low and high difficulty tiers and short or full forms; tasks include protected fields and forbidden controls.They pair 51–76-word goals with initial pages of 18–46 lines, and reference action sequences of 4, 5, 8, or 9 actions.Source: §A.1 · p. 16
  • Table 4 organizes benchmark composition by template and reports high-tier and full-form counts, review requirements, and the reference-action and field values used.Source: Table 4 · p. 16

A.2 TRAJECTORY STATISTICS

  • For requested injections, judged completion is 54.1%, 41.0%, and 46.2% across Adv v1–v3; the attack mean is 41.75% for Base, 46.50% for Capability iter 3, and 49.45% for Robust iter 3.Source: §A.2 · p. 16
  • Episodes without injection requests have judged completion of 73.2%, 76.3%, and 75.4% across Adv v1–v3, while the final clean-to-attacked gap is 31.88 points rather than 23.85.Source: §A.2 · p. 16
  • Clean episodes average 7.41 actions and reach the 12-action budget in 3.7%; learned-adversary episodes use 7.76–8.05 actions and reach the budget in 13.0%–17.3%.The increased action count usually comes from repeated fills or clicks on controls named by an injection; the requested marker is visible in under 0.5% of episodes.Source: §A.2 · p. 16

A.3 CLEAN VERSUS ATTACKED OUTCOMES ON THE SAME TASK AND SEED

Matched clean and attacked episodes show both outcome flips and some verdict changes tied to the executor’s final report. Robust iter 3 reduces clean-good failures against each learned adversary compared with Base.

A.3 CLEAN VERSUS ATTACKED OUTCOMES ON THE SAME TASK AND SEED

  • Of 9,449 resolved matched pairs, 29.1% flip from clean good to attacked bad, while 5.5% flip the other way, measuring rollout variance.Source: §A.3 · p. 16
  • In 13.4% of clean-good-to-attacked-bad flips, actions are identical except for the final message, and the verdict changes with the executor’s report.The passage gives an example in which the executor relays a “Submission blocked” notice after an injected warning.Source: §A.3 · p. 16
  • Among clean-good pairs, attacked episodes fail for Base in 39.8%, 42.4%, and 46.0% of cases against Adv v1–v3, versus 30.6%, 39.1%, and 37.2% for Robust iter 3.Source: Figure 5 · p. 17, §A.3 · p. 17

B CHECKPOINT LEARNING ANALYSIS ON CLEAN TASKS

Checkpoint analysis finds gains and regressions across iterations, with aggregate completion masking uneven performance across task strata and difficulty tiers. The analysis uses saved episodes and reports task- and seed-level comparisons rather than treating all rollouts as independent.

B CHECKPOINT LEARNING ANALYSIS ON CLEAN TASKS

  • The checkpoint analysis uses 3,150 saved clean episodes from seven local checkpoints across 150 tasks and three seeds, without new model calls or changed judge labels.The hosted 9B model is excluded because its trajectories were not transferred.Source: B CHECKPOINT LEARNING ANALYSIS ON… · p. 17

B.1 GAINS AND REGRESSIONS ACROSS ITERATIONS

  • Each update improves and regresses tasks; the final Robust update gains 15 good judgments through 48 bad-to-good and 33 good-to-bad changes on matched task-seed pairs.Stage-1 updates improve 23–28 tasks and regress 19–21; these are judged-trajectory changes, not evidence of acquiring a named skill.Source: §B.1 · p. 17

B.2 STRATA AND DIFFICULTY TIERS

  • Conjunctive selection is hardest at every checkpoint, and high-tier tasks are harder in every condition; Robust iter 3 improves low- and high-tier clean completion over Base.Base completes 83.0% of low-tier and 63.4% of high-tier tasks clean, versus 90.9% and 67.7% for Robust iter 3; several strata decline from Capability iter 3 to Robust iter 3.Source: §B.2 · p. 17

C.1 PER-SEED COUNTS; C.2 VALUES PLOTTED IN FIGURE 4

  • Table 5 reports per-seed episode counts and browser outcomes, while Table 6 supplies plotted Figure 4 values and per-task-seed world-model versus browser comparisons.Table 5 rates average seed fractions before averaging adversaries; matched hosted-model comparisons use nonidentical remaining trajectories and generated attacks.Source: §C.1 · p. 17, Table 5 · p. 18, Table 6 · p. 18

C.3 SIM-TO-REAL TRANSFER OF THE CAPABILITY CHECKPOINTS

Capability checkpoints are tested in a live Chromium browser and against a frontier-model adversary, alongside a Stage-1-removal protocol. Browser results show transfer with important mismatches between judged completion and strict success.

C.3 SIM-TO-REAL TRANSFER OF THE CAPABILITY CHECKPOINTS

  • From Base to Capability iter 3, strict browser success rises most on relational join, ordered repair, and conditional policy, while conjunctive selection passes strict checks in none of 60 episodes.Gains are larger on low-tier and short-form tasks than on high-tier and full-form tasks; two-decimal money amounts remain the weakest field type.Source: §C.3 · p. 19
  • Wrong-value submissions remain the largest browser failure class, while no-submit episodes fall from 16.2% at Base to 2.0% at Capability iter 3.Rejected actions fall from 132 to 67 of 450 episodes, mostly because the browser rejects a fill argument the world model accepts; only 0.2%–2.9% fail solely from rejection.Source: §C.3 · p. 19
  • The world-model judge accepts 7.5%–10.4% of browser trajectories that fail strict checks, and the environments differ in action parsing, observations, and grading.The evaluation does not separate these possible sources of disagreement.Source: §C.3 · p. 19

C.4 FRONTIER-MODEL ADVERSARY

  • Kimi-K3 requests injections in 95.0%–98.0% of episodes, with its median request moving from turn 3 to turn 2 against Robust iter 3.The evaluation keeps executor checkpoints, world model, judge, tasks, and decoding settings unchanged while replacing the learned adversary.Source: §C.4 · pp. 19–20

C.5 STAGE-1 REMOVAL: PROTOCOL

  • The Stage-1-removal ablation compares three Stage-2 rounds on the same tasks against the original learned adversaries, but training histories and retained tasks are not controlled.The ablation protocol also notes that reactive injections may differ across executors at the same task, seed, and adversary checkpoint.Source: §C.5 · p. 20, Table 7 · p. 20

C.6 HISTORICAL TRAINING EXECUTION AND COST

The recorded historical run documents its compute and configuration, while observed feedback failures show that the judge and generated pages can contain errors. The reported feedback examples establish errors but not their prevalence.

C.6 HISTORICAL TRAINING EXECUTION AND COST

  • The historical Stage-1 run used two 96 GB RTX 6000 Pro GPUs and reserved 41.54 GPU-hours for its first iteration, a reservation measure rather than utilization.The run used 150 proposal positions in each of two pools, six assessment rollouts, one curriculum epoch, and two executor epochs; token use, API charges, and active device time were not measured.Source: §C.6 · p. 20

C.7 OBSERVED FEEDBACK FAILURES

  • Structural scans found repeated element identifiers in curriculum and executor-pool pages, and one proposal failed validation; inspected arithmetic errors show reward mistakes but not their frequency.The Stage-1 judge's 1,500-character page cap also omitted filled fields in inspected negative trajectories.Source: §C.7 · pp. 20–21

C.8 FURTHER LIMITATIONS

The paper limits its claims about robustness and evaluation, then specifies how judgments, policy updates, and adversarial training operate. Its extended related-work discussion distinguishes this setup from prior approaches.

C.8 FURTHER LIMITATIONS

  • Three rollout seeds measure rollout variance, not training variance, and the paper reports no confidence intervals or certification of robustness to arbitrary page authors or transfer to browser execution.The hosted 9B reference's served weights and raw trajectories are unavailable for re-running or trajectory-level audit.Source: §C.8 · p. 21

D TECHNICAL DETAILS

  • The technical appendix specifies the update objective, Stage-2 update order, attack interface, and failure handling; the recorded run configuration is in Section C.6.Figure 7 depicts specified difficulty-shaping functions, not measurements.Source: D TECHNICAL DETAILS · p. 21, Figure 7 · p. 21

D.1 ASSESSMENT DETAILS

  • The judge checks the authorized goal against the recorded trajectory, and the validity gate accepts a task only when its estimated success rate exceeds 0.1 and at least one good trajectory has a key.Historical Stage-1 assessments also assign zero after three consecutive identical actions, unlike the Stage-2 judge-only verdict.Source: §D.1 · p. 21

D.2 POLICY UPDATE AND ADAPTER LINEAGE

  • Policy updates mask generated proposal or executor tokens, normalize group advantages, and apply an executor difficulty scale; curriculum replay has no importance correction for older proposals.The update omits GRPO's old-policy likelihood ratio and clipped surrogate, while each invocation initializes a fresh optimizer and saves a new adapter.Source: §D.2 · pp. 21–22

D.3 STAGE 2 UPDATE ORDER

  • Stage 2 screens tasks for clean success, pairs attacks with saved clean controls, updates the adversary, then trains the executor on a mixture of fresh, historical, and clean inputs.The adversary prompt does not see controls or their verdicts, and executor training samples new trajectories rather than reusing assessment rollouts.Source: §D.3 · p. 22

D.4 INJECTION INTERFACE AND PAIRING

  • Injection proposals need a nonempty instruction and marker, but the parser does not check reachability, novelty, or conflict with the goal; only structurally and textually gated observations proceed.Adversarial reward requires a success flip on a saved clean control, and unresolved infrastructure failures cannot earn attack credit.Source: §D.4 · pp. 22–23

D.5 EXTENDED RELATED WORK

  • Compared with ARLAS, AdvSim2Real uses a frozen web world model, success-flip reward, and stored attack inputs relabeled by fresh rollouts rather than earlier attacker checkpoints.The approach uses one injection per trajectory, one 4B adversary and executor, and no refinement stage; its curriculum scores proposals by judged completion while page distribution stays fixed.Source: §D.5 · p. 23

Related papers

Loading 2610.08773v1…