PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Retrain, Remix: Adaptation of Web Agents using Mixture Hypernetworks

Md Farhan Ishmam, Minh Pham Dinh, Kenneth Marino

arXiv:2610.22539v1cs.LG

TL;DR

Web agents often lose performance on unfamiliar websites and redesigns, while adapting small agents can require costly interface-specific demonstrations. WEBMIX scouts a target interface and uses its description to compose existing adapters. Across the evaluated domain and version transfers, it matches or outperforms baselines, often matching target-trained specialists.

  • Problem

    Fine-tuned web agents lose performance across websites and redesigns, while adapting small open-weight agents often requires costly new demonstrations and interface-specific adapters.

    Source: §1 · pp. 1–2

  • Method

    WEBMIX uses a scout’s compact interface manual to condition a router that composes adapters from a frozen source-interface bank, without target-side gradient updates.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    WEBMIX matches or outperforms in-context learning and LoRA across evaluated domain and version transfers, often matching target-trained specialists; its TimeWarp domain-transfer gains over the best baseline average 8.3 points for Qwen and 7.5 for Llama.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    An unseen interface can be adapted through exploration and composition rather than retraining, using a scouting visit and a forward pass.

    Source: §5 · p. 9

    The target interface provides no demonstrations or parameter optimization, and the agent may use only a brief, bounded scouting visit before acting.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

The web is not static or uniform. Websites that serve the same purpose can look and navigate very differently. Even within a single site, successive redesigns can change how familiar features are presented and accessed. Humans adapt to this by recognizing familiar interaction patterns and reusing what they already know. We give web agents a similar ability with WebMix, a mixture hypernetwork that adapts them to unseen interfaces by composing previously trained low-rank adapters. WebMix begins with a bank of adapters trained on source interfaces. At deployment, a scout agent explores the target interface and summarizes its observations into a compact manual. A lightweight router uses this manual to predict mixing coefficients over the adapter bank, which combine the existing adapters into a new one for inference. Adapting to a new interface takes only a scouting run and a forward pass, without gradient updates on the target. With Qwen and Llama models, we evaluate domain transfer on TimeWarp and WebArena-Lite-v2, and version transfer on TimeWarp. In all three settings, WebMix matches or outperforms in-context learning and LoRA baselines, often matching the average success of specialists trained directly on those interfaces. On TimeWarp domain transfer, it gains about 8 percentage points on average over the best baseline with both models. These gains do not come at the cost of seen interfaces, where WebMix keeps its lead, and extend to version transfer. Adapting to a new interface can therefore rely on exploration and composition instead of retraining.

1 INTRODUCTION

Web agents struggle to transfer familiar interaction skills across websites and redesigns, while repeated interface-specific fine-tuning requires costly demonstrations. WEBMIX composes existing adapters from a scout’s interface description, and improves transfer across the evaluated settings without target-side gradient updates.

Motivation

  • Fine-tuned agents lose performance across websites and domains, and redesigns can make agents trained on one era brittle on the next.Mind2Web reports a steady performance drop from cross-task to cross-website and cross-domain splits.Source: §1 · pp. 1–2
  • Small open-weight agents typically need new demonstrations and an adapter for each site or redesign, making adaptation costly as sites change.Demonstrations require a person to operate a live site, which may be redesigned before a sufficiently large corpus is collected.Source: §1 · p. 2

Research gap

  • Prior adaptation methods reduce supervision or reuse website knowledge, but still treat each new interface as a separate adaptation problem.Directly generating adapter weights is difficult with few interface conditionings and non-identifiable LoRA factors.Source: §1 · p. 2

Approach

  • WEBMIX composes LoRA adapters trained on familiar interfaces, using a scout’s compact manual to condition a router that mixes the frozen adapter bank.The target adaptation requires scouting and a forward pass, with no further gradient updates.Source: §1 · p. 2

Results

  • On TimeWarp, WEBMIX matches target-trained specialists on average for both models, gains 8.3 points for Qwen and 7.5 for Llama over the best baseline on held-out domains, and improves on versions and WebArena-Lite-v2 domains.It reaches 56.1% success on unseen News with Qwen-3.5 4B, 10.6 percentage points above transferred LoRA.Source: §1 · p. 2

Analysis

  • The paper analyzes held-out imitation risk as bank approximation and routing terms, and ablates composition against direct adapter generation as well as scouting and adapter regularization.Source: §1 · p. 2

2 RELATED WORK

Prior web-agent research studies cross-site and temporal generalization, and adapts agents through demonstrations, exploration, or stored knowledge. WEBMIX instead learns a rule for composing adapters from an interface description.

Web Agents and Interface Generalization

  • Web-agent benchmarks evaluate cross-site and temporal shifts, and report lower accuracy on interfaces unlike those seen in training.TimeWarp tests successive eras of web design; WebArena-Lite-v2 provides domain shifts.Source: §2 · p. 3

Adapting Web Agents to Unseen Interfaces

  • Prior adaptation methods use human demonstrations, agent exploration, or explicit stored knowledge, while WEBMIX uses an interaction-built description to predict an adapter composition.Unlike AutoManual, which conditions planning on an environment description, WEBMIX uses it to predict a composition of existing adapters.Source: §2 · p. 3

Adapter Composition and Generation

  • Adapter reuse combines updates or optimizes composition weights, while hypernetworks generate weights directly; WEBMIX predicts coefficients over a frozen adapter bank.It emits a standard LoRA adapter that an unmodified inference stack can serve.Source: §2 · pp. 3–4

3 WEBMIX: ADAPTATION THROUGH COMPOSITION

WEBMIX adapts a web agent to an unseen interface by scouting it and composing a frozen bank of source-interface adapters. Training is confined to source interfaces; target adaptation uses the scouting information without demonstrations or gradient updates.

3.1 PRELIMINARIES

  • A web task agent reads a goal and page state, emits browser actions, and succeeds when a verifier accepts the final website state.Interfaces may be domains or versions, differing in presentation, navigation, and content.Source: §3.1 · p. 4
  • The target interface has no demonstrations or parameter optimization; before acting, the agent may make only a brief, bounded scouting visit.The setting asks whether source-interface learning can supply the parameters needed for an unseen domain or version.Source: §3.1 · p. 5

3.2 THE MIXTURE HYPERNETWORK

  • WEBMIX’s hypernetwork produces updates for adapted matrices by composing a frozen bank of source-interface LoRA adapters, then adds the resulting update to the frozen base policy.The composed adapter is a standard LoRA adapter served by an unmodified inference stack.Source: §3.2 · pp. 5–6
  • A frozen scout turns an interface visit into a text manual and its normalized sentence-encoder representation, which conditions the adaptation hypernetwork.Only the encoded manual enters the router, which maps it to mixing coefficients.Source: §3.2 · p. 6

3.3 TRAINING AND ADAPTATION

  • Training proceeds through source-interface adapter training, router training on source demonstrations, and one-pass adaptation after scouting the unseen interface.Adding a source interface later adds an adapter and requires router retraining, while existing adapters remain unchanged.Source: §3.3 · p. 6

3.3.1 STAGE 1: ADAPTER BANK TRAINING

  • Each source adapter is fine-tuned on its own demonstrations and anchored on demonstrations from the other source interfaces to limit changes to their base-policy likelihoods.The anchoring strength β controls the penalty; β = 0 recovers plain SFT.Source: §3.3.1 · pp. 6–7

3.3.2 STAGE 2: TRAINING THE ROUTER

  • With the bank frozen, the router is trained by behavior cloning on pooled source responses, with gradients updating only the router parameters.Prompt tokens are masked, so only expert response tokens contribute to the loss.Source: §3.3.2 · p. 7

3.3.3 STAGE 3: ADAPTATION

  • At deployment, the scout visits the unseen interface once and the router produces its adapter in one forward pass, without target demonstrations or gradient updates.For version transfer, the visit performs no tasks; for domain transfer, the base policy attempts training-split tasks and the writer reads their intents and answer-stripped reference procedures.Source: §3.3.3 · p. 7

4 EXPERIMENTS

The experiments test transfer to unseen interfaces, retention on seen interfaces, and design and routing choices across TimeWarp and WebArena-Lite-v2. WebMix transfers better than the baselines, retains strong seen-interface performance, and benefits from composing adapters with scouting and moderate KL regularization.

4 EXPERIMENTS

  • The experiments ask whether WebMix transfers to unseen interfaces, retains seen-interface performance, and benefits from particular design choices.They also examine the adapter bank and router mixtures.Source: §4 · p. 7

4.1 EXPERIMENTAL SETUP

  • The evaluation covers domain transfer on TimeWarp and WebArena-Lite-v2 and version transfer on TimeWarp, using Qwen-3.5 4B and Llama-3.1 8B against zero-shot, in-context learning, and LoRA baselines.Unseen interfaces are excluded from training data and in-context examples; each result averages three runs.Source: §4.1 · p. 7

4.2 RESULTS AND ANALYSIS

  • WebMix beats every baseline across both models and transfer types, with gains of 7.5–8.3 pp on TimeWarp domains, 3.3–4.6 pp on WebArena-Lite-v2 domains, and 1.8–3.2 pp on TimeWarp versions.It often matches interface-trained baselines; transfer is strongest on domains similar to training domains.Source: §4.2 · p. 8, Table 1 · p. 8, Table 2 · p. 8
  • On seen interfaces, WebMix is best or tied-best in every column, improving over LoRA (all) by 4.6–6.3 pp on TimeWarp domains, 1.8–2.4 pp on TimeWarp versions, and 2.6–4.3 pp on WebArena-Lite-v2.Training LoRA on all domains adds only 0.2–1.3 pp over in-domain training.Source: §4.2 · p. 9
  • For Qwen on unseen TimeWarp domains, replacing generated weights with a composed frozen bank adds 3.75 pp, and conditioning the router on the scout adds another 4.58 pp.The scout performs best with one task; β = 1.0 is best across domains and on average, while removing the KL term lowers average success by over 7 pp.Source: §4.2 · p. 9, Table 3 · p. 9, Figure 3 · p. 9
  • The domain adapters are nearly orthogonal, while the router builds held-out versions from nearby versions and assigns 96% of the weight to v6 when it is in the bank.The KL anchor keeps adapters more distinct; without it, domain adapters are about twice as similar.Source: §4.2 · p. 9

5 CONCLUSION

WebMix adapts frozen web policies by scouting an interface and composing existing adapters, without target-side gradient updates. Its appendix explains how update-space comparisons, KL anchoring, and a small router support that approach.

5 CONCLUSION

  • WebMix uses a scout-written interface manual to condition a router that mixes a frozen adapter bank into a standard LoRA adapter, without target-interface gradient updates.Adaptation requires scouting and a forward pass.Source: §5 · p. 9

A.1 ADAPTER FACTORIZATION

  • Adapter factors are non-unique, so WebMix combines adapter updates rather than comparing factor pairs.Reported comparisons use task success or update-space measures, not factor-space distances.Source: §A.1 · p. 15

A.2 KL ANCHOR

  • The KL-style penalty anchors demonstrated-token probabilities without constraining the full output distribution, encouraging interface specialization with less interference when adapters are mixed.It vanishes when the policies assign the demonstrated token the same probability.Source: §A.2 · p. 15

A.3 ROUTER DETAILS

  • The two-layer router has about 0.26M parameters for dc = 1024 and K ≤ 5, and its zero-initialized output starts with a uniform mixture.One coefficient vector is shared across all adapted matrices.Source: §A.3 · p. 15

A.4 TRAINING HYPERPARAMETERS

The appendix specifies the scout, encoder, and training settings used for adaptation, then introduces T2L as a direct weight-generating hypernetwork baseline.

A.4 TRAINING HYPERPARAMETERS

  • The adapter bank uses rank 16 and scale 32, with KL anchor β = 1.0; the router trains for 500 AdamW steps and the scout uses BGE-large-en-v1.5 with dc = 1024.Domain-transfer scouting attempts n = 1 task.Source: §A.4 · p. 15, Table 4 · p. 16

A.5 SCOUT DETAILS

  • For version transfer, the explorer crawls up to 25 pages per site to depth 3 and records page structure and controls; domain-transfer exploration instead replays pages visited while attempting training-split tasks.A frozen language model writes a fixed-schema manual from the exploration record; the frozen sentence encoder mean-pools its tokens.Source: §A.5 · pp. 15–16

B THE T2L BASELINE IN DETAIL

  • T2L is a direct hypernetwork that maps the same conditioning representation as WebMix to the LoRA factors for every adapted matrix.Its architecture and training variants are described in the following appendix sections.Source: B THE T2L BASELINE IN DETAIL · p. 16

B.1 ARCHITECTURE

The appendix describes T2L’s factor-generation architecture and two training variants, then explains how its initialization is scaled to avoid overwhelming the frozen policy.

B.1 ARCHITECTURE

  • T2L queries layer, module, and factor-role embeddings, then uses shared shape-specific output heads to emit LoRA factors in one forward pass.All 2|Ω| queries are batched.Source: §B.1 · p. 16

B.2 RECONSTRUCTOR

  • T2L RECONS trains the generator to regress stored adapter factors using an ℓ1 objective.It trains for 2000 steps with AdamW over 16 randomly drawn matrices per step.Source: §B.2 · pp. 16–17

B.3 SFT

  • T2L SFT trains the same generator end-to-end through the frozen policy using WebMix’s router objective and training signal.Its output heads are initialized so training starts at the base policy.Source: §B.3 · p. 17

B.4 INITIALIZATION AT THE ORACLE’S SCALE

  • Because generated updates scale quadratically with the factor initialization scale, T2L sets σ0 = (ε/√r)^1/2 to keep initial updates below the trained-adapter scale.The near-no-op initialization avoids an exactly zero factor that would stall the other factor’s gradient.Source: §B.4 · p. 17

C ANALYSIS OF THE BANK AND THE ROUTER

The bank’s domain adapters are distinct, while the router’s version mixtures favor interfaces that are nearby or matching. The KL anchor helps keep cross-domain adapters distinct.

The bank

  • Every pairwise cosine similarity between WAL bank domain-adapter updates is below 0.16.With the KL anchor, different-domain adapters stay below 0.14; without it, they are about twice as similar.Source: C ANALYSIS OF THE BANK AND THE ROUTER · p. 17
  • Same-domain adapter similarity rises with depth, reaching about 0.5 for Qwen and 0.6 for Llama in the last layers.The KL anchor keeps different-domain adapters less similar.Source: C ANALYSIS OF THE BANK AND THE ROUTER · p. 17

The router

  • With v1 held out, the router assigns 81% weight to v2 and 17% to v3; with v6 in the bank, it assigns 96% to v6.The coefficients track interface similarity, and for a linearly independent bank they are determined by the served adapter.Source: C ANALYSIS OF THE BANK AND THE ROUTER · p. 17

D THEORETICAL ANALYSIS

This analysis studies the trade-offs of restricting WebMix’s emitted adapter to a convex hull of a frozen bank, with the interface selecting mixture coefficients. It frames held-out behavior-cloning risk against an oracle adapter trained on that interface’s demonstrations.

D Theoretical Analysis

  • Figure 4 compares pairwise domain-adapter similarities with router weights on TimeWarp versions; the router builds held-out v1 mainly from adjacent v2 and v3.When all versions are in the bank, it assigns almost all weight to matching v6.Source: D THEORETICAL ANALYSIS · p. 18
  • Figure 5 examines whether the KL anchor keeps adapters trained on different domains apart.The figure compares same-domain pairs, different-domain pairs with the anchor, and different-domain pairs without it.Source: Figure 5 · p. 18
  • WebMix’s reachable updates are convex combinations of bank adapters, and the mixture map gives the update served under coefficients c.The analysis represents the coefficients as points in the probability simplex.Source: D THEORETICAL ANALYSIS · p. 18
  • The held-out oracle is an adapter trained on that interface’s demonstrations, which WebMix never uses; those demonstrations define the oracle and risk for analysis.The oracle corresponds to the LoRA specialist described in §4.Source: D THEORETICAL ANALYSIS · p. 18

D.1 WHAT THE HULL CAN REACH

The section analyzes what WebMix’s convex hull can reach and how its best held-out risk compares with an oracle. It also states consequences of growing the bank and conditions on a convex-risk variant.

What the hull can reach

  • The held-out risk splits into a bank-and-target approximation term and a router term, with the former bounded by bank geometry and the target.The router term depends on the head and conditioning; a nearest point in the hull and Lipschitz continuity give the approximation bound.Source: §D.1 · p. 19
  • The best mixture is never worse in held-out risk than the best single transferred bank adapter.The theorem assumes finite Lipschitz risk; the bound uses its direction, not the potentially large constant’s value.Source: §D.1 · p. 19
  • The theorem yields untested diagnostics: a transfer matrix, a uniform mixture to isolate routing gains, and hull distance to assess approximation.The experiments do not include these measurements or the proposed hull-distance panel.Source: §D.1 · p. 19
  • Adding an adapter enlarges the reachable set and cannot raise the best achievable risk, though realized risk can worsen through routing.The experiments do not include sequential arrival or the predicted bank-size curve.Source: §D.1 · pp. 19–20
  • In the convex twin, mixture risk is bounded by the corresponding average of bank-adapter risks; applying this to the paper’s multi-matrix adapters is only a directional simplification.A success-level average comparison additionally requires success to be concave in risk.Source: §D.1 · p. 20

D.2 DIRECT GENERATION IN A FROZEN-TRUNK TWIN

The appendix contrasts direct adapter generation with WEBMIX and analyzes properties of the mixture. It also derives risk bounds and conditions under which adding bank adapters cannot worsen them.

D.2 DIRECT GENERATION IN A FROZEN-TRUNK TWIN

  • With linearly independent source features, direct generation in the frozen-trunk twin maps into the bank span using feature-dependent coefficients that need not be simplex-normalized, so its output scale is uncontrolled.By contrast, WEBMIX's mixture does not emit an update larger than the largest bank adapter.Source: §D.2 · pp. 20–21

D.3 WELL-POSEDNESS OF THE MIXTURE

  • WEBMIX's objective depends on induced model updates rather than factor coordinates, and a linearly independent bank makes its mixture coefficients identify the served adapter; factor-space loss can vary across equivalent factors.The Gram matrix of bank updates defines the distance between generated adapters through their coefficient differences.Source: §D.3 · p. 21

E.1 PROOF OF THEOREM 1

  • Theorem 1 decomposes held-out risk into approximation and nonnegative routing terms, with the mixture's risk no greater than that of its best bank adapter.The approximation term is tied to distance from the target update to the mixture set.Source: §E.1 · p. 22

E.2 PROOF OF COROLLARY 2

  • Adding an adapter enlarges the mixture set, so the minimum distance and minimum risk cannot increase.The proof embeds the old simplex in the expanded simplex by assigning zero weight to the added adapter.Source: §E.2 · p. 22

E.3 PROOF OF COROLLARY 3

  • When risk is convex on the update set, a mixture's risk is at most the corresponding weighted average of bank risks; affine output logits provide one stated condition for convexity.The affine-logit condition makes per-token negative log-likelihood convex in the update.Source: §E.3 · p. 22

E.4 PROOF OF PROPOSITION 4

The appendix proves a closed-form characterization and convergence result for direct generation in a frozen-trunk twin, then examines scale control and gauge invariance. It also records the paper's LLM-use and reproducibility statements.

E.4 PROOF OF PROPOSITION 4

  • For full-rank source features, the minimum-norm interpolating head is BΞ+, and sufficiently small-step gradient descent from zero converges to it.Its output at any conditioning is a linear combination of the stored bank updates, with coefficients determined by the feature map.Source: §E.4 · pp. 22–23

E.5 PROOF OF COROLLARY 5

  • In the frozen-trunk twin, scaling a conditioning feature by t scales its generated update by t, so the output norm is unbounded as t grows for a nonzero bank update.The passage contrasts this feature-controlled scale with WEBMIX's bounded mixture scale.Source: §E.5 · p. 23

E.6 PROOF OF PROPOSITION 6

  • Re-gauging LoRA factors leaves their product, the induced likelihood, and WEBMIX's router loss unchanged, whereas an ℓ1 loss on factors can vary for the same target model.The factor-space loss example uses equivalent targets whose products are unchanged.Source: §E.6 · p. 23

DECLARATION OF LLM USAGE

  • The authors report using LLMs to polish the draft, brainstorm ideas, orchestrate experiments, verify proofs in Lean, and review code, while accepting responsibility for the paper's content and correctness.Source: DECLARATION OF LLM USAGE · p. 23

REPRODUCIBILITY STATEMENT

  • The reproducibility statement says supplementary materials include the source code and Lean 4 formalization, and that datasets are publicly available and cited.The code maps numbered Appendix D results to their Lean declarations.Source: REPRODUCIBILITY STATEMENT · p. 23

Related papers

Loading 2610.22539v1…