PaperlayerView exact revision on arXiv

Source-linked AI summary

FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web Agents

Imene Kerboua, Sahar Omidi Shayegan, Megh Thakkar, Xing Han Lù, Léo Boisvert, Massimo Caccia, Jérémy Espinas, Alexandre Aussem, Véronique Eglin, Alexandre Lacoste

arXiv:2510.03204v2cs.CL

TL;DR

Web agents face long observations that raise processing costs and prompt-injection risks. FocusAgent uses a lightweight LLM to select task-relevant AxTree lines; it reduces observation size by over 50% on average, and its defense variant sharply lowers pop-up attack success.

  • Problem

    Web-agent AxTrees can exceed tens of thousands of tokens, making processing costly and exposing agents to prompt-injection risks.

    Source: §1 · p. 1

  • Method

    FocusAgent uses a lightweight LLM retriever to select task-relevant AxTree lines using the task goal and current observation before action prediction.

    Source: §3 · p. 4

  • Results

    FocusAgent reduces observation size by over 50% on average with modest degradation from full-observation performance; pop-up-attack ASR fell from over 80% to less than 1% for both models.

    Source: §1 · p. 2, §5.3 · p. 9

  • Takeaways & Limitations

    Retrieval can remove prompt-injection threats while preserving consistent performance in attack-free settings.

    Source: §5.3 · p. 9

    DefenseFocusAgent does not nullify attacks, and pop-ups can block actions, leaving task success lower than in no-attack settings.

    Source: §5.3 · p. 10

Abstract

from arXiv · show

Web agents powered by large language models (LLMs) must process lengthy web page observations to complete user goals; these pages often exceed tens of thousands of tokens. This saturates context limits and increases computational cost processing; moreover, processing full pages exposes agents to security risks such as prompt injection. Existing pruning strategies either discard relevant content or retain irrelevant context, leading to suboptimal action prediction. We introduce FocusAgent, a simple yet effective approach that leverages a lightweight LLM retriever to extract the most relevant lines from accessibility tree (AxTree) observations, guided by task goals. By pruning noisy and irrelevant content, FocusAgent enables efficient reasoning while reducing vulnerability to injection attacks. Experiments on WorkArena and WebArena benchmarks show that FocusAgent matches the performance of strong baselines, while reducing observation size by over 50%. Furthermore, a variant of FocusAgent significantly reduces the success rate of prompt-injection attacks, including banner and pop-up attacks, while maintaining task success performance in attack-free settings. Our results highlight that targeted LLM-based retrieval is a practical and robust strategy for building web agents that are efficient, effective, and secure.

1 Introduction

Web-agent observations can be costly and risky to process, while retrieval methods may miss stateful context or discard useful information. FocusAgent uses a lightweight LLM to select task-relevant AxTree lines, aiming to reduce context size and injection risk.

1 Introduction

  • Web-page AxTrees can still exceed tens of thousands of tokens, increasing processing cost and prompt-injection risk even after reducing content about 10× versus the DOM.Typically, only a fraction of a page is relevant to the task goal.Source: §1 · p. 1
  • Semantic-similarity retrieval can miss prior-action consequences and page state, while truncation can discard essential context.The introduction also notes that defensive layers may trade off security and attack-free performance.Source: §1 · pp. 1–2
  • FocusAgent reduces observation size by over 50% on average, with modest degradation from full-observation performance.The authors also report threat removal while maintaining task completion performance in attack-free settings.Source: §1 · p. 2
  • The approach uses a smaller LLM to select observation lines relevant to navigation decisions, guided by task goals and optionally action history.The paper also frames retrieval as a security feature that reduces attack success while maintaining relevant overall performance.Source: §1 · p. 2

2 Related Work

Related work covers observation representations, retrieval and pruning, and web-agent security. FocusAgent differs by pruning AxTrees with planning context and user intent, while treating efficiency and security as complementary motivations.

Observation Processing in Web Agents

  • DOM-based approaches retrieve or rerank chunks, while FocusAgent starts from the already more compact AxTree and reduces it further.The paper notes that extracting an AxTree reduces page content by roughly 10× compared with the full DOM.Source: §2 · p. 3
  • Traditional embeddings may struggle with interactive elements, planning, and user goals; FocusAgent filters observations using planning context and user intent.AxTrees can grow as web applications become more complex, increasing context demands and processing costs.Source: §2 · p. 3

Retrieval in Web Agents

  • This paper retrieves from current observations rather than retrieving prior trajectories for in-context learning, and frames pruning as complementary to long-context reasoning.The passage identifies cost, security, and weaker backbones as settings where pruning is useful.Source: §2 · p. 3

Agents Safety

  • Web-agent security research has moved toward stateful, end-to-end evaluations, where reported gaps include high attack success and defenses vulnerable to adaptive attacks.The passage motivates retrieval-based observation sanitization as a way to neutralize threats without sacrificing task completion.Source: §2 · pp. 3–4

3 FocusAgent

FocusAgent applies a lightweight LLM retriever to each observation before the agent acts. It uses task context to select AxTree lines, removes irrelevant lines, and passes the pruned observation to the agent.

System Architecture

  • The retrieval illustration assigns AxTree line numbers, identifies relevant ranges, then removes irrelevant lines and inserts a placeholder for the number pruned.The caption describes a task-goal prompt and, where applicable, interaction history.Source: Figure 2 · p. 4
  • At each episode step, a lightweight LLM filters the observation before it reaches the acting agent.The prompt includes the task goal, numbered observation lines, optional interaction history, and instructions for returning line spans.Source: §3 · p. 4
  • The selected line ranges are post-processed into a reduced representation that is passed to the agent for planning and task completion.Figure 2 depicts this retrieval process for a WebArena task at step 2.Source: §3 · p. 4, Figure 2 · p. 4

Retrieval Strategy

  • The retriever uses the task goal and current observation without history, and soft prompting favors recall when relevance is uncertain.Longer AxTrees could be processed in sequential chunks, though experiments did not encounter prompts exceeding the models’ 128k-token context length.Source: §3 · p. 5

4 Experiments

Experiments compare FocusAgent with retrieval and truncation baselines on WorkArena L1 and WebArena, evaluating success, pruning, and cost. Results emphasize preserving useful page context and the importance of retriever capability, not simply maximizing compression.

4.1 Benchmarks

  • Experiments use BrowserGym to evaluate WorkArena L1, focused on routine knowledge-work tasks, and WebArena, which has 812 tasks; the BrowserGym test split is a 381-task WebArena subset.Source: §4.1 · p. 5

4.2 Modeling

  • The modeling compares bottom truncation, BM25, embedding retrieval, and FocusAgent with multiple retriever and backbone models under standardized agent and retriever context limits.The agents use a maximum of 40k context tokens, except GenericAgent-4.1 (5k), and the retriever limit is 128k tokens.Source: §4.2 · pp. 5–6

4.3 Metrics

  • Evaluation measures success rate with standard error, observation pruning relative to original token count, and total experiment cost in USD.The cost calculation accumulates agent LLM calls; the reported input-token prices vary by model, and local Qwen3-embed-8b has no applicable cost.Source: §4.3 · p. 6

4.4 Results and Discussion

  • Embedding and BM25 methods reach 42.4%–53.3% success on WorkArena L1, while FocusAgent reaches 43.9%–53.2%; chunk size and retriever capability affect these outcomes.A 400-token chunk yields 30% pruning, compared with 61% for FocusAgent; harder tasks such as sorting and filtering typically remain unsolved regardless of observation size.Source: §4.4 · p. 6
  • Results suggest that retained context matters more than the amount pruned: goal-matched BM25 and embedding retrieval can discard page-understanding chunks, while FocusAgent avoids this failure without chunk-size tuning.The authors hypothesize that needed chunks may not be named in the task goal; stronger retriever models also achieve better pruning and task success.Source: §4.4 · pp. 6–7
  • Pruning effectiveness does not necessarily track observation token count; the retained AxTree must preserve representational information without becoming too abstract to convey page state.The authors note that further pruning analysis appears in Appendix E.Source: §4.4 · pp. 7–8
  • FocusAgent prunes 51%/50% on WorkArena L1 with approximately 19%/16% cost reductions, and 59%/51% on WebArena with 26%/27% savings for GPT-4.1/Claude-3.7-Sonnet, respectively.On WorkArena L1, the reported success-rate drop is 1.5 percentage points; Qwen3-235b-a22b improves from 27% to 34.2% with a Qwen3.5-9B retriever.Source: Table 1 · p. 7, Table 2 · p. 8, §4.4 · p. 8

5 FocusAgent and Security

FocusAgent’s security evaluation tests whether retrieval can remove malicious content while preserving task-relevant observations, using URL-injection and similar-goal attacks. Results show that retrieval-based defense can sharply reduce attack success, but attacks and defense choices can still impair task completion.

5 FocusAgent and Security

  • The security study tests retrieval as a way to remove attacks while preserving the original task, using DoomArena and additional attacks designed to stress-test similarity defenses.The prior guard-layer workflow can stop tasks when an attack is detected, while false positives can reduce performance without attacks.Source: §5 · p. 8

5.1 Attacks

  • The evaluation includes URL-redirection banner and pop-up attacks, plus banner attacks that divert agents to similar task goals drawn from WebArena templates.The reported URL-injection experiments focus on text-only attacks.Source: §5.1 · p. 9

5.2 Experiments

  • Four agent configurations are compared using Attack Success Rate and Task Success Rate on the WebArena Reddit subset.The configurations include base agents, a guard-layer agent, and agents with attack-warning prompts; Table 3 covers the Reddit subset.Source: §5.2 · p. 9, Table 3 · p. 10

5.3 Results and Discussion

  • On pop-up attacks, DefenseFocusAgent reduced ASR from over 80% to less than 1% for both models, while task success increased slightly.The result is reported for GPT-4.1 and Qwen3-235b-a22b; the WebArena Reddit evaluation is summarized in Table 3.Source: §5.3 · p. 9, Table 3 · p. 10
  • Pop-ups can block task actions even when ASR is low; showing the close button risks exposing its injection, and higher TSR can come with an ASR cost.The authors suggest exploring removal of injected text from the attack element before presenting it to the agent.Source: §5.3 · p. 10
  • For pop-up attacks, DefenseGenericAgent had a 73.4% ASR versus 1.0% for GPT-4.1 DefenseFocusAgent; Claude-Sonnet-3.7 task success fell to 20.0% from 58.8% without attacks.Even unsuccessful attacks against Claude-3.7-Sonnet reduced task success, and the authors argue for tiered defenses.Source: §5.3 · p. 11

6 Ablation Study

The ablations test retrieval prompting, use of interaction history, and the format of the pruned AxTree. Soft prompting and omitting history are favored, while formatting reveals a trade-off between pruning and task performance.

Soft Retrieval Prompting is Best

  • Soft retrieval prompting performs best overall: aggressive prompting prunes more but hurts performance, while neutral prompting collapses on WebArena despite good WorkArena L1 results.The authors emphasize specifying how the retriever should handle uncertainty.Source: §6 · pp. 11–12

Streamlining the Retriever by Dropping History

  • The retriever performs better without interaction history, with the authors hypothesizing that prior actions and thoughts can disturb retrieval from the current AxTree.They suggest the current AxTree lets the retriever understand task progress without history.Source: §6 · p. 12

AxTree Structure and Pruning-Performance Trade-off

  • Removing irrelevant lines yields the most pruning, but keeping their bid and role yields the highest performance; pruning reaches about 99% when lines are removed, versus at most 56% or 64% with bid or role retained.The authors note that keeping irrelevant-line metadata would not be cost-efficient because the retriever processes the full AxTree as well as the agent-restricted tree.Source: §6 · p. 12

7 Conclusion

FocusAgent uses a lightweight LLM to prune AxTree observations, reducing their size by up to 50% with minor performance degradation relative to using the full AxTree. Experiments across two benchmarks and two backbone models support its generalizability, while retrieval can eliminate some threats without fully preventing attacks.

7 Conclusion

  • FocusAgent reduces AxTree size by up to 50% with minor performance degradation relative to using the full AxTree.The paper reports experiments on two benchmarks using two backbone models and describes retrieval as eliminating threats under certain attack types while not completely preventing attacks.Source: §7 · p. 12

A Limitations

The system has limitations in prompt dependence, residual malicious content, and evaluation scope.

A Limitations

  • Performance depends heavily on prompt engineering and evolving LLMs; removed attacks may leave malicious content, and contexts longer than 128k tokens remain untested.The benchmarks did not include contexts longer than 128k tokens.Source: A Limitations · p. 17

B Failure Modes Analysis

FocusAgent’s failure analysis identifies recurring retrieval and state-tracking gaps, while its retrieval-model comparisons show performance close to full-observation agents in some settings. Failures vary by task and backbone, including loops, missed chart data, wrong-page navigation, and premature termination.

B Failure Modes Analysis

  • Loop traps dominate the analyzed failures: repeated click, wait, and scroll sequences account for roughly 9 of 15 GPT-4.1 cases and 10 of 18 Claude cases.In order-ipad-mini, the retriever omits the Preview button, so the agent repeatedly scrolls instead of opening the configuration panel.Source: Table 6 · p. 17, Table 7 · p. 17, B Failure Modes Analysis · p. 18
  • For Claude on multi-chart-min-max-retrieval, FocusAgent misses chart values in image alt attributes or ARIA labels and exhausts 15 steps, while GenericAgent solves all four seeds in one step.This failure cluster is not reported for GPT-4.1 on the same task type.Source: B Failure Modes Analysis · p. 18
  • Wrong-page navigation occurs in 2–3 cases per model, and some agents continue interacting before recognizing the page mismatch.In order-sales-laptop, the agent submits the correct item but sees a different product on the confirmation page; it has already declared success.Source: B Failure Modes Analysis · p. 18
  • Five cases end in premature termination after required fields remain unfilled or the final submission button is not clicked.Some agents treat an empty result set or confirmation-like text as completion rather than checking explicit task criteria.Source: B Failure Modes Analysis · p. 18

C Retrieval Models

  • With GPT-4.1, FocusAgent using small models reaches SRs of 51.5% and 51.8%, versus 53.0% for a large model with the full tree; Claude-Sonnet-3.7 degrades by 4 points.The authors note that the pruning rate is higher for GPT-5-mini and suggest a more capable small model may better handle pruning while maintaining performance.Source: C Retrieval Models · p. 18
  • FocusAgent(4.1-mini) performs better than GPT-4.1-mini alone with the full tree, which has an SR of 47.9%.The comparison is reported for Table 10.Source: C Retrieval Models · p. 18

D Latency Analysis

FocusAgent’s two-call design adds retriever latency while leaving the backbone call latency virtually unchanged. The section also reports benchmark comparisons and describes the latency measurement procedure and a pruned-tree example.

D Latency Analysis

  • Latency measurements time each LLM call, separately instrumenting FocusAgent’s retriever and agent and aggregating per-step latency over 33 WorkArena L1 tasks.The reported statistics include mean and 95th-percentile latency and the retriever’s share of overhead.Source: D Latency Analysis · pp. 18–19
  • With a GPT-4.1-mini backbone, FocusAgent achieves higher success rates than GenericAgent on both benchmarks.The comparison is reported in Table 10.Source: Table 10 · p. 19
  • The retriever adds a mean of 7.6 s per step and up to 14.2 s at the 95th percentile; FocusAgent totals 10.1 s versus GenericAgent’s 2.5 s.The backbone LLM mean latency is 2.5 s for both systems; this is the primary practical limitation attributed to the two-call design.Source: D Latency Analysis · p. 19, Table 11 · p. 20

E.1 Pruned AxTree Examples

  • The appendix includes an example of a pruned AxTree processed by FocusAgent during task completion.Figure 6 illustrates the pruned observation supplied to the agent.Source: §E.1 · p. 20

E.2 WorkArena Pruning Distributions

Pruning patterns vary by task type in WorkArena and by website in WebArena. The accompanying analysis relates those patterns to task structure and reports that higher pruning can increase savings while reducing success rates for retrieval baselines.

E.2 WorkArena Pruning Distributions

  • WorkArena pruning clusters vary by task: sort tasks are pruned around 80%, filter tasks around 20%, and form, order, and chart tasks between 20% and 80%.The analysis associates high pruning for sort tasks with removal of unnecessary tables, and describes filter tasks as hard.Source: §E.2 · p. 20

E.3 WebArena Pruning Distributions

  • WebArena DBSCAN identifies 9 clusters and 288 noise points, with the highest pruning rates in Shopping Admin, Gitlab, Map, and Reddit tasks.Tasks on Shopping alone did not receive much pruning.Source: §E.3 · p. 21, Table 12 · p. 21

G BM25 and Embeding Retrieval Agents

  • Embedding and keyword retrieval baselines use fixed chunk counts and sizes, whereas the LLM retriever selects observation sizes dynamically; higher pruning yields greater savings but lower success rates.The baselines are capped at 2,000 and 4,000 tokens, based on FocusAgent’s median and mean observation sizes of 1,980 and 3,833 tokens.Source: G BM25 and Embeding Retrieval Agents · p. 23

H.1 General Estimation

  • Cost efficiency is achieved when observation size is reduced by at least 20% under the stated model-price assumptions.The estimate excludes API latency and full prompt processing and uses prices of 0.4 USD/1M tokens for GPT-4.1-mini and 2 USD/1M tokens for GPT-4.1.Source: §H.1 · pp. 23–25

J Ablation Study

The appendix presents prompt variants and examples illustrating different AxTree pruning strategies, including cases with very high token reduction. It also includes examples of retriever outputs and baseline observations.

J.1 Prompts

  • The appendix shows the regular soft-prompting instruction alongside aggressive and neutral prompting variants.The regular prompt is identified as the FocusAgent prompt; the two alternatives are shown as ablations.Source: §J.1 · pp. 26–27, §J.3 · p. 28

J.2 Examples of AxTrees with Different Strategies

  • Three WorkArena examples illustrate removing irrelevant lines, retaining only their role, or retaining both bid and role.The examples correspond to multi-chart-value-retrieval, create-hardware-request, and create-incident tasks.Source: §J.2 · p. 27, Figure 28 · p. 39, Figure 29 · p. 39

J.3 Max AxTree Pruning Examples

  • The appendix includes examples of AxTrees pruned by 96% on WorkArena L1 and by 99% on WorkArena L1 and WebArena.It also shows retriever chain-of-thought examples, EmbeddingAgent and BM25Agent observation chunks, and GenericAgent observation flags.Source: §J.3 · p. 27, Figure 14 · p. 29, Figure 15 · p. 30

Related papers

Loading 2510.03204v2…