PaperlayerView exact revision on arXiv

Source-linked AI summary

Token Efficient Task Execution via Application Behavior Modeling for Web Agents

Alexandru Ianta, Eleni Stroulia

arXiv:2609.13491v1cs.AI

TL;DR

Web-agent token costs rise as LLMs process complex UI observations, motivating a more token-efficient approach. OdoBot builds an application behavioral model from task trajectories, and on Canvas tasks it uses fewer tokens than Agent-E while outperforming WebVoyager in task success.

  • Problem

    Web agents rely on LLMs to analyze UI observations, while token costs are increasing and depend partly on UI complexity.

    Source: §I · p. 1

  • Method

    OdoBot builds an application behavioral model from task trajectories and uses it to guide task execution.

    Source: §I · p. 1, §III · p. 3

  • Results

    OdoBot uses 900 thousand tokens at 76.4% task success, outperforming WebVoyager; Agent-E has +9% raw task success but requires 77% more tokens per task.

    Source: §VII · p. 9

  • Takeaways & Limitations

    OdoBot reduces runtime token consumption in exchange for upfront model construction, which the paper says is amortized after a relatively small number of tasks.

    Source: §I · p. 1

    OdoBot struggles with highly dynamic content when different elements replace others at the same xpath, a pattern common in quizzes or surveys.

    Source: §VII · p. 9

Abstract

from arXiv · show

The strong performance of AI Agents across an impressive variety of tasks is driving an unprecedented investment in agentic infrastructures, however the cost of processing tokens is fast increasing. Web agents automate the execution of web-application tasks described in natural language, by analyzing the web-application's user interface (UI) and interacting with it. This work introduces OdoBot, a novel web-agent architecture that completes tasks at a fraction of the cost when compared to conventional web agents. This is achieved by leveraging a behavioral model of the underlying application constructed by analyzing successful task-execution demonstrations. Our experiments with 45 tasks on the Canvas Learning Management System (LMS) demonstrate that OdoBot uses 44% and 80% fewer tokens than two state-of-the-art competitor agents (Agent-E and WebVoyager), while also surpassing WebVoyager in terms of task success rate.

I. INTRODUCTION

Web agents use LLMs to interpret UI observations and act on natural-language tasks, while token costs rise with UI complexity and increasingly expensive models. OdoBot uses an application behavior model and broader event observations, and the paper introduces a Canvas-based evaluation dataset.

I. INTRODUCTION

  • Rising token costs make web-agent deployment more expensive, with analysis costs tied to UI complexity and the observation lens.Newer, more powerful LLMs can also require more expensive tokens for the same observation.Source: §I · p. 1
  • OdoBot models application behavior, tracks browser-location and network events, and sends only DOM regions relevant to the current action to an LLM.Execution progress is checked against prior observations rather than by sending execution history to an LLM.Source: §I · p. 1
  • The evaluation compares OdoBot with Agent-E and WebVoyager on 45 tasks in a Canvas LMS environment.The paper presents this as the first web-agent evaluation environment in the teaching domain, to the authors’ knowledge.Source: §I · p. 1

II. BACKGROUND AND RELATED RESEARCH

  • Mind2Web’s offline evaluation requires agents to reproduce recorded trajectories exactly, so valid alternative ways of completing a task can count as failures.The dataset contains trajectories from 137 real-world websites.Source: §II · p. 2
  • The Canvas dataset focuses on tasks that change application state through network requests, enabling deterministic task-success checks.The paper positions the dataset as an evaluation environment for agents interacting with a widely used LMS.Source: §II · p. 2
  • OdoBot reframes execution as matching event sequences, including application-emitted network events, rather than relying only on UI observation-action pairs.The paper distinguishes its behavioral model from agents that retain trajectories in context or retrieve past demonstrations.Source: §II · p. 2

III. MODEL CONSTRUCTION

OdoBot treats web applications as event-producing systems and builds a behavioral model from task trajectories. Execution then becomes navigation through that model to recreate an observed event sequence associated with completing a task.

III. MODEL CONSTRUCTION

  • OdoBot treats user actions and application processes as sources of events that indicate changes in application state.A trajectory is a sequence of events.Source: §III · pp. 2–3
  • The model-construction process has three steps: trajectory collection, graph-model construction, and graph annotation.The graph organizes expected event sequences associated with task completion.Source: §III · p. 3
  • Task execution becomes navigation through the behavioral model to recreate an observed event sequence associated with completing the task.Source: §III · p. 3

A. Trajectory Collection

OdoBot collects task demonstrations as trajectories of user and application events, then uses language-model descriptions to connect those demonstrations with natural-language task requests.

A. Trajectory Collection

  • Tasks are high-level descriptions of operations or outcomes that produce material server-side changes.A task instance adds enough detail for execution, such as a post’s title and content.Source: A. Trajectory Collection · p. 3
  • OdoX records chronological trajectories containing ten event types grouped into user interactions and application events.User events include clicks and data entry; application events include API calls, location changes, and DOM effects.Source: A. Trajectory Collection · p. 3
  • User-interaction events capture filtered DOM snapshots, normalized base paths, and the interacted element’s properties; TinyMCE events also store an editor ID.XPath-linked elements expose button text, input placeholders, and available selection options.Source: A. Trajectory Collection · p. 3
  • Application events record non-GET API calls and GraphQL mutations, plus URL transitions and DOM changes; captured paths replace dynamic segments with wildcards.Only non-GET calls and mutations are retained as network events because they correspond to material server-side changes.Source: A. Trajectory Collection · p. 3
  • An LLM describes trajectory events and summarizes them into imperative, high-level trajectory descriptions that omit task-specific inputs and are stored as vector embeddings.The embeddings use OpenAI’s text-embedding-3-large model and are stored in a vector database.Source: A. Trajectory Collection · pp. 3–4

B. Graph Model Construction

OdoBot builds a graph by merging equivalent events across trajectories and connecting them in observed temporal order, with special handling for variable DOM effects and task-specific parameters.

B. Graph Model Construction

  • Equivalent events share fixed parameters and map to one graph vertex, while directed edges connect vertices in the trajectories’ observed temporal order.Click-event equivalence uses matching base paths, xpaths, and text values.Source: B. Graph Model Construction · pp. 4–5
  • DOM effects are grouped by neighboring events because their element lists and order can vary with factors such as network latency.After a location event, subsequent DOM effects are grouped together as page-loading effects.Source: B. Graph Model Construction · p. 5
  • Graph construction processes non-DOM events first, then DOM effects, and finally adds directed edges in the trajectories’ temporal order.The first pass ensures predecessor and successor vertices are available when DOM-effect vertices are created.Source: B. Graph Model Construction · p. 5
  • Each vertex stores the IDs of its mapped events, so the number of instances indicates how common that event is among collected trajectories.Event IDs combine a trajectory ID and event index.Source: B. Graph Model Construction · p. 5
  • Separating fixed from free event parameters lets OdoBot resolve task-specific values at execution time while reusing observed event patterns.For data entry, the input location is fixed while the value to enter is free.Source: B. Graph Model Construction · p. 5

C. Graph Annotation

After constructing the graph, OdoBot annotates selected click vertices and adds semantic information to graph vertices.

C. Graph Annotation

  • Graph annotation adds properties for execution-time click parameters and vertex semantics, including annotation and dynamicXpath properties.Source: C. Graph Annotation · p. 5

1) Free Parameter Click Events:

OdoBot must identify task-specific click targets among options that are only known at execution time. It detects related HTML structures and encodes targets as dynamic xpaths.

1) Free Parameter Click Events:

  • Some clicks target task-specific elements chosen from options that are themselves known only during execution.Examples include a particular post, menu item, or course card.Source: 1) Free Parameter Click Events · p. 5
  • Because option lists are often built from complex div or span structures rather than li elements, OdoBot relies on visually similar elements being grouped together under a shared HTML parent.The paper bases this assumption on web-design best practices.Source: 1) Free Parameter Click Events · pp. 5–6
  • OdoBot clusters similar local DOM structures and marks matching clicks with a dynamicXpath tuple that identifies the container, child tag, and target’s path within the child.At execution time, this structure is used to retrieve child elements containing the desired click target.Source: 1) Free Parameter Click Events · p. 6

2) Event Descriptions:

OdoBot summarizes user-interaction event descriptions and uses event annotations to describe graph paths in natural language. DynamicXpaths identify click targets within repeated HTML structures.

2) Event Descriptions:

  • Event descriptions for graph vertices are generated by summarizing the descriptions of all events associated with each vertex.These descriptions are generated earlier in Section III-A and added as an annotation property.Source: 2) Event Descriptions · p. 6
  • A dynamicXpath combines a container prefix, child tag, and target suffix to identify click targets in repeated HTML structures.A click annotated with the dynamicXpath can resolve to either identified target, and either target's xpath maps back to that click event.Source: Figure 2 · p. 6
  • Concatenating event annotations along a graph path produces a natural-language description of that path.Source: 2) Event Descriptions · p. 6

IV. MODEL-DRIVEN TASK EXECUTION

OdoBot identifies source and destination states in its application graph, plans a path between them, and executes it while monitoring application events. When multiple paths are available, it uses task descriptions and trajectory overlap to select one.

IV. MODEL-DRIVEN TASK EXECUTION

  • OdoBot executes tasks by identifying the completion vertex and current-state vertex, then following a graph path between them.Source: §IV · p. 6
  • Correct trajectories appear among OdoBot's top-1, top-5, top-10, and top-15 retrieved trajectories in 65.7%, 90.4%, 98.7%, and 99.1% of experimental runs, respectively.OdoBot rewrites each task without specific parameters, embeds it, retrieves up to 15 trajectory descriptions, and selects the best match.Source: §IV · pp. 6–7
  • OdoBot selects the destination from API-call or GraphQL-mutation events associated with the similar task and identifies the source from the browser's current URL.Source: §IV · p. 7
  • For vertices sharing trajectories, estimated distance is the absolute difference between their average event indices; vertices with no shared trajectory are assigned infinite distance.The metric uses events in trajectories containing both vertices to estimate their separation in the graph.Source: §IV · p. 7
  • Beam search ranks candidate vertices by estimated distance, explores the top three, and permits up to five infinite-distance steps to bridge gaps between observed trajectories.Source: §IV · p. 7
  • When multiple paths are found, an LLM weighs natural-language path descriptions and overlap with a similar task's path; execution advances only when live events match the expected event.A mismatch triggers graph re-localization and a recomputed path; execution terminates if no path can be found.Source: §IV · p. 7

V. THE EXPERIMENTAL SANDBOX

The Canvas sandbox was built to support 45 manually curated side-effect task templates across multiple mock courses. Scripts generated task instances and expected API calls, while the environment was populated with validated data and resettable database snapshots.

V. THE EXPERIMENTAL SANDBOX

  • The evaluation sandbox uses Canvas LMS and creates task-relevant entities such as students, courses, instructors, and assignments.Source: §V · p. 7
  • A generation script emits specific task instances and records the expected API call and parameters for task completion.For example, the passage describes a POST request to an assignment-submissions path for a text-submission task.Source: §V · pp. 7–8
  • Table III maps graph actions to the instructions OdoBot uses to resolve them at task time, with Click* denoting clicks annotated by dynamicXpaths.Source: Table III · p. 8
  • Task descriptions were generated from Canvas's 165 student-guide pages, converted into parameterized templates, and manually curated; 45 side-effect tasks remained after exclusions.Excluded examples include third-party tools, file transfers, information-seeking tasks, and actions outside course content.Source: §V · p. 8
  • One seed YAML file was expanded into data for 10 mock courses, supporting 450 task instances—10 for each of the 45 task templates.A Canvas database snapshot permits resetting and initializing the environment without regenerating the data.Source: §V · p. 8
  • Generated course content used gpt-5-mini, with complex entities generated individually and checked against known relationships to flag and regenerate invalid data.Source: §V · p. 8

VI. THE STUDY

The study constructed OdoBot's model from 45 Canvas task trajectories, then compared agents on a separate set of 45 task instances. Each agent completed the comparative evaluation five times.

VI. THE STUDY

  • OdoBot's model was constructed from 45 task trajectories, requiring approximately 10.3 million tokens with gpt-5.4-mini-2026-03-17.Trajectories began from a clean database-snapshot environment, but the environment was not reset between recordings.Source: §VI · p. 9
  • Comparative evaluation used a different set of 45 task instances, with OdoBot, WebVoyager, and Agent-E each completing the tasks five times using gpt-5.5-2026-04-23.Source: §VI · p. 9
  • The evaluation logged network events to determine task success and recorded token use; WebVoyager was also tested at screenshot sizes of 854x480, 1280x720, and 1920x1080.Source: §VI · p. 9

VII. FINDINGS AND DISCUSSION

OdoBot reduces token use through model-guided execution, but its performance is bounded by task-specific modeling and interaction errors. The evaluation also entails substantial task/environment construction effort and costly human demonstrations.

VII. FINDINGS AND DISCUSSION

  • 900 thousand tokens yielded 76.4% task success; OdoBot outperformed WebVoyager’s best (1080p) configuration in raw success, while Agent-E had 9% higher success but required 77% more tokens per task.WebVoyager required 4.6 million tokens, or 511% more; OdoBot’s mean task success and token use are reported across 45 tasks and five runs.Source: §VII · p. 9, Table IV · p. 9
  • OdoBot avoids sending full-state observations to an LLM, instead checking live trajectory events against expected events along its model path.For task-specific choices, it prompts using the task description and filtered HTML elements.Source: §VII · p. 9
  • OdoBot failed 10–12 tasks: five from modeling errors, three from UI-widget interaction, one from uncaptured behavior, one from an unresolved XPath, and 0–2 from stochastic LLM choices.The remaining stochastic errors occurred when the LLM made incorrect choices during task execution.Source: §VII · p. 9
  • Highly dynamic content, including elements replaced at the same XPath, challenges OdoBot; fully mapping possible states requires many trajectories, and mappings cannot rely so heavily on XPath.The passage notes quizzes and surveys as examples of this dynamic-content setting.Source: §VII · p. 9
  • Constructing the real-application Canvas environment and completable test data required substantial effort, so task exclusions constrained the evaluation.Human demonstrations used to collect trajectories are also costly.Source: §VII · p. 10
  • OdoBot focuses on token-use costs, and its reliance on XPath is fragile; decoupling XPath from model construction is proposed as a possible remedy.The paper suggests structural fingerprints combined with element-level screenshots as a possible route to non-XPath identifiers.Source: §VII · p. 10

VIII. CONCLUSION

OdoBot and a Canvas-based testbed are the paper’s two contributions, with OdoBot’s behavioral-model approach substantially reducing token use. Its flexibility is limited when tasks require deviations from past observations.

VIII. CONCLUSION

  • OdoBot takes 15,257.80 fewer tokens per task than Agent-E; construction cost amortizes after 676 tasks against Agent-E and 120 against WebVoyager.The paper proposes a hybrid that passes execution control between OdoBot-style and conventional agents to capture the benefits of both.Source: §VIII · p. 10
  • The Canvas testbed lets web agents interact with a live real application.The paper identifies OdoBot as its most important contribution: a web agent that models application behavior from demonstrations to use LLMs more efficiently.Source: §VIII · p. 10
  • OdoBot struggles where task execution requires flexibility to deviate from past observations.The paper links its token efficiency to using observed behavior to reduce the information needed to reproduce it later.Source: §VIII · p. 10
Loading 2609.13491v1…