PaperlayerView exact revision on arXiv

Source-linked AI summary

Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale

Tianyue Ou, Frank F. Xu, Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, Shuyan Zhou

arXiv:2409.15637v2cs.AI

TL;DR

Digital agents lack large-scale direct demonstrations, while collecting them through environments or human annotation is difficult and costly. Synatra converts indirect knowledge into direct trajectories, and its trained agent performs strongly across web-task benchmarks at substantially lower demonstration cost.

  • Problem

    Digital agents lack readily available large-scale direct demonstrations, while automatic collection requires environment setup and human annotation is costly and can have limited scope.

    Source: §1 · pp. 1–2

  • Method

    Synatra uses an LLM to transform indirect knowledge, including tutorials and web-page observations, into direct action trajectories for training.

    Source: §1 · p. 2

  • Results

    6.28% of WebArena tasks were completed by Synatra-CodeLlama versus none by CodeLlama-instruct-7b; overall it surpassed comparable-size models on three web benchmarks and GPT-3.5 on Mind2Web and WebArena.

    Source: §1 · p. 2, §6.1 · p. 7

  • Takeaways & Limitations

    Synthetic demonstrations cost approximately 3% as much as human demonstrations and can outperform an equal quantity of human demonstrations collected from limited domains on WebArena.

    Source: §1 · p. 2

    The method was tested in limited settings and depends on a strong, potentially costly and slow LLM for generation and on source web-page data quality.

    Source: §8 · p. 11

Abstract

from arXiv · show

LLMs can now act as autonomous agents that interact with digital environments and complete specific objectives (e.g., arranging an online meeting). However, accuracy is still far from satisfactory, partly due to a lack of large-scale, direct demonstrations for digital tasks. Obtaining supervised data from humans is costly, and automatic data collection through exploration or reinforcement learning relies on complex environmental and content setup, resulting in datasets that lack comprehensive coverage of various scenarios. On the other hand, there is abundant knowledge that may indirectly assist task completion, such as online tutorials that were created for human consumption. In this work, we present Synatra, an approach that effectively transforms this indirect knowledge into direct supervision at scale. We define different types of indirect knowledge, and carefully study the available sources to obtain it, methods to encode the structure of direct demonstrations, and finally methods to transform indirect knowledge into direct demonstrations. We use 100k such synthetically-created demonstrations to finetune a 7B CodeLlama, and demonstrate that the resulting agent surpasses all comparably sized models on three web-based task benchmarks Mind2Web, MiniWoB++ and WebArena, as well as surpassing GPT-3.5 on WebArena and Mind2Web. In addition, while synthetic demonstrations prove to be only 3% the cost of human demonstrations (at $0.031 each), we show that the synthetic demonstrations can be more effective than an identical number of human demonstrations collected from limited domains.

1 Introduction

Synatra addresses the shortage and collection difficulty of direct digital-agent demonstrations by converting indirect human-facing knowledge into trajectories, then uses those demonstrations to improve web-task agents at scale.

1 Introduction

  • Reliable digital agents lack readily available large-scale demonstrations, while environment setup, limited benchmark scope, and costly human annotation constrain existing collection approaches.Some demonstrations require access to real accounts and histories, such as a legitimate subscription for canceling a PayPal order.Source: §1 · pp. 1–2
  • Synatra uses an LLM to turn indirect knowledge, including human-oriented tutorials, into trajectories specifying actions for concrete observations.Tutorials provide procedural guidance but do not specify executable actions or their associated observations.Source: §1 · p. 2
  • The approach can scale with human-created indirect knowledge rather than relying on direct human annotations or LLM-generated trajectories.Source: §1 · p. 2
  • Synatra-CodeLlama surpassed comparable-size open-source models on Mind2Web, MiniWoB++, and WebArena, and was more effective than an equal quantity of limited-domain human demonstrations on WebArena.The synthetic examples cost approximately 3% as much as human-annotated demonstrations.Source: §1 · p. 2

2 Problem Formulation

The paper models browser-based agents as choosing actions from task context and observations, and distinguishes direct action demonstrations from three forms of indirect knowledge.

2.1 Controlling Digital Agents through Natural Language

  • A digital agent operates in an environment defined by states, actions, observations, and transition dynamics; this work uses a browser as the entry point to web applications.Source: §2.1 · p. 2
  • The action space follows WebArena’s universal keyboard-and-mouse operations, while each web application determines its own states and action effects.Source: §2.1 · p. 3
  • At each step, the agent chooses an action from the task intent, current observation, and previous actions, continuing until it predicts a stop action.Source: §2.1 · p. 3
  • A benchmark scores the final environment state as 1 when the desired goal is satisfied and 0 otherwise.Source: §2.1 · p. 3

2.2 Definition of Direct Demonstrations and Indirect Knowledge

  • A direct demonstration is a state paired with its expected next action, enabling an agent to learn action prediction from a given state.Source: §2.2 · p. 3
  • Procedural knowledge gives task steps that are not grounded in a particular observation or tied to a specific action space.A tutorial may say to log in without specifying the concrete page or fields.Source: §2.2 · p. 3
  • The other indirect-knowledge types are environment knowledge about hypothetical action effects and ungrounded observations that lack an associated task or trajectory.Source: §2.2 · p. 3

3 Scalable Demonstration Synthesis for Digital Agents

The synthesis pipeline encodes trajectories as structured programs, transforms tutorials and sampled web pages into action-and-observation examples, and filters generated samples for completeness and responsiveness.

3.1 Trajectories as Programs

  • Trajectories use Python functions that combine natural-language planning comments with executable action calls, separating task-level decomposition from action-level goals.An action summary is also added to future prompts because detailed reasoning about a current step may not be relevant later.Source: §3.1 · pp. 3–4

3.2 Synthesizing from Text Procedural Knowledge with Generative Environment

  • Because tutorials are ambiguous, abstract, and ungrounded, the LLM generates intermediate HTML observations reflecting prior actions and exposing the target element for the next action.The observation should reflect past actions and contain the elements needed to perform the next action.Source: §3.2 · p. 4
  • For tutorial-based synthesis, the LLM rewrites procedural steps into a hypothetical program trajectory by concretizing a scenario, parsing actions, and grouping them into subtasks.The tutorial source is filtered to GUI-navigation articles, yielding 25k wikiHow articles for synthesis.Source: §3.2 · p. 4

3.3 Synthesizing from Random Observations

  • ClueWeb pages are sampled with temperature T = 0.6 to up-sample more interactive domains while retaining diversity among rarer domains.Source: §3.3 · p. 5
  • From a sampled page, the model brainstorms domain-relevant task categories, develops selected categories into scenarios, and generates prior actions and a grounded next action.The target action includes an action, its target element in the accessibility tree, and a natural-language summary.Source: §3.3 · p. 5

3.4 Data Filtering

  • Filtering first checks sample completeness and coherence, then removes actions predicted to leave the environment state unchanged.Source: §3.4 · p. 5

4 Data Statistics

The data statistics describe trajectory length and the coverage of generated task intents and page observations, including comparisons with Mind2Web.

4 Data Statistics

  • The statistics treat history length as an indicator of task complexity and task objectives and accessibility trees as measures of data-domain diversity, with Mind2Web shown as a human-collected comparison.Source: §4 · pp. 5–6
  • Figure 2 compares task-intent embeddings on the left and accessibility-tree embeddings on the right; random-observation intents cover more broadly than tutorial-derived intents.Source: Figure 2 · p. 6, §4 · p. 6
  • Most synthetic trajectories have fewer than eight history steps, while the longest exceed 20 and represent more complex scenarios.Source: §4 · p. 6
  • Despite independent synthesis, generated intents substantially cover Mind2Web tasks, and tutorial-generated observations overlap significantly with real pages from random observations and Mind2Web.Source: §4 · p. 6
  • The average end-to-end cost per valid sample is $0.031, including $0.028 for generation and $0.003 for prediction-based filtering.Source: §4 · p. 6

5 Experimental Setup

The experiments finetune CodeLlama-7b on web-navigation instructions and evaluate it on three web-navigation benchmarks against 12 baseline models. The setup uses a shared general prompt for most baselines, with original prompts retained for two MiniWoB++ models.

5 Experimental Setup

  • 99,920 web-navigation instruction examples from WikiHow articles and ClueWeb pages were used to finetune CodeLlama-7b.The model was selected after comparing several model families on Mind2Web’s train and test sets.Source: §5 · p. 6
  • Three web-navigation benchmarks were evaluated: Mind2Web, MiniWoB++, and WebArena.Mind2Web uses overall step accuracy across held-out categories; MiniWoB++ reports the average score across five runs, while MiniWoB++ and WebArena use outcome-based programmatic checkers.Source: §5 · p. 7
  • The comparison includes 12 baselines, with general web-navigation instructions and no dataset-specific explanations in the shared prompt.AgentLM and CodeActAgent use their original MiniWoB++ prompts, with task-specific in-context examples removed for zero-shot comparison.Source: §5 · p. 7

6 Results

Synatra-CodeLlama leads comparable-size models across three web benchmarks. The analyses examine its error patterns and compare synthetic demonstrations with human annotations, trajectory formats, knowledge sources, and tutorial retrieval.

6.1 Main Results

  • Synatra-CodeLlama leads comparable-size models across all three benchmarks; Mind2Web rises 6.62% to 15.85%, MiniWoB++ 23.04% to 38.20%, and WebArena goes from no completed tasks to 6.28%.It also outperforms GPT-3.5 on Mind2Web and WebArena, and Lemur-chat-70b across all three benchmarks.Source: §6.1 · p. 7, Table 1 · p. 8
  • On held-out benchmarks, Synatra-CodeLlama uses generic web tasks without dataset artifacts and outperforms interactive-data models whose Regex matching and selected examples limit complex-page use.The reported Regex limitation applies to AgentLM and CodeActAgent; the authors say pattern matching is prohibitive on complex Mind2Web and WebArena pages.Source: §6.1 · pp. 7–8

6.2 Analysis

  • Synthetic-data training reduces non-existent-element actions from about 95% for CodeLlama-instruct-7b to 0.7%, and makes Synatra-CodeLlama 74% less likely than GPT-4-turbo to repeat form-field input.It also reduces invalid interactions with non-clickable or non-typable elements to under 6%, versus over 15% for Llama3-chat-8b.Source: §6.2 · p. 8, Table 2 · p. 9
  • On MiniWoB++, human annotations improve performance by 3.96% versus Synatra's 17.81%; on WebArena, human-only fails while Synatra improves 4.56%.The authors hypothesize that task diversity partly explains the gap; they presume human demonstrations are higher quality but require meticulous design and control.Source: §6.2 · p. 9

6.3 Ablations

  • WebArena success rates steadily increase as synthetic training data scales from 18k to 100k samples, with other parameters held constant.The section reports the trend but does not give the individual success rates.Source: §6.3 · p. 9
  • Converting 30k trajectories from program format to natural language format reduces performance on both MiniWoB++ and WebArena.The authors suggest programs may naturally represent action sequences as API calls and embed chain-of-thought reasoning in code comments.Source: §6.3 · p. 9
  • Training on either tutorials or sampled web pages alone degrades performance versus mixing both sources, which combine procedural precision with realistic observations.Tutorial procedures are human-written and verified, while sampled-page observations are real; each source has a different limitation.Source: §6.3 · p. 10
  • Retrieving tutorials as prompt context only marginally improves WebArena performance with Llama3.1-instruct-8B, while finetuning on Synatra data shows a substantial advantage.The authors conclude that trajectories are preferable for agent learning because models have difficulty using indirect knowledge alone.Source: §6.3 · p. 10

7 Related Work

Prior work explores indirect supervision, environment feedback, prompting, and synthetic instruction data, but complex long-horizon web navigation has not been systematically studied with fully synthetic data. Synatra focuses on generating training data for realistic web-agent tasks from existing indirect resources.

Learning from Indirect Supervision

  • Earlier indirect-supervision and reinforcement-learning work mostly targets simplistic agents in synthetic environments, while later reward-based self-improvement has been applied to more complex agents in the wild [8] [35] [55] [21] [32] [39] [34] [49] [28] [44] [14].Synatra addresses training on synthetic data from existing resources for complex web navigation.Source: §7 · p. 10

Prompting Approaches for AI Agents

  • Prior agent-prompting methods use reasoning, search and planning, or self-verification [43] [48] [13] [18] [34] [24].This work instead focuses on generating instruction-tuning data without human supervision.Source: §7 · p. 10

Data Generation for Interactive Agents

  • Fully synthetic generation for complex, long-horizon web navigation had not been systematically studied; prior approaches used environment execution or human assistance [11] [19].Synatra aims to generate realistic data without human intervention while preserving real-world navigation complexity and trajectory lengths.Source: §7 · p. 11

Data Generation for Instruction Finetuning

  • Instruction-tuning work generates task-specific instructions from examples [59] [50] [16], task-agnostic data [42] [15], or iterative data and finetuning [20].Synatra applies instruction-data generation to web-agent tasks using existing indirect resources such as web pages and tutorials.Source: §7 · p. 11

8 Conclusion

Synatra synthesizes trajectories from indirect knowledge, including a single static source, and the authors report that synthesis is more cost-effective than collecting a similar quantity of human demonstrations. They also note limited testing and dependencies on generation-model resources and source-page quality.

8 Conclusion

  • Synatra synthesizes trajectories from indirect knowledge, including a single static source, and is more cost-effective than collecting a similar quantity of human demonstrations.The authors argue that a capable LLM can turn a human-oriented tutorial and a random observation into a trajectory.Source: §8 · p. 11
  • Testing covered limited settings; the authors see potential to extend Synatra to other prompting techniques, more indirect knowledge, and base models of different sizes and configurations.These are proposed extensions, not settings established by the reported tests.Source: §8 · p. 11
  • Synatra requires a strong data-generation LLM, which may cost significant money and processing time, and depends on source web-page quality.Source: §8 · p. 11

A Action Space

This appendix illustrates Synatra’s browser action space and trajectory representations, including retail-task categories, generated demonstrations from tutorials or page observations, and baseline prompt instructions for observations and actions.

A Action Space

  • The excerpt identifies a WebArena action-space table but does not include its action descriptions.Source: Table 3 · p. 17

B Trajectory Representation

  • A sample trajectory pairs a browser task and page observation with sub-tasks and concrete actions such as typing, scrolling, and clicking.Source: B Trajectory Representation · p. 17

C Prompt to Filter wikiHow Articles

  • The wikiHow filter retains tasks performed entirely through a computer or phone GUI, excluding tasks requiring physical interaction, a command line, or unsupported devices.Examples classify changing a Linux Mint desktop wallpaper as eligible, but reject setting up Chromecast, deleting a file through Linux CLI, and rebooting an iPad.Source: C Prompt to Filter wikiHow Articles · pp. 17–18

D Prompt to Generate Demonstrations from Tutorials

  • The tutorial-rewriting prompt turns browser-task articles into accessible, concrete step sequences, choosing a common scenario, unrolling repeated steps, and assuming equivalent web apps when tutorials use desktop or mobile apps.It specifies browser APIs including click, hover, click-and-type, key presses, navigation, and tab management.Source: D Prompt to Generate Demonstrations… · pp. 18–19
  • The task supplies a browser-based task, past-action sequence, and next action; create static, visually rich HTML reflecting past actions and design the interactable next-action element with id="next-action-target-element".The page should have around 80 elements, at least 20 interactable elements, DOM depth around 7, and length of at least 3000 tokens.Source: D Prompt to Generate Demonstrations… · p. 19

E Example Synthetic Demonstration from Tutorials from a wikiHow Article

F Prompt to Generate Direct Demonstrations from Random Observations

  • The random-observation prompt asks for diverse, unambiguous browser tasks with prior-action histories and a justified next action grounded in the page’s accessibility tree.It requires one action per step, forbids goto() or new_tab() as the first step, and requires the next action’s element_id to come from the tree when available.Source: F Prompt to Generate Direct… · pp. 21–22

G Example Generated Trajectories from Random Observation

H Prompt Example

  • The baseline prompt supplies the objective, accessibility tree, website, open tabs, and previous action, and instructs the agent to issue one valid action at a time.It permits click, type, and hover actions and specifies a next-action response format; the example shows an Under Armour page, shopping objective, and prior navigation steps.Source: H Prompt Example · p. 26, Table 1 · p. 27

I Case Study

In hard, multi-step tasks, Synatra-CodeLlama succeeds where GPT-4-turbo misses visible page details or misreads task progress.

I Case Study

  • Across the examples, Synatra-CodeLlama uses visible page details and task state that GPT-4-turbo overlooks.It finds the link listing GitLab issues labeled “bug,” notices the commits information already on the page, and proceeds from a correctly set date range.Source: I Case Study · pp. 27–28, Figure 6 · p. 28, Figure 7 · p. 28
  • For the GitLab issue task, Synatra-CodeLlama finds the link listing “bug” issues, while GPT-4-turbo guesses a search term that returns no results.Both agents were asked to display issues labeled “bug.”Source: I Case Study · p. 27, Figure 6 · p. 28
  • For the commit-count task, GPT-4-turbo tries to view commits already displayed, while Synatra-CodeLlama identifies the relevant link.Both agents had navigated to Killian’s commits and could answer by counting.Source: I Case Study · p. 28, Figure 7 · p. 28
  • For the refund report, Synatra-CodeLlama proceeds from the correctly set Q1 date range, while GPT-4-turbo predicts re-entering the start date.Both agents had entered the correct start and end dates.Source: I Case Study · p. 28, Figure 8 · p. 29

J Data Selection from Random Observations in ClueWeb

The ClueWeb domain sample is highly dispersed: most domains appear only once.

J Data Selection from Random Observations in ClueWeb

K Training Settings

The paper reports the resources used to fine-tune CodeLlama and discusses potential benefits and societal concerns of the approach.

K Training Settings

  • Training on the 100k dataset used six A100 80G GPUs for about 40 hours.The setup used a batch size of 48, learning rate 4e-5, cosine annealing, and a 0.03 warm-up ratio.Source: K Training Settings · pp. 29–30

L Broader Impacts

  • The approach could make digital-agent training more affordable by using public data instead of expensive data collection.The paper suggests this could broaden adoption and innovation, particularly for resource-limited organizations and sectors.Source: L Broader Impacts · p. 30
  • The authors flag possible job displacement and the need for robust guidelines to keep deployment aligned with ethical standards.Source: L Broader Impacts · p. 30

M Limitations

The authors identify limitations involving indirect-source quality and representativeness, synthetic-data generalization, and the costs of LLM-based synthesis.

M Limitations

  • Indirect knowledge sources vary in quality and relevance, leaving a risk that the data will be unrepresentative despite filtering and sampling efforts.The authors used high-quality sources such as WikiHow and broad sources such as ClueWeb to mitigate this risk.Source: M Limitations · p. 30
  • Synthetic demonstrations may miss the complexity of human interactions, so agents may struggle with unexpected or atypical scenarios absent from training.Source: M Limitations · p. 30
  • If indirect-knowledge sources lack diversity, agents risk overfitting to their formats and tasks and may lack flexibility across platforms.Source: M Limitations · p. 30
  • Reliance on LLMs and complex synthesis may carry substantial computational costs and environmental impacts.The authors specifically raise energy consumption and carbon footprint concerns.Source: M Limitations · p. 30
Loading 2409.15637v2…