PaperlayerView exact revision on arXiv

Source-linked AI summary

Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments

Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, Sercan Ö. Arık

arXiv:2501.10893v1cs.LGcs.AI

TL;DR

High annotation costs and few fully autonomous data-construction pipelines constrain self-adaptation to new environments. Learn-by-interact synthesizes agent trajectories and derives instructions from them; across four benchmarks, its data improve performance in ICL and training, including on OSWorld for Claude-3.5-sonnet, from 12.4% to 22.5%.

  • Problem

    Agent-data annotation for new environments is costly, while fully autonomous data-construction pipelines for self-adaptive agents remain little explored.

    Source: §1 · p. 2

  • Method

    Learn-by-interact uses environment resources to generate tasks and interaction trajectories, then constructs instructions by summarizing trajectory steps or abstracting their purposes.

    Source: §2.2 · p. 4, §6 · p. 12

  • Results

    Synthetic data improves performance in ICL and training across four benchmarks; on OSWorld, Claude-3.5-sonnet rises from 12.4% to 22.5%.

    Source: §1 · p. 3, §6 · p. 12

  • Takeaways & Limitations

    The authors position synthetic agent data as a potential foundation for wider deployment of LLM agents in real-world environments.

    Source: §6 · p. 12

    The framework requires many LLM calls for generation and filtering, and may be constrained when environment-related resources are incomplete or unavailable.

    Source: §7 · p. 12

Abstract

from arXiv · show

Autonomous agents powered by large language models (LLMs) have the potential to enhance human capabilities, assisting with digital tasks from sending emails to performing data analysis. The abilities of existing LLMs at such tasks are often hindered by the lack of high-quality agent data from the corresponding environments they interact with. We propose Learn-by-interact, a data-centric framework to adapt LLM agents to any given environments without human annotations. Learn-by-interact synthesizes trajectories of agent-environment interactions based on documentations, and constructs instructions by summarizing or abstracting the interaction histories, a process called backward construction. We assess the quality of our synthetic data by using them in both training-based scenarios and training-free in-context learning (ICL), where we craft innovative retrieval approaches optimized for agents. Extensive experiments on SWE-bench, WebArena, OSWorld and Spider2-V spanning across realistic coding, web, and desktop environments show the effectiveness of Learn-by-interact in various downstream agentic tasks -- baseline results are improved by up to 12.2\% for ICL with Claude-3.5 and 19.5\% for training with Codestral-22B. We further demonstrate the critical role of backward construction, which provides up to 14.0\% improvement for training. Our ablation studies demonstrate the efficiency provided by our synthesized data in ICL and the superiority of our retrieval pipeline over alternative approaches like conventional retrieval-augmented generation (RAG). We expect that Learn-by-interact will serve as a foundation for agent data synthesis as LLMs are increasingly deployed at real-world environments.

1 Introduction

Learn-by-interact synthesizes agent data from environment interactions and backward-constructs task instructions, improving performance across coding, web, desktop, and database benchmarks.

1 Introduction

  • LLM agents remain below human capabilities in realistic digital tasks; prompt-based adaptation is limited by foundation models, while human-labeled agent data is difficult and costly to produce.Annotation requires long environment-interaction trajectories and domain expertise.Source: §1 · pp. 1–2
  • The framework uses documentation and tutorials to generate diverse tasks, gathers agent-environment trajectories, then backward-constructs instructions by summarizing or abstracting sub-trajectories.It filters the resulting instruction-trajectory pairs for training or in-context learning.Source: §1 · p. 2, Figure 1 · p. 2
  • For in-context learning, agentic retrieval uses model-generated queries or matches current observations to states encountered in synthesized trajectories.The model-based approach conditions queries on instructions, interaction histories, and current observations; observation-based retrieval finds examples containing the current observation.Source: §1 · p. 2, Figure 1 · p. 2
  • Across four benchmarks, Learn-by-interact achieves best-class leaderboard performance; Claude-3.5-sonnet on OSWorld rises from 12.4% to 22.5%, and Codestral-22B on WebArena increases from 4.7% to 24.2% after training.The OSWorld improvement is reported for in-context learning; the WebArena improvement follows training on synthesized data.Source: §1 · p. 3
  • Ablations find backward construction increases synthesized-data quantity and quality; the data improve inference performance and efficiency, and agentic retrieval outperforms alternatives in in-context learning.Source: §1 · p. 3

2 Learn-by-interact

Learn-by-interact autonomously synthesizes environment-specific agent data from documentation and applies it through in-context learning or model training.

2.1 Task formulation

  • Given an environment and instruction, an agent uses interaction history to choose actions until it stops or reaches the maximum step limit.Source: §2.1 · p. 3

2.2 Agentic data synthesis

  • Documentation seeds diverse instructions; LLMs execute them, then backward construction summarizes sub-trajectories or abstracts purposes, correcting misalignment and yielding quadratically many examples per trajectory.Reusing sub-trajectories can reduce the interactions needed for a target dataset size, especially when interactions are costly.Source: §2.2 · pp. 3–4

2.3 Filtering

  • Filtering removes consecutive duplicate action-observation states and keeps instruction-trajectory pairs only when an LLM committee judges them coherent, natural, reasonable, and aligned.Source: §2.3 · p. 4

2.4 Adaptation

  • For ICL, observation matches and model-generated queries retrieve examples; retrieved demonstrations condition action prediction, while synthesized pairs also support supervised fine-tuning.Source: §2.4 · p. 5

3 Experiments

Experiments evaluate Learn-by-interact across four realistic, multi-round agent benchmarks in training-free and training-based settings. The results report gains from synthesized data in both settings, with training outperforming the baseline and data distillation.

3.1 Baselines

  • The experiments compare agentic retrieval against vanilla prompting, conventional RAG, data distillation, Reflexion, and Language Agent Tree Search.For training-based evaluation, data distillation is the primary comparison, alongside model performance before training.Source: §3.1 · pp. 5–6

3.2 Datasets

  • The evaluation uses four multi-round datasets spanning code, web, desktop, and professional-software tasks.They are SWE-bench (verified by default), WebArena, OSWorld, and Spider2-V.Source: §3.2 · p. 6

3.3 Settings

  • The study synthesizes separate environment-specific data for each benchmark and uses examples for ICL or training.Claude-3.5-sonnet generates trajectories, Gemini-1.5-pro and Claude-3.5-sonnet filter data, and three task instructions are sampled per document; retrieval is capped at five examples for each retrieval method.Source: §3.3 · p. 6

3.5.1 Training-free Evaluation

  • Learn-by-interact data significantly improves results over other frameworks for both Gemini and Claude; on OSWorld, Claude’s result almost doubles the baseline.The comparison covers training-free approaches; the paper attributes the gains to high-quality generated data.Source: Table 2 · p. 7, §3.5.1 · p. 8

3.4 Evaluation

  • Evaluation follows benchmark-specific metrics, including SWE-bench pass@1, WebArena fuzzy and string match, OSWorld functional correctness, and Spider2-V validation checks.WebArena uses Claude-3.5-sonnet as the LLM judge replacement; results are reported as percentages of resolved instances.Source: §3.4 · p. 7

3.5.1 Training-free Evaluation

  • RAG only marginally improves over baseline, distilled examples yield at most 2% improvement, and Reflexion and LATS consistently improve over baseline across four benchmarks.The authors attribute distilled data’s limited gains to noise from misaligned instructions and trajectories, and identify backward construction as a way to correct this misalignment.Source: §3.5.1 · p. 7

3.5.2 Training-based Evaluation

  • Codestral-22B on WebArena rises from 4.7% to 24.2% when trained on Learn-by-interact data, versus a 5.5% ICL improvement, and surpasses Gemini using demonstrations.Training with Learn-by-interact data substantially surpasses both the baseline and data distillation.Source: Table 3 · p. 8, §3.5.2 · p. 8
  • Training uses weight updates, and the trained models achieve better results than using Learn-by-interact data only as ICL examples.The trained Codestral-22B model on WebArena even outperforms Gemini when Gemini uses the synthesized data as demonstrations.Source: §3.5.2 · p. 8

4 Analysis

The analysis examines inference efficiency, retrieval strategies, trajectory granularity, and data scaling. It finds that agentic retrieval and larger or more varied synthetic data improve agent performance, while Learn-by-interact can improve performance with modest inference costs.

4.1 Inference Efficiency

  • 2.5% average improvement from LATS costs nearly four times the baseline tokens, while Learn-by-interact improves performance with fewer calls and slightly more tokens.The authors attribute its efficiency to environment information in synthesized examples, which can help LLMs finish tasks in fewer steps; data synthesis occurs in advance.Source: §4.1 · pp. 8–9, Figure 2 · p. 9

4.2 The Impact of Retrieval

  • 4.4% absolute improvement on Spider-2V with Gemini over instruction-based retrieval; combining model-based and observation-based retrieval achieves the best results.Observation-based retrieval also outperforms instruction-based retrieval, while model-based retrieval performs even better than observation-based retrieval alone.Source: Table 4 · p. 9, §4.2 · pp. 9–10

4.3 Data granularity

  • Short trajectories consistently outperform medium and long ones in training-free and training-based evaluations, while combining data groups is more effective than using any single group.Groups were compared at 200M tokens each; short trajectories can serve as sub-steps or partial workflows, and short-plus-medium data improved WebArena results with Codestral-22B.Source: Table 5 · p. 10, §4.3 · p. 10

4.4 Scaling Laws

  • Both training-free and training-based evaluation benefit from more synthetic data; training-free gains are larger for Claude and Gemini than for Codegemma and Codestral.For a given data amount, fine-tuning smaller models is more effective than using the data as demonstrations; results are averaged across WebArena and OSWorld.Source: §4.4 · p. 11

5 Related work

Prior work improves LLM agents through reasoning and acting, training data generation, and reinforcement-learning-based data use. Learn-by-interact distinguishes itself by using backward construction to align instructions with trajectories in realistic environments.

5 Related work

  • Zhou et al.’s MCTS-integrated approach incurs increased computational costs and relies on environments allowing state reversals; Learn-by-interact removes such assumptions and improves agent efficiency and performance by synthesizing high-quality data in advance.Source: §5 · p. 11
  • Prior training uses human-labeled or GPT-4-distilled examples, while AgentGen (Hu et al., 2024) and AgentTuning (Zeng et al., 2023) automate different parts of data generation.Learn-by-interact instead targets realistic environments and generates tasks and trajectories through backward construction.Source: §5 · p. 11
  • Offline reinforcement-learning approaches use LLM-policy data or iteratively train on experience trajectories (Gulcehre et al., 2023; Aksitov et al., 2023), typically relying on reward models or LLM/execution feedback.Learn-by-interact instead uses backward construction to improve data quality by aligning instructions and trajectories.Source: §5 · p. 12

6 Conclusion

Learn-by-interact adapts LLM agents without human annotations by synthesizing environment interactions from documentation and deriving instructions from trajectories; its synthetic data improves performance in ICL and training.

6 Conclusion

  • Learn-by-interact uses documentation to propose tasks and generate multi-round interactions, then derives updated instructions from trajectories to address instruction–trajectory misalignment.The framework is intended to adapt LLM agents to given environments without human annotations.Source: §6 · p. 12
  • Its retrieval approaches use agent instructions, interaction histories, and current observations to retrieve synthesized examples.Source: §6 · p. 12
  • Experiments show Learn-by-interact’s synthetic data enhances performance in ICL and training, with better performance, lower latency and computational costs than leading agent-task approaches, and superiority over classical RAG.Source: §6 · p. 12
  • Future work will explore multimodal settings and train general agent models for realistic environments.Source: §6 · p. 12

7 Limitations

Learn-by-interact has two practical limitations: its generation and filtering require many LLM calls, and it depends on environment resources that may be incomplete or unavailable. The authors identify more efficient annotation without sacrificing quality as a direction for future work.

7 Limitations

  • Learn-by-interact requires many LLM calls for data generation and filtering.The authors call for more efficient annotation approaches that do not sacrifice quality.Source: §7 · p. 12
  • Learn-by-interact relies on environment-related resources to generate instructions, but those resources may be incomplete or unavailable.Source: §7 · p. 12

A Baseline implementations

The baselines follow each benchmark’s existing interaction and prompting frameworks, adapting the observations supplied to agents where needed.

A Baseline implementations

  • SWE-bench follows CodeAct (Wang et al., 2024b), where language models interact with environments to solve problems.Source: A Baseline implementations · p. 16
  • WebArena follows Drouin et al. (2024), prompting models with task objectives, action descriptions, general instructions, and webpage observations to predict the next action.The default observation space is the accessibility tree.Source: A Baseline implementations · p. 16
  • OSWorld and Spider2-V follow their original benchmark prompt styles, using accessibility-tree observations for OSWorld and set-of-mark for Spider2-V.Set-of-mark is used for Spider2-V because the accessibility tree loses significant information in the original benchmark.Source: A Baseline implementations · p. 16

B Dataset examples

The appendix presents one full example for each dataset, including its instructions and environment interaction history.

B Dataset examples

  • Tables 8–17 provide one example for each dataset, with full instructions and environment interaction history.Source: B Dataset examples · p. 16

C Experimental settings · D Document sources

The sections specify retrieval, agent, and training settings, and describe how documentation sources are selected when exact environment documentation is unavailable. For example, they use Amazon documentation for WebArena’s OneStopMarket shopping tasks.

C Experimental settings

  • RAG retrieval runs to the LLM’s maximum length, capped at 50 documents.The passages do not specify the actual retrieved document count below that upper bound.Source: C Experimental settings · p. 16
  • Retrieved documents remain unchanged throughout each interaction trajectory because retrieval uses only instructions as its query.Source: C Experimental settings · p. 16
  • Reflexion uses 3 trials; LATS uses 5 generated actions, a depth limit of 15, and a value-function weight of 0.8.Source: C Experimental settings · p. 16
  • The default dense retriever uses Vertex AI text embeddings, and experiments use temperature 0 for reproducibility.Source: C Experimental settings · p. 16
  • Training uses batch size 128, learning rate 0.00002, warmup ratio 0.03, maximum length 8192, and 3 epochs on 80GB H100 machines.Source: C Experimental settings · p. 16

D Document sources

  • For SWE-bench-Verified, the document sources are all non-repeated Python files; when exact documentation is unavailable, the authors use same-domain sources such as Amazon for OneStopMarket.The passages cite frequent shopping tasks such as order changes, product searches, and delivery checks as rationale for this substitution.Source: D Document sources · pp. 16–17

E Synthesized data examples · F Case study on filtered examples

The examples show how Learn-by-interact generates interaction trajectories from documentation-based instructions, then uses backward construction to align instructions with trajectories and sub-trajectories. The filtered examples illustrate removing detours and repeated actions to avoid negative downstream effects.

E Synthesized data examples

  • Instructions are generated from tutorials, documentation, or FAQs, such as “Upload CSV data in Google Drive to BigQuery.”Source: E Synthesized data examples · p. 17
  • The LLM attempts each task through predicted actions and environmental feedback, producing a long interaction trajectory.Source: E Synthesized data examples · p. 17
  • A wrong action can misalign a trajectory: choosing Google Cloud Storage instead of Drive leads to subsequent incorrect predictions and noisy data.Source: E Synthesized data examples · p. 17
  • Backward construction replaces misaligned instruction-trajectory pairs with instructions that summarize the trajectory or propose a task aligned with it.For example, the objective changes to “Link CSV file in Google Cloud Storage to BigQuery.”Source: E Synthesized data examples · p. 18
  • New instructions for sub-trajectories can increase the utility of a full trajectory, including questions grounded in a short observation-action sequence.One example asks when dataset “demo” was created.Source: E Synthesized data examples · p. 18

F Case study on filtered examples

  • Representative synthesized examples failing the designed criteria are filtered when Action 1–6 are unnecessary for the goal or clicks on “My Orders” and “View Order” repeat back and forth across states, to avoid negative downstream influences.Source: F Case study on filtered examples · p. 18

G Synthesized data from environments · H Cross-website generalization

This section presents filtered OSWorld and WebArena interaction examples and evaluates Learn-by-interact in resource-free and held-out WebArena CMS settings. It reports cross-website improvements over the baseline in training and in-context learning, alongside concerns about controlling data distribution and diversity in the resource-free variant.

G Synthesized data from environments

  • The resource-free WebArena comparison synthesizes 10k examples using the same procedures, except that task instructions are sampled from LLMs based on the environment.Its Claude-3.5-sonnet in-context learning results are reported in Table 6, but the supplied passages do not give the results.Source: G Synthesized data from environments · p. 18
  • Without prior documents, controlling generated-data distribution and diversity is difficult and requires intensive prompt engineering to guide instruction generation.The authors note that expert-crafted or real-user resources cover many important scenarios of interest to environment users.Source: G Synthesized data from environments · p. 18

H Cross-website generalization

  • The CMS generalization test uses synthetic data from other WebArena websites, with training-set sizes matched by downsampling before evaluating Claude-3.5-sonnet in-context learning and Codestral-22B training.The authors interpret improvements in both settings as indicating potential cross-website generalization and suggest that data from more websites may improve performance.Source: H Cross-website generalization · pp. 18–19
  • Learn-by-interact significantly improves over the baseline in both training and in-context learning on held-out WebArena CMS tasks.The study trains on synthetic data from the remaining websites, equalizes training-set sizes by downsampling, and evaluates Claude-3.5-sonnet and Codestral-22B; the result is reported in Table 7.Source: H Cross-website generalization · pp. 18–19, Table 7 · p. 19
  • An OSWorld example enters =SUM(A1:A3) into a spreadsheet formula box and presses Enter.The trace also shows menu and cell-selection actions.Source: H Cross-website generalization · pp. 48–49, Table 40 · p. 48
  • A WebArena example asks for items in the latest cancelled order and navigates through My Account, My Orders, and View Order.The supplied passages show these navigation steps but do not provide the requested items.Source: H Cross-website generalization · pp. 50–51, Table 42 · p. 50, Table 43 · p. 51

Related papers

Loading 2501.10893v1…