PaperlayerView exact revision on arXiv

Source-linked AI summary

Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments

Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, Sercan Ö. Arık

arXiv:2501.10893v1cs.LGcs.AI

TL;DR

LLM agents need high-quality, environment-specific data, but collecting human-labeled interaction trajectories is costly and autonomous construction remains limited. Learn-by-interact synthesizes and filters trajectories from documentation, reconstructs instructions through backward construction, and retrieves demonstrations for ICL or training. Across four realistic benchmarks, it improves agent performance in both settings, while backward construction and agentic retrieval contribute to data quality and effectiveness.

  • Problem

    High-quality agent data for new environments are difficult and expensive to obtain because annotation requires long interactions and domain expertise.

  • Method

    Learn-by-interact generates environment-specific instructions from standard resources, collects LLM-environment trajectories, applies backward construction, and uses the data for ICL and training.

  • Results

    Across SWE-bench, WebArena, OSWorld, and Spider2-V, synthesized data improve performance with both ICL and training; Claude-3.5-sonnet rises from 12.4% to 22.5% on OSWorld.

  • Takeaways & Limitations

    Backward construction improves synthesized-data quantity and quality, while agentic retrieval is superior to conventional RAG for ICL.

  • Takeaways & Limitations

    The framework requires many LLM calls, and its instruction generation depends on environment resources that may be incomplete or unavailable.

Abstract

from arXiv · show

Autonomous agents powered by large language models (LLMs) have the potential to enhance human capabilities, assisting with digital tasks from sending emails to performing data analysis. The abilities of existing LLMs at such tasks are often hindered by the lack of high-quality agent data from the corresponding environments they interact with. We propose Learn-by-interact, a data-centric framework to adapt LLM agents to any given environments without human annotations. Learn-by-interact synthesizes trajectories of agent-environment interactions based on documentations, and constructs instructions by summarizing or abstracting the interaction histories, a process called backward construction. We assess the quality of our synthetic data by using them in both training-based scenarios and training-free in-context learning (ICL), where we craft innovative retrieval approaches optimized for agents. Extensive experiments on SWE-bench, WebArena, OSWorld and Spider2-V spanning across realistic coding, web, and desktop environments show the effectiveness of Learn-by-interact in various downstream agentic tasks -- baseline results are improved by up to 12.2\% for ICL with Claude-3.5 and 19.5\% for training with Codestral-22B. We further demonstrate the critical role of backward construction, which provides up to 14.0\% improvement for training. Our ablation studies demonstrate the efficiency provided by our synthesized data in ICL and the superiority of our retrieval pipeline over alternative approaches like conventional retrieval-augmented generation (RAG). We expect that Learn-by-interact will serve as a foundation for agent data synthesis as LLMs are increasingly deployed at real-world environments.

1 Introduction

Learn-by-interact addresses the shortage and cost of high-quality agent data by synthesizing environment-specific interactions without human annotations. Across coding, web, and desktop benchmarks, the synthesized data improve both in-context learning and training, while backward construction and agentic retrieval strengthen data quality and use.

  • High-quality agent data remain scarce because annotating long environment interactions requires substantial cost and domain expertise.
  • Learn-by-interact uses self-instruct and environment interactions to synthesize diverse trajectories, then applies backward construction to create instructions for sub-trajectories.
  • Across SWE-bench, WebArena, OSWorld, and Spider2-V, the generated data improve agent performance in both training-free ICL and model training.
  • 12.4% to 22.5%: generated data nearly double Claude-3.5-sonnet’s OSWorld baseline performance with ICL.
  • Backward construction increases synthesized-data quantity and quality, while agentic retrieval improves ICL efficiency and performance.

2 Learn-by-interact

Learn-by-interact autonomously synthesizes agent data by generating tasks from standard resources, executing them in environments, and reconstructing instructions from resulting trajectories. It filters the data and adapts agents through agentic retrieval for ICL or action-prediction pairs for fine-tuning.

  • Canonical agentic tasks: The agent objective is to achieve a target through multi-step environment interactions, predicting each action from the instruction, history, and current observation.
  • Agentic data synthesis: Self-instruct uses documentation and other standard resources to generate diverse environment-specific task instructions without human annotation.
  • Agentic data synthesis: LLMs execute generated tasks to produce long trajectories, after which backward construction summarizes sub-trajectories or abstracts their purposes into new instructions.
  • Data filtering: Filtering removes duplicate states and retains instruction-trajectory pairs judged coherent, natural, reasonable, and aligned by an LLM committee.
  • Adaptation: Agentic retrieval combines observation-based matching with model-based queries from the instruction, interaction history, and current observation to retrieve demonstrations for next-action prediction.
  • Adaptation: Synthesized examples support fine-tuning by converting generated trajectories into supervised action-prediction input-output pairs.

3 Experiments

The experiments evaluate Learn-by-interact across four realistic agentic benchmarks using training-free retrieval and training-based adaptation, against established baselines and data distillation. Results compare standard resources, distilled trajectories, and synthesized data under benchmark-specific metrics.

  • Training-free evaluation: Training-free experiments compare the benchmark baselines, conventional RAG, data distillation, Reflexion, LATS, and Learn-by-interact with agentic retrieval.The training-free setting supports commercial LLMs with prediction-only API access and uses retrieved examples during inference.
  • Training-based evaluation: Training-based evaluation compares models trained on Learn-by-interact data with data distillation and pre-training baselines, using models with access to weight updates.The experiments also report performance before training, where synthesized data serves as demonstration examples.
  • Datasets: The evaluation covers SWE-bench, WebArena, OSWorld, and Spider2-V across realistic coding, web, desktop, and professional software environments.SWE-bench measures software-engineering problems, WebArena web tasks, OSWorld open-ended computer tasks, and Spider2-V multimodal data-science and engineering workflows.
  • Training-free evaluation: Marginal RAG gains and at most 2% improvement from distilled data contrast with stronger results from backward construction, which realigns instructions and trajectories.The authors attribute the distilled-data limitation to noise in multi-round agent-environment interactions and describe backward construction as curating new task objectives.
  • Training-based evaluation: 4.7% to 24.2%: training Codestral-22B on synthesized data raises WebArena performance, while ICL produces only a 5.5% improvement.The trained Codestral-22B model also outperforms Gemini when Gemini uses the synthesized data as demonstrations.

4 Analysis

The analysis examines inference efficiency, retrieval strategies, synthetic-data granularity, and scaling behavior across agent benchmarks. Learn-by-interact combines strong performance with lower inference effort, while agentic retrieval and larger datasets improve results.

  • Inference Efficiency: Learn-by-interact exhibits superior performance while using fewer LLM calls and slightly more tokens than the baseline.This contrasts with pipelines whose higher performance requires substantially more inference computation.
  • Inference Efficiency: LATS improves performance by 2.5% on average but requires nearly four times more tokens per instance than the baseline.
  • Synthetic-Data Granularity: Short-trajectory data is generally more advantageous for training and ICL, while mixing short, medium, and long trajectories provides the best performance.Trajectory groups are constrained to 200M tokens for fair comparison, with token count used to measure data size.
  • The Impact of Retrieval: Combining model-based and observation-based retrieval achieves the strongest results across the evaluated retrieval strategies.Instruction-based retrieval shows the least improvement, while observation-based retrieval is particularly effective for agent tasks.
  • The Impact of Retrieval: Observation-based retrieval improves Spider-2V performance by 4.4% absolute over instruction-based retrieval when using Gemini.Model-based retrieval uses task instructions, interaction history, and the current observation.
  • Scaling Laws: Both training-free and training-based approaches benefit from larger synthetic datasets, with tuning producing more substantial gains as data scales.The scaling results are averaged across WebArena and OSWorld; larger models show stronger training-free improvements.

5 Related work

Related work spans LLM agents, human- or model-labeled training data, and autonomous trajectory construction. These approaches include reasoning-and-acting systems, search and reflection, synthetic environments, and iterative learning from experience.

  • LLM Agents: LLM-agent research combines reasoning and acting, with some methods adding Monte Carlo Tree Search, value functions, and self-reflection.
  • Agent Data: Training-based approaches use human-labeled examples or data distilled from LLMs such as GPT-4.
  • Autonomous Data Construction: AgentGen automatically synthesizes environments and tasks before using FastDownward to generate trajectory data.
  • Autonomous Data Construction: Iterative experience-based approaches typically require reward models or LLM- and execution-generated feedback to improve data quality.

6 Conclusion

The conclusion presents Learn-by-interact as an annotation-free framework for adapting LLM agents through environment interactions and synthesized data. Its experiments report gains in both ICL and training, with lower latency and computational costs than leading agent approaches.

  • Conclusion: Learn-by-interact adapts LLM agents to given environments without human annotations by using documentation and LLM-environment interactions.
  • Conclusion: Backward construction addresses instruction-trajectory misalignment by deriving new instructions from trajectories.
  • Conclusion: Synthetic data from Learn-by-interact significantly enhances model performance in both ICL and training.
  • Conclusion: Learn-by-interact achieves better performance with lower latency and computational costs than other leading approaches in agent tasks.The analysis also reports superiority over classical RAG.

7 Limitations

Learn-by-interact has two stated limitations: substantial LLM-call requirements during generation and filtering, and dependence on environment-related resources that may be incomplete or unavailable.

  • Limitations: Learn-by-interact requires many LLM calls for generation and filtering, and its instruction synthesis depends on environment resources that may be incomplete or unavailable.

A Baseline implementations

The baselines represent each environment with task objectives, action-space descriptions, general instructions, and environment observations. Observation formats vary by benchmark, with accessibility trees used by default in WebArena and OSWorld.

  • SWE-bench baselines follow CodeAct, where LLMs interact with environments to solve problems.
  • WebArena prompts concatenate task objectives, action-space descriptions, general instructions, and webpage observations before predicting the next action.
  • WebArena uses the accessibility tree as its default observation space.
  • OSWorld and Spider2-V follow their original prompt styles, combining task objectives, action-space descriptions, general instructions, and computer observations.

B Dataset examples

The appendix provides dataset-level examples containing complete instructions and interaction histories with their environments.

  • Tables 8–17 provide one example for each dataset.
  • Each example includes full instructions.
  • Each example includes the interaction history with the environment.

C Experimental settings

Experimental settings specify retrieval limits, baseline trial and search parameters, document sources, and documentation substitutions when exact environment versions are unavailable.

  • RAG retrieval uses up to 50 documents and keeps the retrieved set unchanged throughout each agent trajectory.Documents are retrieved until the LLM maximum length, using only instructions as the retrieval query.
  • Reflexion uses a maximum of 3 trials, while LATS uses 5 generated actions, depth limit 15, and value-function weight 0.8.
  • SWE-bench uses all non-repeated Python files in SWE-bench-Verified as document sources.
  • When exact target-environment documentation is unavailable, the study uses documentation from the same domain to sample task instructions.The paper gives Amazon documentation as a replacement for a WebArena shopping website.
  • The experimental setup separately specifies documentation sources for OSWorld and Spider2-V.

E Synthesized data examples

Learn-by-interact generates instructions from documentation, produces interaction trajectories, and applies backward construction to realign instructions with observed behavior. It also filters low-quality examples before downstream use.

  • Data synthesis: An LLM first generates task instructions from tutorials, documentation, and FAQs, such as uploading CSV data from Google Drive to BigQuery.
  • Data synthesis: The instructed LLM attempts the task by predicting actions and collecting environmental feedback, producing a long interaction trajectory.
  • Data synthesis: A wrong action can misalign the trajectory with its initial instruction and produce noisy data.The example selects Google Cloud Storage instead of Drive, causing subsequent predictions to diverge from the requested task.
  • Backward construction: Backward construction creates new instructions by summarizing trajectory steps or abstracting the trajectory purpose.
  • Backward construction: The method can generate instructions for each sub-trajectory, increasing the utility of a generated full trajectory.
  • Quality filtering: Examples with unnecessary detours or repeated back-and-forth actions are filtered to avoid negative downstream influence.

G Synthesized data from environments

The section evaluates Learn-by-interact’s synthesized data through controlled comparisons, held-out WebArena testing, and examples of generated interaction data. Results indicate gains in both training and in-context learning, including cross-website generalization, while resource-free synthesis raises diversity and distribution concerns.

  • Controlled comparison: The resource-free comparison synthesizes 10k WebArena examples while following the same procedures as Learn-by-interact except for using existing resources.Evaluation uses Claude-3.5-sonnet for in-context learning and Codestral-22B for training, with equalized training-set sizes across WebArena domains.
  • Resource-free synthesis: Without prior documents, the distribution and diversity of generated data are hard to control, requiring intensive prompt engineering for diverse task instructions.The cited discussion contrasts this with expert- or user-created resources that cover important interaction scenarios.
  • Cross-website generalization: Learn-by-interact significantly improves both training and in-context learning on held-out CMS websites without using CMS-specific synthetic data.The authors interpret this result as evidence of potential cross-website generalization.
  • Synthesized examples: The examples include sub-trajectory instructions that replicate interaction trajectories in a BigQuery environment.A displayed example describes a browser focused on a BigQuery project and a dataset named “demo.”
Loading 2501.10893v1…