PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning

Minghao Chen, Yihang Li, Yanting Yang, Shiyu Yu, Binbin Lin, Xiaofei He

arXiv:2405.16247v4cs.AIcs.CL

TL;DR

Directly using saved skills as in-context examples can cause the Path Dependence problem, making agents blindly replicate prior successes and fail to adapt to new scenarios. AutoManual uses a structured rule system with a Planner for actionable code, a Builder for online rule updates, case-conditioned prompting, and a Formulator for manual compilation. Starting from one demonstration, AutoManual achieves 97.4% with GPT-4-turbo and 86.2% with GPT-3.5-turbo on ALFWorld, and 98.3% and 92.7% on MiniWoB++.

  • Problem

    Directly using saved skills as in-context examples can cause the Path Dependence problem, making agents blindly replicate prior successes and fail to adapt to new scenarios.

    Source: §1 · p. 1

  • Method

    AutoManual uses a structured rule system with a Planner for actionable code, a Builder for online rule updates, case-conditioned prompting, and a Formulator for manual compilation.

    Source: §1 · p. 2

  • Results

    Starting from one demonstration, AutoManual achieves 97.4% with GPT-4-turbo and 86.2% with GPT-3.5-turbo on ALFWorld, and 98.3% and 92.7% on MiniWoB++.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The generated manuals improve agent adaptability and provide human-readable guidance for planning in complex environments.

    Source: §1 · p. 2, §5 · p. 10

    AutoManual heavily relies on GPT-4-turbo to generate reliable rules, which may limit applicability with less advanced language models.

    Source: A Limitations · p. 13

Abstract

from arXiv · show

Large Language Models (LLM) based agents have shown promise in autonomously completing tasks across various domains, e.g., robotics, games, and web navigation. However, these agents typically require elaborate design and expert prompts to solve tasks in specific domains, which limits their adaptability. We introduce AutoManual, a framework enabling LLM agents to autonomously build their understanding through interaction and adapt to new environments. AutoManual categorizes environmental knowledge into diverse rules and optimizes them in an online fashion by two agents: 1) The Planner codes actionable plans based on current rules for interacting with the environment. 2) The Builder updates the rules through a well-structured rule system that facilitates online rule management and essential detail retention. To mitigate hallucinations in managing rules, we introduce a *case-conditioned prompting* strategy for the Builder. Finally, the Formulator agent compiles these rules into a comprehensive manual. The self-generated manual can not only improve the adaptability but also guide the planning of smaller LLMs while being human-readable. Given only one simple demonstration, AutoManual significantly improves task success rates, achieving 97.4\% with GPT-4-turbo and 86.2\% with GPT-3.5-turbo on ALFWorld benchmark tasks. The code is available at https://github.com/minghchen/automanual.

1 Introduction

AutoManual addresses the difficulty of adapting LLM agents to unfamiliar, variable environments by building and updating structured environmental rules through interaction. Its generated manuals improve benchmark task success from a single demonstration while organizing planning knowledge for reuse.

Motivation

  • AutoManual builds a well-organized understanding of unfamiliar environments to address path dependence and distributional shift in prior experience-based methods.Its rules retain skills, reflections, and error-prone details while remaining continuously updateable in response to new situations.Source: §1 · pp. 1–2

Framework

  • The Planner writes actionable code from current rules, the Builder updates relevant rules from trajectories, and the Formulator compiles them into a Markdown manual.Planner–environment interaction continues until an episode ends, after which the Builder revises rules and the Formulator organizes them by application scenario.Source: §1 · p. 2

Rule Management

  • Case-conditioned prompting directs the Builder to manage rule types according to trajectory cases and distinguish new environmental situations from failures to follow existing rules.This strategy is intended to reduce hallucinations when extracting applicable rules from long trajectories.Source: §1 · p. 2

Contributions

  • The framework uses actionable code and a structured rule system to manage multiple kinds of knowledge from code-based interactions.The rule system supports the management of knowledge gathered through the Planner’s environment interactions.Source: §1 · p. 2
  • AutoManual alternates Planner practice with Builder rule optimization online, then uses the Formulator to reorganize and formalize rules for readability.The alternating process is presented as a way to resolve the Path Dependency problem while producing a human-readable manual.Source: §1 · p. 2

Results

  • 97.4% with GPT-4-turbo and 86.2% with GPT-3.5-turbo on ALFWorld, plus 98.3% and 92.7% on MiniWoB++, are achieved from a single demonstration.The reported values correspond respectively to GPT-4-turbo and GPT-3.5-turbo on ALFWorld and MiniWoB++.Source: §1 · p. 2

2 Related Works

Related work develops LLM agents that plan with code, improve through reflection and stored skills, manage episodic memory, and discover rules. AutoManual differs by updating environmental rules online and verifying their applicability through interaction.

2.1 LLM for Agents Planning

  • LLM agents use reasoning, planning, environmental feedback, and executable Python code to support tasks in robotics, games, software development, and other domains.Code-based plans can respond automatically to in-plan feedback and outperform action- and JSON-format outputs in the cited prior work.Source: §2.1 · p. 2

2.2 Self-improvement of LLM Agents

  • Reflection and stored successful skills improve future planning, but stopping skill updates can produce Path Dependency when agents repeat prior strategies in new scenarios.Reflexion reflects on failures, while Voyager and AdaPlanner archive successful programs for later use.Source: §2.2 · p. 3
  • Prompt-optimization methods improve LLM reasoning, whereas AutoManual targets unfamiliar environments and maintains richer environmental knowledge than typically brief optimized prompts.The distinction is between general reasoning optimization and environment-specific adaptation.Source: §2.2 · p. 3

2.3 Memory Management of LLM Agents

  • Memory-management approaches update, retrieve, or select episodic memories, but Generative Agents’ reflections target continuous scenarios rather than task-oriented rules.Examples include causal abstractions, situation-matched retrieval, and controllable working or long-term memory.Source: §2.3 · p. 3

2.4 LLM for Rule Discovery

  • Rule-discovery methods induce, extract, or retrieve rules, while AutoManual updates rules online to verify their reliability and applicability.ExpeL and AutoGuide extract rules from offline experiences, unlike AutoManual’s online rule updates.Source: §2.4 · p. 3

3 Methods

AutoManual builds and tests environment knowledge through alternating Planner–Builder interaction, then organizes validated rules into a Markdown manual. Its Planner uses free-form code and trajectory conclusions, while the Builder manages structured rules with case-conditioned prompts and consolidation.

3.1 AutoManual Overview

  • AutoManual alternates interactive planning with online rule updates, then formulates the resulting rules into a Markdown manual for test-time planning.The framework comprises building, formulating, and testing stages, with a Consolidator merging or deleting redundant rules when needed.Source: §3.1 · pp. 3–4, Figure 1 · p. 4

3.2 Planner Agent for Interactive Planning

  • The Planner receives rules, task context, observations, and retrieved experiences, then iteratively generates executable Python code and uses environmental feedback to continue planning.Each response contains analysis, related rules, an overall plan, and code; the interaction continues until the episode ends or a response limit is reached.Source: §3.2 · p. 4, Figure 2 · p. 5
  • Planner conclusions classify episodes as Direct Success, Indirect Success, or Failure and preserve code, mistakes, or failure reflections for skill and reflection libraries.These conclusions provide detailed success processes or error reflections that assist later rule management, while retrieved skills remain more task-specific than rules.Source: §3.2 · p. 5

3.3 Builder and Consolidator Agents for Rule Management

  • The Builder manages six rule types with scope-specific content, trajectory examples, and validation logs, while case-conditioned prompts distinguish imperfect rules from imperfect agents.The rule system supports writing, updating, stopping, and consolidating rules; consolidation investigates dependent trajectories and retains rule details and examples.Source: §3.3 · p. 6

3.4 Manual Formulation

  • The Formulator groups rules by target scenario, adds introductions and key principles, and compiles them into a structured Markdown manual.This organization is intended to improve readability, logical flow, and accessibility of the validated rules.Source: §3.4 · p. 7

4 Experiments

AutoManual is evaluated across ALFWorld, MiniWoB++, and WebArena using limited human demonstrations, with comparisons against prior agents and ablations of its components. Results show strong gains from online rule management, structured knowledge, case-conditioned prompts, cross-task learning, and manual formulation.

4.1 Main Results

  • On ALFWorld, AutoManual significantly outperforms existing methods, achieving overall success rates of 86.2% with GPT-3.5-turbo and 97.4% with GPT-4-turbo using one human example.All experiments were run three times, with averages reported.Source: §4.1 · p. 8, Table 1 · p. 8
  • AutoManual exceeds prior methods and Planner+Lib. by a large margin on MiniWoB++ tasks with feedback, where diverse interruptions demand adaptation beyond successful-path imitation.The evaluation covers 9 feedback-enabled task types and compares success rates across methods and human-example budgets.Source: Table 2 · p. 8, §4.1 · p. 9
  • Cross-task rule sharing improves ALFWorld learning curves because rules developed for one task type deepen understanding that benefits other task types.Figure 4 compares cross-task and single-task rule building while success rates are measured with GPT-4-turbo or GPT-3.5-turbo.Source: §4.1 · p. 9, Figure 4 · p. 9
  • Planner+Lib. tends to plateau at lower MiniWoB++ success rates because skill libraries cannot express rules behind highly diverse scenarios, producing Path Dependency.AutoManual is compared with Planner+Lib. on 9 feedback-enabled task types in Figure 4(b).Source: §4.1 · p. 9, Figure 4 · p. 9

4.2 Ablation Study

  • 90.7% versus 88.1% shows that offline rule management only slightly improves planning, because rules cannot be verified through timely environmental feedback.The offline variant collects trajectories before managing rules, exposing it to distributional shift and mundane trajectory mistakes.Source: §4.2 · p. 9, Table 4 · p. 9
  • Removing skill and reflection libraries lowers success from 97.4% to 89.5%, indicating that historical interaction details complement induced rules during planning.Using plain experiences without rule induction still risks Path Dependency.Source: §4.2 · pp. 9–10
  • Case-conditional prompts improved rule management by reducing hallucination, increasing success rate from 93.8% to 97.4%, and ensuring the Builder updates rules reasonably and grounded.Source: §4.2 · p. 10
  • Manual formulation contributes to 97.4% success and fewer average error steps by presenting rule-based knowledge in an organized, accessible format.The formulated manual supports both multi-rule planning and human readability.Source: Table 4 · p. 9, §4.2 · p. 10

5 Conclusion

AutoManual enables LLM agents to improve adaptability and continual learning through online rule optimization, generating comprehensive manuals for diverse environments.

5 Conclusion

  • AutoManual enables adaptability and continual learning through online rule optimization, generating comprehensive manuals for LLM agents.The framework is designed to improve agent generalization while reducing reliance on human-provided examples and expert intervention.Source: §5 · p. 10

A Limitations

The paper identifies limitations in model dependence, rule-context scalability, exploration, and rule adherence, while discussing potential safety and human-use benefits.

A Limitations

  • AutoManual relies heavily on GPT-4-turbo for reliable rule generation, which may limit applicability with less advanced language models.Source: A Limitations · p. 13
  • Placing all rules directly in the LLM context may not scale well to larger, more dynamic environments.The paper suggests retrieval-based selection of relevant rules as a possible remedy.Source: A Limitations · p. 13
  • AutoManual agents may explore complex unfamiliar tasks insufficiently because they attempt solutions based only on current knowledge.The paper suggests curiosity or tree-search methods to improve exploration.Source: A Limitations · p. 13
  • Planner agents may ignore manual rules or hallucinate observations, preventing ideal code from transferring across situations.The paper identifies rule-compliance mechanisms and stronger validation as needed improvements.Source: A Limitations · p. 13

B Broader Impacts

  • AutoManual may improve predictability and operational safety, but unanticipated scenarios and incomplete rule adherence can produce unpredictable behavior.The generated manuals may also support worker training, decision support, and task efficiency.Source: B Broader Impacts · p. 14

C Difference with Prior Methods

Compared with ExpeL and AutoGuide, AutoManual uses code-based interaction, online rule updates, agent collaboration, richer rule attributes, and consolidation for rule management.

Interactive Form

Online v.s. Offline

  • Online rule updates verify rule reliability and applicability during interaction, unlike the offline extraction used by ExpeL and AutoGuide.The authors also state that online updates help planners improve trajectories over time.Source: C Difference with Prior Methods · p. 14

Collaboration Between Agents

  • Planner–Builder collaboration lets the Planner describe considered rules and special phenomena, providing the Builder richer input than action sequences alone.Source: C Difference with Prior Methods · p. 14

Rule System

  • Each AutoManual rule stores Type, Content, Example, and Logs, and rules can build on one another.These attributes provide a more comprehensive representation than the content-and-score format used by ExpeL and AutoGuide.Source: C Difference with Prior Methods · p. 14

Handling excessive rules

D Implementation Details

The implementation fixes model assignments, context and retry limits, task-building procedures, environment formatting, feedback handling, and baseline reproduction details.

D Implementation Details

  • Experiments cap rules at 12, use zero temperature and 16,000-token contexts, and limit Planner replanning to 3 times on ALFWorld and 6 times on an unspecified task.Source: D Implementation Details · p. 14
  • Building uses GPT-4-turbo for all agents, while testing compares GPT-4-turbo and GPT-3.5-turbo as Planner models.The implementation identifies these models as gpt-4-1106-preview and gpt-3.5-turbo-1106.Source: Table 5 · p. 15
  • ALFWorld building uses 36 tasks, stopping a task type after three consecutive successes, while the API cost for building and formulation is about $14.Source: D Implementation Details · p. 15
  • ALFWorld text is reformatted for code by changing object identifiers and removing preceding articles, with a 50-action task limit.Source: D Implementation Details · p. 15
  • MiniWoB++ building uses 7 feedback task types and 42 tasks, while all 9 types are evaluated during testing.A two-step strategy additionally updates skill and reflection libraries for 44 no-feedback task types among all 53 tasks.Source: D Implementation Details · p. 15
  • MiniWoB++ feedback concatenates action success or failure logs with HTML text for the Planner.Source: D Implementation Details · p. 15
  • AdaPlanner and RCI use official GitHub implementations with bug fixes, but reproduce lower performance than reported, possibly because of code-release or GPT-version differences.Source: D Implementation Details · p. 15

E Presentation of Generated Manuals

AutoManual’s generated ALFWorld manual organizes discovered rules into categories and success processes, while revealing remaining redundancy in the formulation.

E Presentation of Generated Manuals

  • AutoManual organizes ALFWorld knowledge into four rule categories and records navigation, interaction, task-specific, and validation guidance.Examples include searching all possible locations, holding one object at a time, and selecting among same-type objects.Source: E Presentation of Generated Manuals · p. 16
  • The manual captures fixed success processes for “Cool,” “Heat,” “Clean,” and “Examine,” while classifying the more complex “Put Two” process.Source: E Presentation of Generated Manuals · p. 16
  • Rule composition lets later rules derive solutions from basic mechanisms, such as handling co-located objects sequentially according to an earlier rule.Source: E Presentation of Generated Manuals · p. 16
  • The generated manual still contains unnecessary duplicate rules, including separate Success Process rules for “Examine” tasks.Source: E Presentation of Generated Manuals · p. 16

F.1 Ablation Study of Rule System

Ablations show that structured rule attributes, reusable helper methods, agent cooperation, and case classification materially support AutoManual’s performance, while validation logs mainly aid debugging.

F.1 Ablation Study of Rule System

  • Removing the rule “Type” attribute reduces success rates from 86.2% to 74.6% and from 97.4% to 91.5%.The ablation removes both the attribute and the Builder instructions for managing rule types.Source: §F.1 · p. 19
  • Removing the “Example” attribute causes a large performance drop because rules lose necessary details and may mislead the Planner.Source: §F.1 · p. 19
  • Removing useful helper methods lowers success rates from 86.2% to 79.6% and from 97.4% to 94.3%, indicating benefits from reusable programming abstractions.Source: §F.1 · p. 19
  • Removing Planner thoughts and relevant-rule outputs decreases performance, showing that cooperation between Planner and Builder supports rule management.Source: §F.1 · p. 19
  • Using only Imperfect Rules prompts instead of classifying all five cases performs worse, while Validation Logs have little performance impact but aid debugging.Source: §F.1 · p. 19

F.2 Sensitivity Analysis of Examples and Initial Rules

AutoManual remains robust when its initial example and rules provide less relevant knowledge, learning task knowledge online that resembles human-provided guidance.

F.2 Sensitivity Analysis of Examples and Initial Rules

  • AutoManual can learn required knowledge through online optimization even when initialized with an enter-text example instead of a search-engine example.The sensitivity study evaluates nine MiniWob++ task types with feedback.Source: §F.2 · p. 20
  • The learned search-engine knowledge closely matches standard human knowledge, including turning pages when the search index exceeds 3.Source: §F.2 · p. 20

G.1 An Example of Path Dependency Problem

The path-dependency example shows a Planner reusing a strategy that succeeds when target objects share a receptacle but fails when they are distributed across receptacles.

G.1 An Example of Path Dependency Problem

  • A previously successful “Put Two” strategy fails in a new scenario because the target objects are located in different receptacles.Source: §G.1 · p. 20
  • The reused plan searches a fixed list of receptacles, parses observations for target objects, stops after finding two, and then transfers them to the destination.Source: §G.1 · p. 21
  • In the illustrated execution, the Planner opens cabinet_1 and cabinet_3, observes their contents, then opens drawers 1–3, finding nothing in each drawer.Source: §G.1 · p. 21
  • The execution terminates with an error because fewer than two pillows were found, illustrating that the copied strategy does not adapt to the new configuration.Source: §G.1 · p. 22

G.2 A Trajectory of The Building Stage

This section presents the Building Stage as a process for learning, diagnosing, validating, and consolidating executable environmental rules from trajectories and execution errors. It covers the prompts and agents that manage these updates and formulate the resulting rules into an accessible, structured manual.

G.2 A Trajectory of The Building Stage

  • The initial heating attempt failed because the agent was at shelf_1 rather than at the microwave’s location.The error output states “Nothing happens,” and the Planner identifies the missing location precondition.Source: §G.2 · p. 23
  • The corrected trajectory succeeded after the agent went to the microwave before heating the mug and placed it in cabinet_1.The final feedback records successful heating, placement, and task completion.Source: §G.2 · p. 24
  • The Builder attributed the failure to imperfect rules because the existing “Success Process” rule omitted the requirement to be at the appliance before using it.The Builder classified the documented rule as incomplete rather than identifying a failure in the object-search rule.Source: §G.2 · p. 25
  • The Builder proposed a “Corrected Error” rule and an expanded “Success Process” rule that explicitly encode the appliance-location precondition.The new rule targets appliance use specifically, while the existing success rule is updated to incorporate the same prerequisite.Source: §G.2 · pp. 22, 25, 26
  • The Builder updates the success-process rule and adds a corrected-error rule after an agent fails to heat a mug without first reaching the microwave.The revised rule requires locating the heating appliance before using it, preserving the error’s cause and subsequent correction.Source: §G.2 · p. 26

H.1 Prompts for Planner Agent

  • Planner prompts require agents to interpret observations and applicable rules, generate directly executable Python, and use feedback to diagnose mistakes and revise plans.The prompts also require stepwise code with assertions and prohibit undefined functions or placeholder implementations.Source: §H.1 · pp. 27–28

H.2 Examples for ALFWorld and MiniWob++

  • The ALFWorld put example instantiates rules for searching receptacles, taking an object, and placing it, while MiniWob++ planning pauses between step segments to observe outcomes.The induced rules distinguish initial visibility, a reusable find_object helper, and the success process for putting an object on a receptacle.Source: §H.2 · pp. 28–30
  • MineWob++ examples show trajectories becoming executable rules for search navigation and text entry, including pagination helpers and successful action sequences.The search example derives rules for entering a query, paging when the target index exceeds three results, and clicking the target; the text-entry example succeeds.Source: §H.2 · pp. 31–32

H.3 Prompts for Builder Agent

  • The Builder classifies each trajectory as an imperfect rule or agent, then writes, updates, or stops rules while preserving validated details and limiting the rule set to 12.Rules are organized as success processes, helper methods, corrected errors, unresolved errors, and special phenomena, with conservative scope and verification requirements.Source: §H.3 · p. 33
  • Case-specific Builder prompts preserve successful procedures, record corrective code for missteps, and log unresolved failures without inventing definitive explanations.Successful trajectories update “Useful Helper Method” and “Success Process” rules; unresolved failures create only “Unresolved Error” rules, while missteps update or emphasize existing rules.Source: §H.3 · p. 34

H.4 Prompts for Consolidator Agent

  • The next subsection begins by presenting the system prompt for the Consolidator agent observing a housekeeper agent in a simulated environment.Only the Consolidator’s role introduction is provided in this part.Source: §H.4 · p. 35
  • The Consolidator Agent merges, updates, or deletes trajectory-derived rules, maintains at most 12 rules, and preserves details in Success Process and Useful Helper Method rules.Rules include descriptions, types, examples, and validation records linked to epoch and rule IDs; merging rewrites one rule before deleting another.Source: §H.4 · pp. 35–36

H.5 Prompts for Formulator Agent

  • The Formulator Agent organizes current rules by use case and topic, assigns each rule to one category, and writes a structured manual with general understandings and ordered categories.The manual presents an overview, category introductions, and rule IDs, with categories arranged from easy to difficult and basic to complex.Source: §H.5 · pp. 36–37

Related papers

Loading 2405.16247v4…