Source-linked AI summary
Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions
Hui Yang, Sifu Yue, Yunzhong He
TL;DR
The paper asks whether Auto-GPT styled agents can effectively handle real-world-like online decision-making despite limited benchmarks and engagement. It benchmarks several LLMs and adds supervised learners’ opinions to Auto-GPT, finding improved performance, especially for GPT-4, without extensive fine-tuning. The results support lightweight expert-model collaboration while remaining limited to two benchmark tasks.
Problem
Evidence on the effectiveness and flexibility of Auto-GPT styled agents for real-world decision-making remains limited, partly because real-world engagement and benchmarks are lacking.
Method
The paper benchmarks GPT-4, GPT-3.5, Claude, and Vicuna on online decision-making tasks and introduces Additional Opinions from supervised or imitation-based expert models.
Results
Across the evaluated tasks, Auto-GPT adapts successfully through prompt design, GPT-4 performs best among the tested foundational LLMs, and Additional Opinions further improve performance, particularly for GPT-4.
Takeaways & Limitations
Additional Opinions provide a lightweight way to introduce supervision into Auto-GPT styled agents without extensive fine-tuning of the LLMs.
Takeaways & Limitations
The two benchmarked tasks do not fully represent the range of potential real-world scenarios and are only a starting point for evaluating Additional Opinions.
Abstract
from arXiv · showhide
Auto-GPT is an autonomous agent that leverages recent advancements in adapting Large Language Models (LLMs) for decision-making tasks. While there has been a growing interest in Auto-GPT stypled agents, questions remain regarding the effectiveness and flexibility of Auto-GPT in solving real-world decision-making tasks. Its limited capability for real-world engagement and the absence of benchmarks contribute to these uncertainties. In this paper, we present a comprehensive benchmark study of Auto-GPT styled agents in decision-making tasks that simulate real-world scenarios. Our aim is to gain deeper insights into this problem and understand the adaptability of GPT-based agents. We compare the performance of popular LLMs such as GPT-4, GPT-3.5, Claude, and Vicuna in Auto-GPT styled decision-making tasks. Furthermore, we introduce the Additional Opinions algorithm, an easy and effective method that incorporates supervised/imitation-based learners into the Auto-GPT scheme. This approach enables lightweight supervised learning without requiring fine-tuning of the foundational LLMs. We demonstrate through careful baseline comparisons and ablation studies that the Additional Opinions algorithm significantly enhances performance in online decision-making benchmarks, including WebShop and ALFWorld.
1 INTRODUCTION
The paper studies whether Auto-GPT styled agents can adapt to online decision-making tasks with limited human guidance and real-world-like complexity. It benchmarks multiple LLMs and introduces external expert opinions as a lightweight way to improve autonomous decisions without fine-tuning.
- Auto-GPT styled agents receive high-level goals, reason and plan step by step, use tools, retrieve long-term memory, and require minimal task-specific adaptation.
- The study adapts Auto-GPT to online decision-making tasks involving unknown external environments to examine performance and limitations.
- Additional Opinions incorporate a second opinion from a supervised learner, providing supervision without fine-tuning the foundational model.
- The benchmark compares GPT-4, GPT-3.5, Claude, and Vicuna as foundations for autonomous agents.
2 METHODOLOGY
The methodology evaluates Auto-GPT in WebShop and ALFWorld using direct task goals and tool representations, then augments its decision phase with top-k opinions from external expert models. The workflow uses imitation-learning models as experts and adds their suggestions to the Auto-GPT context before the LLM decides.
- WebShop simulates online shopping with product search, browsing, navigation, and purchase actions, while success requires matching the intended product, attributes, options, and price.
- ALFWorld combines ALFRED language-directed tasks with TextWorld interactive fiction to test language understanding and decision-making in interactive environments.
- Auto-GPT is adapted to both tasks without extensive tuning by providing task requirements as goals and representing available actions as tools.
- Considering additional opinions: Top-k opinions are sampled from an external expert model and inserted into the Auto-GPT prompt context during the decision phase.
- Considering additional opinions: The experiments use readily available imitation-learning models as external experts whose action suggestions are presented for the LLM’s independent judgment.
3 EXPERIMENTS
Experiments benchmark Auto-GPT agents and imitation-learning baselines on WebShop and ALFWorld, then test how additional expert opinions affect performance. GPT-4 generally outperforms the compared models, while the usefulness of imitation-learning advice depends on task context.
- WebShop: The WebShop evaluation uses the first 50 instructions and compares Auto-GPT agents using multiple LLMs with imitation-learning models.The test-set restriction reflects cost and computational-efficiency concerns, especially for GPT-4.
- WebShop: GPT-4 outperformed both imitation-learning variants on WebShop, whereas Claude and GPT-3.5 performed worse than the original imitation-learning models.The comparison included imitation-learning models with and without image embeddings.
- ALFWorld: On ALFWorld, GPT-4 exceeded the imitation-learning model despite receiving fewer allowed steps, while Claude and GPT-3.5 did not surpass it.The imitation-learning success rate fell from 0.306 to 0.179 without Beam Search.
- WebShop: GPT-4 achieved a 0.24 success rate alone on WebShop, compared with 0.140 for Claude and 0.120 for GPT-3.5.Vicuna was assigned 0 because it could not generate formatted responses; Claude was faster than GPT-3.5 and GPT-4.
- Additional opinions: Adding one imitation-learning choice improved every LLM, and GPT-4 with five additional opinions achieved the best Success Rate, Rewards, and Precision across the tested groups.GPT-4 benefited despite already outperforming the additional opinions supplied by imitation-learning models.
- Additional opinions: The effect of imitation-learning advice was context-dependent: WebShop guidance condensed choices, whereas ALFWorld advice could be repetitive or misleading.GPT-4 considered opinions more often as the number increased from 1 to 5, rising from 0.549 to 0.602, and showed a disagreement rate of 0.854 in the reported comparison.
4 RELATED WORK
Prior work has improved LLM planning and reasoning through prompting, Chain of Thought, memory retrieval, and self-consistency techniques, but online decision-making remains comparatively underexplored.
- LLMs have been adapted for decision-making, logic reasoning, external API interaction, virtual character simulation, and tool manipulation.
- Prompting, planning, and memory-retrieval techniques have been proposed to address limitations in using LLMs directly as autonomous agents.
- Chain of Thought methods enable explicit reasoning processes, while prompting and decoding techniques target improved self-consistency.
- Most existing techniques focus on offline reasoning tasks that can be planned ahead, leaving their implications for online decision-making rarely discussed.
5 CONCLUSION
The paper adapts Auto-GPT styled agents to complex online decision-making tasks and finds that GPT-4 and external supervised opinions improve performance without extensive LLM fine-tuning.
- Auto-GPT styled agents successfully handle complex online decision-making tasks through straightforward prompt design, surpassing IL-based baselines.
- GPT-4 demonstrates superior performance among the foundational LLMs evaluated for Auto-GPT agents.
- Additional Opinions from external expert models further enhance Auto-GPT decision-making, particularly for GPT-4.
- The approach provides lightweight supervised training for Auto-GPT styled agents without requiring extensive fine-tuning of the underlying LLMs.
- The approach is especially adaptable to tasks with easily collectible training data for action policy.
A.1.1 Example 1. Webshop: AutoGPT(GPT-4) disagreed with the Additional Opinion.
In this WebShop example, Auto-GPT(GPT-4) rejects an unnecessary back-navigation suggestion and selects a matching pink two-pack hair towel priced below $20.
- Example 1. Webshop: AutoGPT(GPT-4) disagreed with the Additional Opinion.: The instruction requests a pink Niuta two-pack hair towel for dry hair priced below $20.
- Example 1. Webshop: AutoGPT(GPT-4) disagreed with the Additional Opinion.: The selected product is a NIUTA 2 Pack Hair Towel Wrap in Pink+Light Blue, listed at $4.99.
- Example 1. Webshop: AutoGPT(GPT-4) disagreed with the Additional Opinion.: The agent identifies item B08G14B779 as matching the description and reports its price as $4.99.
- Example 1. Webshop: AutoGPT(GPT-4) disagreed with the Additional Opinion.: The agent plans to click the matching item, review its details, and make the purchase.
- Example 1. Webshop: AutoGPT(GPT-4) disagreed with the Additional Opinion.: It criticizes the suggestion to click back to search because the matching item has already been found.
A.1.2 Example 2. Webshop: AutoGPT(GPT-4) agreed with the Additional Opinion.
In this WebShop example, Auto-GPT selects a storage ottoman matching the stated faux-leather, size, and price requirements after agreeing with the Additional Opinion.
- Example 2. Webshop: AutoGPT(GPT-4) agreed with the Additional Opinion.: The instruction requests a folding faux-leather storage ottoman, sized 60x40x40cm, priced below $160, and easy to install.
- Example 2. Webshop: AutoGPT(GPT-4) agreed with the Additional Opinion.: The selected item is B09NXP95GC, a faux-leather storage stool listed at $139.42.
- Example 2. Webshop: AutoGPT(GPT-4) agreed with the Additional Opinion.: The agent states that B09NXP95GC matches the size, material, and price requirements.
- Example 2. Webshop: AutoGPT(GPT-4) agreed with the Additional Opinion.: The agent plans to click the item, review its details, and make the purchase.
A.2.1 Example 1. ALFWorld: DAgger Agent repeated actions for "pick and place" task.
The ALFWorld pick-and-place examples show long, repetitive navigation sequences before completing or attempting object placement tasks. The traces illustrate repeated actions across cabinet, shelf, and desk environments.
- The PepperShaker task repeats navigation to cabinets many times before placing the saltshaker in drawer 3.The trace eventually takes saltshaker 1 from cabinet 3, cleans no object, and places it in drawer 3.
- The Watch task repeatedly returns to shelf 12 without showing a completed placement action.The sequence contains many repeated “go to shelf 12” actions and ends mid-trace.
- The Mug task similarly repeats visits to shelf 5 after initially navigating through shelves 5 and 2.The trace ends during continued shelf navigation rather than showing the requested examination completed.
A.2.2 Example 2. ALFWorld: DAgger Agent effective action for "pick clean then place" task.
The pick-clean-then-place examples include concise successful action traces and detailed decision logs showing planning, tool execution, and responses to suggested actions. The logs also show repeated or corrective behavior when observations do not confirm the target.
- Successful action traces: The Cloth, Pan, and Spatula traces each clean an object at sinkbasin 1 and place it at a specified countertop or drawer.The examples show completed sequences for cloth, pan, and spatula, including opening drawer 2 when required.
- Suggestion-based decisions: The soapbar agent rejects an irrelevant handtowel suggestion, then plans to take, clean, and return the soapbar to the countertop.Its reasoning identifies soapbar 1 on countertop 1 and selects “take soapbar 1 from countertop 1” as the next command.
- Suggestion-based decisions: After checking cabinets without finding the soapbar, the agent follows a suggestion to go to countertop 1 and continues the same task plan.The recorded plan remains to find and take the soapbar, clean it, and put it on the countertop.
- Search and correction: For the mug-and-desklamp task, the agent searches likely shelves, desks, and drawers sequentially before attempting the requested examination.The initial plan is to find both objects and examine the mug with the desklamp, beginning with shelf 3.
- Search and correction: When shelf 2 yields nothing, the agent repeats its examination plan and acknowledges that the first examination was insufficiently thorough.It uses the alfworld_action command to go to shelf 2 again and search for the desklamp.