AgentBench: Evaluating LLMs as Agents

Motivation

• Unlike prior text-game benchmarks with closed action spaces and embodied multimodal simulators, AGENTBENCH targets practical, diverse, text-only agent evaluation.Existing benchmarks often emphasize commonsense grounding, use complicated multimodal simulators, or focus on a single environment (Osborne et al., 2022; Reed et al., 2022).Source: §1 · p. 2

Benchmark design

• AGENTBENCH evaluates LLM-as-Agent across eight interactive environments spanning code, game, and web grounding, including five environments created for the first time.The benchmark reformulates datasets to support autonomous operation by text-only LLMs in interactive environments.Source: §1 · p. 2

Capabilities evaluated

• AGENTBENCH systematically tests instruction following, coding, knowledge acquisition, logical reasoning, and commonsense grounding through interactive tasks.The benchmark is designed as a testbed for both LLM and agent evaluation.Source: §1 · p. 2

Empirical findings

• The benchmark evaluates 29 API-based and OSS LLMs, finding that top-tier models such as GPT-4 handle diverse real-world tasks while many OSS models no larger than 70B lag considerably.These results demonstrate a significant performance gap between leading commercial API models and OSS competitors.Source: §1 · pp. 2–3

Failure analysis and improvements

• Failure analysis attributes agent limitations mainly to insufficient long-term reasoning, decision-making, and instruction following, while alignment on high-quality data may help and code training has mixed effects.Code training improves some agent tasks but harms others, making it a double-edged intervention.Source: §1 · p. 3

Toolkit

• The authors release an integrated, modular Server-Client evaluation toolkit with associated datasets and environments to support customizable LLM assessments over HTTP.The toolkit is designed for scalable evaluation and is openly accessible to the research community.Source: §1 · p. 3

Definition: Interactive Evaluation of LLM-as-Agent

• Interactive evaluation models LLM-as-Agent as a Partially Observable Markov Decision Process with state, action, transition, reward, instruction, and observation spaces.The LLM agent is denoted M.Source: §2 · p. 3

Chain-of-Thought (CoT) and Other Reasoning Strategies

• AgentBench evaluates LLMs using primitive Chain-of-Thought with actions rather than ensemble, reflection, search, repeated generations, or multiple trials.The paper characterizes this CoT setup as the easiest, cheapest, and most common deployment strategy for LLM agents.Source: §2 · p. 3

Typical Types of Finish Reasons

• Even gpt-4 is not qualified as a practically usable agent, and AgentBench categorizes agent outcomes into five typical finish reasons.These include Context Limit Exceeded, Invalid Format, Invalid Action, Task Limit Exceeded, and Complete.Source: §2 · pp. 3–4
• Invalid Format and Invalid Action mainly reflect poor instruction following, whereas Task Limit Exceeded often indicates weak multi-turn ability in certain tasks.Context Limit Exceeded occurs only for the 2,048-length LLMs text-davinci-002 and text-davinci-003.Source: §2 · p. 4

3 COMPOSITION OF AGENTBENCH: A BRIEF LOOK

• AgentBench combines code-, game-, and web-grounded scenarios to evaluate LLM agents’ practical reasoning and decision-making through Chain-of-Thought prompting.Its environmental versatility is intended to avoid overperformance by task-specific models, such as code-specific LLMs.Source: §3 · p. 4

3.1 CODE-GROUNDED ENVIRONMENTS

• Code-grounded environments test interactive OS manipulation, authentic SQL workflows, and partially observable knowledge-graph question answering.They use success rate for OS and database tasks, and answer F1 for knowledge-graph question answering.Source: §3.1 · p. 4

3.2 GAME-GROUNDED ENVIRONMENTS

• Game-grounded environments assess strategy, instruction following, commonsense, and reasoning through Aquawar, lateral-thinking puzzles, and ALFWorld house-holding tasks.The reported metrics are win rate for Aquawar and success rate for ALFWorld, while lateral-thinking puzzles target unconventional reasoning.Source: §3.2 · pp. 4–5

3.3 WEB-GROUNDED ENVIRONMENTS

• Web-grounded environments adapt WebShop and Mind2Web to evaluate prompted LLMs on autonomous shopping and intricate cross-domain website tasks.The adaptations support evaluation without additional fine-tuning, covering searching, viewing, choosing, clicking, selecting, and typing.Source: §3.3 · p. 5

4.1 EVALUATION SETUP

• AgentBench evaluates 29 models on eight publicly available, practical multi-round datasets requiring an estimated 5–50 rounds per problem, with 269 Dev and 1,014 Test samples.The evaluation uses approximately 3k and 11k inference calls, respectively, and computes overall scores with task-normalizing reciprocal-average weights.Source: §4.1 · pp. 6–7
• The toolkit supports HTTP-accessible model servers and isolates complex environments in Docker images and separate workers for reproducible API-centric evaluation.The prompt format alternates user instructions or feedback with agent responses, while non-chat completion models receive role-prefixed histories.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

• gpt-4 achieves the best performance on 6 of 8 datasets, including a 78% House Holding success rate, while all API-based models exceed 1.00 overall.claude-2 and claude follow gpt-4 and outperform gpt-3.5-turbo.Source: Table 3 · p. 7, §4.2 · p. 7
• OSS models average 0.51 versus 2.32 for API-based models; codellama-34b leads OSS models at 0.96 but remains below gpt-3.5-turbo.OSS models commonly fail on challenging tasks including Knowledge Graph, Digital Card Game, and House Holding.Source: Figure 3 · p. 8

4.3 ANALYSIS

• Task Limit Exceeded is the predominant failure type, indicating weak reasoning and decision-making, while formatting and action-space errors concentrate in specific environments.Invalid Format errors frequently occur in Database and Digital Card Game, whereas Invalid Action errors often affect House Holding and Web Browsing.Source: §4.3 · p. 8, Table 4 · p. 8
• Code tuning improves performance on relatively static procedures such as Web Shopping but can reduce general thinking performance on Digital Card Game and Operating System.The comparison of codellama and llama-2 indicates a trade-off between procedural following and general reasoning.Source: §4.3 · p. 8
• High-quality alignment improves agent performance: vicuna-13b outperforms llama-2-13b and performs comparably to the three-times-larger codellama-34b.The models share the same base LLM, but vicuna-13b uses ShareGPT alignment data while llama-2-13b is aligned from scratch.Source: §4.3 · p. 8
• llama-2-13b and llama-2-70b perform similarly despite their size difference, potentially because llama-2-70b received insufficient pre-training or instruction alignment.Both models were pretrained on 2T tokens, and the analysis also identifies weaker instruction following as a possible explanation.Source: §4.3 · p. 8

• Existing LLM benchmarks broadened evaluation beyond specified tasks but generally remained limited to traditional datasets and tasks, excluding open-ended generation, multi-round interaction, and agent behavior.Source: §5 · p. 9
• LLM-as-Agent research evolved from text-game environments to Chain-of-Thought-based reasoning, ReAct, advanced strategies, frameworks, and multi-agent applications, but lacked a standard comprehensive benchmark.AgentBench presents the first systematic benchmark with broad task coverage and available LLMs, while proposing agent tasks as a measure of LLM performance.Source: §5 · p. 9
• Interactive-environment evaluation expanded from text games to code execution for functional correctness, yet most code frameworks omitted multi-round interaction; InterCode supports Bash and SQL interactions similar to AgentBench’s OS and DB tasks.Source: §5 · p. 9

6 CONCLUSION

• AGENTBENCH evaluates up to 29 LLMs through a systematically designed, multi-dimensional benchmark and unified testing toolkit intended to support subsequent LLM-agent research.The benchmark also reports insights into agent tasks, LLM behaviors, and potential improvement methods.Source: §6 · p. 9

Appendix

• The benchmark includes operating-system tasks answered with valid bash commands and database tasks answered with valid SQL commands.The OS example asks for non-empty directories inside /etc, while the database example asks for total U.S. Olympic medals.Source: Appendix · p. 19

A.1 TRADITIONAL EVALUATION FRAMEWORKS

• Traditional evaluation frameworks comprise single-round tasks such as generation and classification, which may not suit complex multi-round interactions.These frameworks are designed for specific tasks rather than broader interactive evaluation.Source: §A.1 · p. 20
• Agent-based evaluation frameworks support multi-round interactions but are typically tailored to one task by dataset creators.Their task-specific design limits applicability to other tasks.Source: §A.1 · p. 20
• Traditional agent-based frameworks often require task, agent, and evaluation communication within one process or through child processes on the same device.This constrains how evaluation can be deployed.Source: §A.1 · p. 20
• Traditional agent-based frameworks can evaluate only one task with one agent at a time.This limits evaluation breadth and parallelism.Source: §A.1 · p. 20

A.2 OUR DESIGNED EVALUATION FRAMEWORK

• The decoupled Task Server, Agent Server, and Evaluation Client communicate through HTTP, enabling separate deployment across devices without co-location.Source: §A.2 · p. 20
• The framework evaluates multiple agents and tasks in varied combinations simultaneously, enabling more comprehensive testing scenarios.Source: §A.2 · p. 20
• The framework combines network-flow-based efficiency with resumable evaluation to maximize worker utilization and recover interrupted evaluations.Source: §A.2 · p. 20
• These design features provide a more versatile, efficient, and scalable solution for evaluating intelligent agents in multi-round tasks.Source: §A.2 · p. 20

A.3 IMPLEMENTATION OF MAX-FLOW ALGORITHM

• The evaluation process uses Edmonds–Karp, an implementation of Ford–Fulkerson, to compute maximum flow with time complexity O(|V||E|^2).The algorithm is attributed to Edmonds & Karp (1972) and Ford Jr & Fu˛lkerson (1962).Source: §A.3 · p. 20
• The flow model represents n agents and m tasks, organizes evaluations into l agent-task groups, and assigns s_k samples to each pair.Source: §A.3 · p. 20
• Max-flow allocation assigns f(i,j) samples to each agent-task edge, then reduces capacities and replenishes source or destination edge weights after completed evaluations.Source: §A.3 · p. 21
• The algorithm is periodically reapplied to the network to schedule newly available evaluation triples.Source: §A.3 · p. 21

B.1 DATASET DETAILS

• OS tasks comprise QA commands that require final answers and operation commands that produce verifiable system-state changes, unified by checking pipelines.Each execution initializes a container, supports bash or commit actions, enforces an 8-round default limit, and succeeds only when every checking script exits with code 0.Source: §B.1 · p. 22
• The OS benchmark curates 144 diverse evaluation samples with interactive Docker environments, checking pipelines, and 1-shot CoT formatting.About half the instructions come from humans and the remainder are GPT-4-generated QA problems filtered by unit tests.Source: §B.1 · p. 22

B.2 ACTIONS

• OS agents choose between bash execution and commit, which submits either a QA answer or completion of a system operation for checking.The prompt example additionally requires exactly one of bash, finish, or answer at each turn and demonstrates recovery from truncated output by scripting directly.Source: §B.2 · p. 22, §B.3 · p. 23

C.1 DATASET DETAILS

• The database benchmark contains 300 augmented samples spanning select, insert, and update operations, with table metadata, contents, and operation-specific correct-answer representations.It combines WikiSQL, WikiTableQuestions, SQA, HybridaQA, and FeTaQA with GPT-3.5-Turbo-generated rows, SQL queries, and paraphrases.Source: §C.1 · p. 24
• Database evaluation initializes MySQL in Docker, iterates executable SQL interactions, and checks exact selection answers or post-operation table hashes; success rate macro-averages three categories.Equivalent numeric answer representations are accepted for single-number selections.Source: §C.1 · p. 25

C.2 DATA AUGMENTATION

• Database augmentation extends QA-only source datasets with generated insert and update statements, paraphrases them without changing meaning, and retains queries that pass unit tests.Existing queries are also categorized as Counting, aggregation, Ranking, Comparison, or Other.Source: §C.2 · p. 25

C.3 PROMPT EXAMPLE

• The database prompt requires explained reasoning plus one executable SQL statement per round, then an explicitly formatted final answer that must exactly match the correct answer.Malformed responses fail immediately, while modification tasks may use any answer field after the database operation completes.Source: §C.3 · pp. 25–26

C.4 STUDY ON BIAS IN DATA AUGMENTATION

• A small batch was re-annotated with Claude-2 to validate whether model-induced biases entered the dataset during augmentation.gpt-4 and gpt-3.5-turbo served as example models in the validation.Source: §C.4 · p. 26
• The preserved relationships across operations indicate that the augmentation approach is unlikely to introduce substantial biases.This conclusion follows from the consistently similar scoring patterns between re-annotated and original data.Source: §C.4 · p. 26
• gpt-4 performed less effectively on UPDATE operations but showed enhanced proficiency in INSERT tasks, consistently matching the original data’s scoring patterns.The comparison used re-annotated data tested on gpt-4 and gpt-3.5-turbo against the original data.Source: Table 5 · p. 26, §C.4 · p. 26

D.1 DATASET DETAILS

• The dataset contains 1,663 Freebase questions requiring at least five tool invocations, selected from GrailQA, ComplexWebQuestions, and GraphQuestions to test long-term planning.S-expression annotations define the optimal tool sequence for each question.Source: §D.1 · p. 26
• Each task provides a natural-language question, topic entities, a gold tool-action sequence, and a gold answer, while avoiding entity linking so the model focuses on planning.The gold answer is typically a set of knowledge-graph entities.Source: §D.1 · p. 26
• The partially observable setting omits a complete knowledge-graph description from the input, requiring the LLM to discover relevant information through interaction.Unlike database environments whose contents are integrated into the input, the extensive knowledge graph cannot be fully described to the model.Source: §D.1 · p. 27
• Evaluation uses API-mediated Virtuoso access and measures answer F1, exact match between predicted and gold answer sets, and whether the generated action sequence executes successfully.The evaluation uses the first 500 dataset tasks; executability is 1.0 when execution produces any answer and 0 otherwise.Source: §D.1 · p. 27

D.2 PROMPT EXAMPLE

• The prompt defines sequential operations for relation discovery, neighbor retrieval, set intersection, attribute lookup, extrema selection, and counting over entities or variables.Operations impose dependencies, such as requiring relation discovery before neighbor retrieval and attribute lookup before argmax or argmin.Source: §D.2 · pp. 27–28
• Because zero-shot models struggle to produce substantive knowledge-base queries, the prompt includes a teaching example demonstrating iterative thought, action, observation, variable composition, and final-answer selection.The example uses relation lookup, neighbor retrieval, intersection, and designer retrieval, while the protocol permits one action at a time and at most 15 actions.Source: §D.2 · pp. 28–29

E.1 DATASET DETAILS

• The game uses two teams of four fish, hidden identities, round-based attacks and assertions, triggered passive abilities, and victory by having more fish alive.Each fish starts with 400 health and 200 attack power; final scores average performance across assertion-free and assertion-enabled stages.Source: §E.1 · p. 30

E.2 THE ATTRIBUTES OF FISH

• The ten-fish pool combines overlapping active and passive abilities to conceal identities and increase strategic uncertainty.Examples include area attacks, damage deflection, healing, critical attacks, damage avoidance, and conditional counterattacks.Source: §E.2 · pp. 30–31

E.3 PROMPT EXAMPLE

• Prompts provide game rules, fish attributes, hidden-identity information, and required JSON actions, while requiring reasoning and a move every round.Stage-two prompts separately request an enemy-fish identity guess and target position.Source: §E.3 · pp. 31–33

E.4 BATTLE GENERATION

• Battle generation fixes presets for fair main-text evaluation but can randomize team composition and fish attributes across selectable difficulty levels.Difficulty variation is defined through the relative combat power of opposing teams.Source: §E.4 · p. 33

E.4.1 COMBAT POWER

• For a single fish, combat power is defined as HP · ATK; for teams, it aggregates total health with average attack power under equal attack-frequency assumptions.The formulation is motivated by comparing survival duration and expected standard-attack damage.Source: §E.4.1 · pp. 33–34

E.4.2 DIFFICULTY: A RATIO OF COMBAT POWERS

• Game difficulty is quantified by the ratio of friendly to hostile combat power, with a ratio of 1 representing balanced difficulty.This ratio enables researchers to set difficulty levels and compare strategy win rates.Source: §E.4.2 · p. 34

F.1 DATASET DETAILS

• Each sample pairs a story with its truth and assigns one of four difficulty levels: easy, medium, hard, or expert.The solver must infer the truth from the host’s responses during an interactive game.Source: §F.1 · p. 34
• The solver asks one Yes/No/Irrelevant question per round for up to a fixed maximum, may receive hints, and declares the truth when it believes the main plot is identified.The host knows the story and truth, while the LLM acts as solver and synthesizes the host’s answers.Source: §F.1 · p. 34
• LTP performance is measured with Single Game Accuracy, Round Efficiency, Query Relevance, and Game Progress, with Game Progress serving as the main metric.Game Progress counts the proportion of ground-truth points reached by the agent.Source: §F.1 · pp. 34–35

F.2 EVALUATION ON LTP SYSTEM

• Automatic evaluation can overestimate Single Game Accuracy and Query Relevance, especially for open-sourced models, whereas Game Progress and Round Efficiency match human evaluation more closely.Human validation assessed milestone recognition and fact verification; the authors plan a dedicated host model for more precise evaluation.Source: §F.2 · p. 35

F.3 LTP GAME PROGRESS AND TERMINATION

• Game Progress represents the proportion of truth key points hit, using dataset-level answer_keys summarized by gpt-3.5-turbo.The evaluation converts relevant questions into declarative sentences, simplifies them, and removes already identified points and inferences to prevent repeated credit.Source: §F.3 · p. 35

F.4 PROMPT EXAMPLE

• The solver prompt enforces one question at a time, answerability by Yes, No, or Irrelevant, deduction rather than direct requests, and prohibits ending or restarting the game.The host prompt requires responses aligned with the hidden truth, permits Irrelevant only when no direct or indirect answer exists, and allows limited entry-point hints.Source: §F.4 · pp. 35–36

G.1 DATASET DETAILS

• ALFWorld models household scenarios in which agents decompose high-level goals into sequential actions and adapt plans using feedback after each step.Each sample includes an environment description, objective, and simulated environment that evaluates actions and reports changes or observations.Source: §G.1 · p. 38
• The evaluation uses 134 solvable out-of-distribution problems spanning six categories: pick and place, clean-then-place, heat-then-place, cool-then-place, look-at-object, and pick-two-objects.Source: §G.1 · p. 38
• Each category receives one successful training example in a 1-shot setup, while invalid outputs are mapped to the most similar legal action using BLEU.The setup follows the few-shot examples and prompts from ReAct (Yao et al., 2023b).Source: §G.1 · pp. 38–39
• During interaction, the model generates thoughts and actions from environment feedback until completing the goal or reaching the action limit; three identical consecutive outputs count as failure.Source: §G.1 · p. 39

G.2 PROMPT EXAMPLE

• The prompt requires choosing either THOUGHT or ACTION and strictly formatting the response with a legal action selected from the environment’s available options.A “Nothing happened” response indicates an invalid action, prompting the agent to try other options.Source: §G.2 · p. 39
• The worked example demonstrates searching cabinets for a spraybottle, taking it, navigating to a toilet, and placing it there through successive feedback-driven actions.Source: §G.2 · pp. 39–40

H.1 DATASET DETAILS

• WebShop uses about a million scraped Amazon products, each annotated with attributes, alongside 12,087 human instructions linked to goals and expected attributes.The construction follows Yao et al. (2022) for additional dataset details.Source: §H.1 · p. 40
• The test set comprises the first 500 of the 12,087 instructions, following Yao et al. (2022)’s official implementation.Each interaction begins with environment and response-format information before the product-buying instruction is issued.Source: §H.1 · p. 40
• Agents repeatedly search or click through simplified webpage observations until buying a product or exceeding the round limit.The environment returns textual webpage content and available buttons after each action.Source: §H.1 · p. 40
• Performance is measured by a matching reward from 0 to 1 based on similarity between expected and purchased-product attributes.TextMatch compares pronouns, nouns, and proper nouns in the chosen and goal product titles, while U and Y denote the goal and chosen product.Source: §H.1 · p. 40

H.2 PROMPT EXAMPLE

• The prompt instructs agents to respond to each observation and available-action list using search[keywords] or click[value].Search keywords are freely chosen, whereas click values must come from the listed available actions.Source: §H.2 · p. 41
• The required response format separates internal reasoning from the selected action with the labels “Thought:” and “Action:”.The prompt includes an observation and available-actions structure for each round.Source: §H.2 · p. 41
• A one-shot example searches for a long-lasting 6.76 fl oz L’eau d’Issey bottle priced below 100.00 dollars and clicks the product identified as B000VOHH8I.Source: §H.2 · pp. 41–42

I.1 DATASET DETAILS

• Mind2Web’s Cross Domain test set contains 912 tasks from 73 websites, with each sample pairing a high-level task, reference action sequence, and webpage observations.Evaluation ranks HTML elements with a fine-tuned model, presents five candidates as a multi-choice question, and supplies operation arguments for typing or selecting options.Source: §I.1 · p. 43
• Mind2Web agents receive current page content and action histories, then predict the next target element and operation, while HTML ranking and filtering improves inference efficiency.Step Success Rate is the main metric because task success remains difficult; task success requires every step to succeed and is currently single-digit even for the best LLMs.Source: §I.1 · p. 43

I.2 PROMPT EXAMPLE.

• The Mind2Web prompt uses three chain-of-thought examples that format webpage HTML, task instructions, prior actions, multiple-choice candidates, and the predicted operation with its value.Examples include selecting a pickup option, rejecting unavailable actions with “None of the above,” and clicking a date after entering a rental location.Source: §I.2 · pp. 43–45

J.1.2 DEFINITION OF VALIDITY ANALYSIS

• Validity analysis classifies outcomes as Completed, Context Limit Exceeded, Invalid Format, Invalid Action, or Task Limit Exceeded to identify model-specific failure modes.These categories distinguish API length constraints, response-format failures, action-space or parameter errors, and termination after exceeding the allowed rounds.Source: §J.1.2 · p. 45

J.1.3 VALIDITY ANALYSIS OF MODELS

• The validity study examines 29 models, with text-davinci uniquely subject to an inherent strict API context-length constraint and other models mainly distributed across four remaining categories.Figure 6 visualizes validity distributions across models and categories; invalid format, invalid action, and text-limit errors are common, while context-limit errors appear only for text-davinci.Source: §J.1.3 · pp. 45–46, Figure 6 · p. 47

J.2.1 INSTRUCTION FOLLOWING MATTERS

• Open-source models produced more invalid formats (10.4% vs. 6.0%) and invalid actions (13.6% vs. 4.6%) than commercial API-based models.The differences may reflect robustness, generalization, or instruction-following weaknesses; even GPT-4 sometimes returned neither the required Action label nor a correct SQL statement.Source: Table 6 · p. 46, §J.2.1 · p. 46

J.2.2 CONSISTENCY AND EXECUTION IN AGENT PLANNING

• House Holding exposes wide differences in whether models maintain coherent plans from formulation through execution, because its many entities and actions demand consistent reasoning.Models may form an initial plan but later deviate from or forget it, making consistency a major determinant of agent effectiveness.Source: §J.2.2 · pp. 47–48
• gpt-4 completed the task through a Find -> Clean -> Put plan, depth-first search, and successful backtracking after each exploration.Figure 7 illustrates this systematic sequence and coherent execution.Source: §J.2.2 · p. 50, Figure 7 · p. 51
• gpt-4 self-reflected after failing to find the soapbar in the toilet, reassessed its assumption, and checked the remaining countertop.This readjustment enabled it to complete the task after deeper contemplation.Source: §J.2.2 · pp. 50–51
• gpt-3.5-turbo decomposed tasks but struggled to follow its initial plan, gradually losing sight of it after failed attempts.Source: §J.2.2 · p. 54

J.2.3 UNDERSTANDING TASK COMPLETION: DISTRIBUTION OF TOKENS AND ROUNDS IN COMPLETED TRAJECTORIES

• 6.0 rounds median, 7.95 average, and 1850.0 tokens median, with middle-50% ranges of 4.0–9.0 rounds and 761–2709 tokens; most tasks finish within 3000 tokens.Source: Figure 8 · p. 54, §J.2.3 · p. 54

J.2.4 PRIMARY CAUSE OF TASK LIMIT EXCEEDED: THE MODELS TEND TO REPEAT THE PREVIOUS CONTENT

• The TLE analysis measures repeated responses across the last n rounds after truncating trajectories to prefixes of at most 3500 tokens, capturing cyclic behavior beyond adjacent-turn duplication.Repetition is defined using a high Rouge-L f-score between two responses, allowing loops through multiple states to be detected.Source: §J.2.4 · p. 55
• More than 90% of TLE trajectories contain two responses in the last 10 rounds with a Rouge-L score of at least 0.8, indicating repetition is the main failure cause.TLE trajectories average 25.5 rounds and are typically terminated by the round limit.Source: §J.2.4 · p. 55

J.2.5 THE INFLUENCE OF CODE TUNING ON LLM ACTING AS AGENTS

• CodeLlama consistently outperforms Llama2 on WebShop tasks, where following a one-shot procedural shopping template is sufficient for satisfactory performance.The WebShop completion-rate comparison is illustrated in Figure 10.Source: §J.2.5 · p. 56, Figure 10 · p. 56
• Code tuning can compromise logical reasoning and situational awareness: CodeLlama lags behind Llama2 in the Digital Card Game, which requires assessing competitors and devising counter-strategies.The contrast is attributed to WebShop’s explicit procedural template versus the card game’s need for complex state-based reasoning.Source: §J.2.5 · p. 56

J.2.6 SELF CORRECTING ABILITY

• Models that self-correct SQL statements from environmental error feedback significantly outperform models without this ability, especially on the DB task.The main failure mode is inability to identify mistakes revealed by the environment.Source: §J.2.6 · p. 56
• Claude-2 corrected successive SQL syntax errors by inferring from MySQL feedback that backticks were required around fields and the table name containing spaces.The log shows successful diagnosis of the omitted backticks after repeated execution errors.Source: §J.2.6 · pp. 57–58