LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordination

1 INTRODUCTION

• API calls and complex prompts can make LLM inference take seconds to minutes, a poor fit for high-frequency, real-time coordination such as gaming.The introduction contrasts this with low-frequency settings such as code generation.Source: §1 · p. 1
• In Overcooked, the AI must resolve context-dependent commands such as “one more,” act promptly, and report progress to its human partner.Figure 1 presents a concrete example of human-AI cooperation and communication.Source: §1 · p. 1, Figure 1 · p. 2
• Traditional gaming AI favors fast inference with small models or scripts, but this can limit task completion and interaction abilities.The passage cites prior work for this trade-off.Source: §1 · p. 1
• HLA pairs a proficient LLM for intention reasoning and language interaction with a lightweight LLM for macro actions and a reactive Executor for atomic actions.Human commands are processed simultaneously by Slow Mind and Fast Mind; Executor supports feasible, high-frequency actions.Source: §1 · pp. 1–2
• HLA was an order of magnitude faster than the best competitor in action responsiveness, outperformed baselines in command reasoning, and achieved approximately 50% higher game scores and highest human preference.The human studies support the game-score and preference findings.Source: §1 · p. 2

Language Agents

• Instruction-following agents trained on paired text and trajectories have been studied in games, visual navigation [22] [62], and robotics [25], but these works are limited to simple domains.Source: §2 · p. 2
• Recent LLM agents use prompt engineering in complex domains, while some combine LLMs with different cooperative functions; this work studies language agents in response-speed-sensitive environments.Inference speedup approaches include parallel response generation [39], distillation [6] [12] [56] [57] [59], and quantization [16] [32] [36].Source: §2 · p. 2

Human-AI Cooperation

• Prior human-cooperation studies include communication-free games [10] [51] [68] and robotics [17], language-guided navigation and robotics [22] [62] [25], and turn-based Diplomacy.The passage notes concurrent LLM-based Overcooked work [1] [64] [69] and contrasts Diplomacy’s low real-time requirements.Source: §2 · p. 2

3.1 Environment Details

• Orders require players to retrieve, chop, mix, cook, plate, and serve ingredients before deadlines; failed deliveries incur penalties.Each player chooses among four movement actions at each time step.Source: §3.1 · p. 2
• The extended environment lets humans pause to message the AI or speak during play, while the AI can respond by text or speech.Source: §3.1 · p. 2
• Figure 2 shows four maps: Ring and Bottleneck assess general cooperation, Partition separates the players, and Quick adds concurrent orders and faster actions.The maps vary the coordination demands of the testbed.Source: §3.1 · pp. 2–3, Figure 2 · p. 3

3.2 Challenges of Overcooked

• Real-time play makes directly prompting an LLM for each next action impractical, while commands can require quantity tracking, semantic inference, or resolving ambiguous references.Examples include “chop 3 tomatoes,” a warning that soup may time out, and “one more”; interpretation may depend on commands, environment composition, and action history.Source: §3.2 · p. 3

4.1 Overview

• HLA assigns command interpretation and chat feedback to Slow Mind, medium-frequency macro-action generation to Fast Mind, and high-frequency atomic execution to Executor.The three components use a proficient LLM, a lightweight LLM, and pre-defined scripts, respectively.Source: §4.1 · p. 3

4.2 Slow Mind

• Slow Mind interprets vague commands, tracks completion, and provides responses and suggestions, using a two-stage design powered by GPT-3.5.It sends inferred intentions and completion status to Fast Mind.Source: §4.2 · p. 3

4.2.1 Intention Reasoning Stage.

• For each new command, Slow Mind uses command history and environmental information to resolve vague wording into a concrete intention, then sends it to Fast Mind.For example, “Cook the first soup on the orders” can resolve to “Cook Alice Soup” when Alice Soup is listed first.Source: §4.2.1 · p. 3

4.2.2 Chat & Assessment Stage.

• Slow Mind periodically assesses command completion and generates chat; when no command is ongoing, it switches to casual chat and omits assessment and intention input.After judging a command complete, it clears the inferred intention and tells Fast Mind the command is complete.Source: §4.2.2 · p. 4

4.3 Fast Mind

• Fast Mind uses a lightweight LLM with Slow Mind’s conditional prompting and action filtering to produce macro actions; lightweight models can make sub-optimal moves or violate vague commands.A powerful LLM is considered too slow for Fast Mind’s role.Source: §4.3 · p. 5

4.3.1 Intention Selection.

• While Slow Mind reasons, Fast Mind conditions on the raw command; it switches to inferred intention when available, then to Slow Mind’s chat after command completion.Fast Mind estimates macro-action probabilities by filling each candidate into the prompt’s next-action slot, with action history included.Source: §4.3.1 · p. 5, §4.3.2 · p. 5

4.3.2 Macro Action Generation.

• Fast Mind filters macro actions by combining language-model probabilities with hardcoded task-relevant values, adjusting their influence with command-completion status and choosing the highest-scoring action.The adjustment term is smaller before the command is assessed as met and larger afterward.Source: §4.3.2 · p. 5

4.4 Executor

• Executor uses scripts and target-directed path planning to turn Fast Mind’s 21 target-specific macro actions into atomic moves.The action set includes chopping, mixing, cooking, plating, serving, extinguishing fires, and discarding charred soup. The authors identify training a similar goal-conditioned policy with reinforcement learning as future work.Source: §4.4 · pp. 5–6

5 EXPERIMENT

• The evaluation uses three agents lacking an HLA component and tests latency, simple and complex commands, and human-player game scores and preferences.The human study has 60 volunteers, who play with SMOA, FMOA, and HLA and rank the agents based on gaming experience.Source: §5.1 · p. 6, §5 · p. 6, §5.5.1 · pp. 7–8

5.2 Latency

• HLA produces 74.3% lower macro-action latency than SMOA and 53.5% lower than FMOA, suggesting its hierarchical design significantly contributes to reducing response latency.Source: §5.2 · p. 6

5.3 Performances with Simple Commands

• NEA misses all soup orders and scores minimum −20; in One Command, SMOA often overcooks dishes, while FMOA keeps making requested soup even when it disappears from the order list.Source: §5.3 · pp. 6–7

5.4 Interpreting Complex Commands

• HLA outperforms baseline agents in success rate and completion time on complex commands; FMOA struggles with ambiguity, while SMOA is slow on quantity and semantics.NEA fails to execute any complex command.Source: §5.4 · p. 7, Table 2 · p. 7

5.5.2 Game Score.

• HLA achieves ∼50% higher game scores than SMOA and FMOA across maps, with increases over 40 points on Partition and Quick.The reported increases correspond to serving at least two additional orders.Source: §5.5.2 · p. 8, Table 4 · p. 8
• HLA executes the highest ratio of valuable macro actions and minimizes fire accidents; its Serve macro-action ratio is 100%.A valuable macro action produces positive utility upon execution.Source: §5.5.2 · p. 8, Table 5 · p. 8

5.5.3 Human Preference.

• HLA exceeds SMOA and FMOA in overall human preference by more than 30%, and is favored over SMOA by as much as 50% in competition.More than 20% of participants also prefer HLA for communication accuracy and message-action consistency.Source: §5.5.3 · p. 8, Figure 11 · p. 8, Figure 10 · p. 8

• HLA is proposed as an agent that cooperates with humans using natural language in environments requiring real-time execution.Source: §6 · p. 8
• Across comprehensive experiments in extended Overcooked, HLA excels in game score, response latency, and human preference.Source: §6 · p. 8
• The authors identify GPT-4 for Slow Mind semantic analysis and a goal-conditioned reinforcement-learning Executor as possible improvements.The latter is intended to streamline scripting and boost low-level execution performance.Source: §6 · p. 8

• The original Overcooked benchmark centers on onion soup, four preparation steps, and a 20-point reward; the extended testbed adds multiple ingredients and recipes.The new recipes use onion, tomato, and lettuce, with cooking time set to 15 seconds.Source: A ENVIRONMENT DETAILS · p. 11
• The extension adds chopping, overcooking and fires, a trash can, timed orders with rewards or penalties, and text or speech communication.Players can pause to chat or speak during gameplay; the action space includes communication texts and four movement actions.Source: A ENVIRONMENT DETAILS · p. 11
• Four maps vary layout and coordination demands; games last 100 seconds, with three valid orders at a time, or four on Quick.Quick also raises player speed to 3.5 Hz; the other default action frequency is 2.5 Hz.Source: A ENVIRONMENT DETAILS · p. 11

B.1.1 Prompt of Slow Mind in Intention Reasoning Stage.

• Slow Mind classifies human messages into six categories; questions pass directly to Chat & Assessment Stage for a response.The prompt marks mutable components such as soup orders.Source: §B.1.1 · p. 11

B.1.2 Prompt of Slow Mind in Chat & Assessment Stage.

• Slow Mind’s output depends on whether the human intention is satisfied: it may provide reasoning, completion assessment, and a chat message, or only a response.Source: §B.1.2 · p. 11

B.1.3 Fast Mind.

• Fast Mind uses conditional prompts based on whether an intention is inferred and satisfied, taking chat, inferred intention, or Slow Mind’s generated message as input.Source: §B.1.2 · p. 11

B.2 Script Policy

• The Executor implements seven scripted macro-action types; their utility depends on context such as map items, orders, and readiness.Actions are flagged available when immediately executable; a mid-action failure prompts a new macro action.Source: §B.2 · pp. 11–12
• For a macro action such as Chop Tomato, the script uses breadth-first search to pathfind and tracks the environment while translating the task into atomic movements.The policy is designed to minimize interference with the human player’s actions.Source: §B.2 · p. 12

C.1 Slow-Mind-Only Agent

• SMOA removes Fast Mind and has Slow Mind generate macro actions directly, regenerating when an action is illegal and omitting the Action Filter.GPT-3.5 lacks an API for evaluating the action probabilities required by HLA’s filter.Source: §C.1 · p. 12

C.2 Fast-Mind-Only Agent

• FMOA removes Slow Mind, so Fast Mind receives orders, items, and command history, and generates both macro actions and chat messages.It assumes intention satisfaction when its action history reaches nine actions and uses HLA’s adjustment value when intention is unsatisfied.Source: §C.2 · p. 12

C.3 No-Executor Agent

• NEA removes Executor, requiring Fast Mind to generate atomic actions directly and receive player and item positions as spatial input.The baseline workflows show Slow Mind generating macro actions in SMOA and Fast Mind generating chat directly in FMOA.Source: §C.3 · p. 12, Figure 1 · p. 13, Figure 2 · p. 13

D.1 Command Set for Latency Test

• The latency test records response times for 15 commands issued sequentially at 20-second intervals, without evaluating agents’ actual actions.The command set ranges from open-ended instructions to ingredient, order, and information requests.Source: §D.1 · p. 14
• Macro-action latency runs from receiving a human command to generating a macro action; atomic-action latency differs because NEA generates each atomic action directly.For SMOA, FMOA, and HLA, atomic-action latency is averaged as macro-generation latency divided by the number of atomic actions in that macro action.Source: §D.1 · p. 14

D.2 Complex Command Set

• The 30 complex commands cover quantity specification, semantic analysis, and ambiguous references, including follow-up commands such as “Cook it again.”The tests use Map Quick.Source: §D.2 · p. 14
• Tests fixed the soup orders and kept chopping and cooking targets outside active orders, except for the final four ambiguous-reference commands; experiments used an A100-80G GPU.Source: §D.2 · p. 14, E ADDITIONAL RESULTS · p. 14

E.1 Latency of Each Module

• SMOA and FMOA have high latency when generating macro actions and chat messages together; the authors say this supports HLA’s decoupling of those components.The module measurements cover Intention Reasoning, Chat & Assessment, and Macro Action Generation.Source: §E.1 · p. 14

E.2 Visualization of Simple Commands

• In simple-command tests, NEA loops left and SMOA misses plating; FMOA gives mismatched chat, whereas HLA uses all three pots and aligns its chat with subsequent action.FMOA describes a burning pot when none is burning and refers to an already-served Bob Soup; HLA’s Cathy Soup message precedes plating.Source: §E.2 · p. 15

E.3 Introducing Action Filter into SMOA

• SMOA+action filter increased macro-action latency by about 50% and reduced simple- and complex-command performance, partly because GPT-3.5-generated action probabilities were less accurate than unavailable logprobs.Generating actions and probabilities together also extended GPT-3.5’s output and response time.Source: §E.3 · p. 15

E.4 Comparison of Different LLMs

• HLA-Fast7B reduced macro-action latency 29% and atomic-action latency 25% versus HLA but weakened on One Command and semantic commands; HLA-Slow70B had similar latency yet struggled.The authors selected Llama2-13B-chat for Fast Mind and GPT-3.5 for Slow Mind as a balance of speed and reasoning performance.Source: §E.4 · p. 15

E.5 Interpreting Complex Commands

• 10% of semantically complex commands and 30% of ambiguous commands were not successfully interpreted.Source: §E.5 · p. 15
• Tab. 7 attributes all observed failures to Slow Mind misidentifying intentions, including interpreting a semantic command as “None” and confusing the intended soup in an ambiguous command.Source: §E.5 · p. 15
• Even with correct intentions and task completion supplied, Fast Mind had a 33% failure rate; unnecessary actions could waste time and prevent finishing before the limit.For one ambiguous command, Fast Mind took unnecessary actions before cooking the second Cathy Soup.Source: §E.5 · pp. 15–16

E.6 Human Studies

• The following human-studies subsections report module and end-to-end latency, Fast Mind’s macro-action hit rate, human preference, and visualized replay.Source: §E.6 · p. 17

E.6.1 Details on Behavior Analysis. 𝑁𝑣𝑎𝑙𝑢𝑎𝑏𝑙𝑒

• Behavior analysis counts valuable specified macro actions and rounds with pot fires, with Chop, Cook, and Serve usefulness tied to order and ingredient conditions.Partition provides no AI access to the delivery point, so its Serve result is reported as “/”.Source: §E.6.1 · p. 17

E.6.3 End-to-end Latency.

• Tab. 13 reports module latencies across AI players during human-study preparation, and simultaneous macro-action and chat generation adds significant latency to SMOA and FMOA.IR, CA, and MA denote Intention Reasoning, Chat & Assessment, and Fast Mind Macro Action Generation.Source: §E.6.2 · p. 17

E.6.2 Latency of Each Module.

• End-to-end latency is reported for preparation and competition phases; varying participant network conditions increase chat-generation latency for HLA and SMOA.The supplied passage gives no latency values.Source: §E.6.3 · p. 17

E.6.4 Hit Rate of Immediate Macro Action in the Fast Mind.

• Fast Mind’s immediate macro actions are notably faster and match the final macro actions based on human intention most of the time.The reported hit rate measures the proportion of immediate actions consistent with final actions.Source: §E.6.4 · p. 18

E.6.5 Human Preference.

• HLA remains most preferred across all human-rating aspects, while FMOA is preferred over SMOA in every aspect.The aspects include AI effectiveness, assistance, responsiveness, communication accuracy and action consistency, and overall performance.Source: §E.6.5 · p. 18

E.6.6 Replay

• The replay presents four human players on different maps: one remains silent, two text instructions, and one speaks to the AI player.Screenshots show gameplay at one-third, two-thirds, and completion, with scores and chat messages visible.Source: §E.6.6 · p. 18
• HLA achieves the highest game scores; its lower latency enables more chopping and counter placement in Bottleneck and Partition, and greater simultaneous pot use in Ring and Partition.The replay describes these activity differences relative to other AI players.Source: §E.6.6 · p. 19
• Speaking makes the game pace faster, while texting gives the human player more time to think.These communication differences are described alongside the replay examples.Source: §E.6.6 · p. 19

E.6.6 Replay.

• Competition-phase replays show player-128 on Ring, player-322 on Bottleneck, player-221 on Partition, and player-421 on Quick.Each visualization includes screenshots at 1/3 and 2/3 of gameplay and upon completion.Source: Figure 17 · p. 32, Figure 18 · p. 33, Figure 19 · p. 34