Source-linked AI summary
DuoCook: An Overcooked-Inspired Benchmark for Visual Planning and Coordination
Ziyi Jiao, Heng Zhang, Drishti Regmi, Xiaokang Ye, Lianhui Qin, Peter Yichen Chen
TL;DR
DuoCook addresses whether a vision-language model can coordinate interdependent cooking work under deadlines. It evaluates a single model controlling two players across varied Unreal Engine kitchens, with gpt-6-astra achieving the highest mean order completion and notebook memory showing mixed score effects.
Problem
Coordinating cooking requires tracking changing scene states, sequencing recipe steps, and allocating shared resources under time limits; the paper asks whether a VLM can coordinate this interdependent work from visual and structured information.
Source: §1 · p. 1Method
DuoCook evaluates one VLM controlling two players across 20 Unreal Engine maps and compares four models using performance, action-validity, and API-cost measures, including a notebook-memory ablation.
Source: §1 · pp. 1–2Results
gpt-6-astra achieves the highest mean order completion (26.2%), while notebook memory has mixed score effects, with observed mean gains for Claude and Qwen and declines for GPT and Gemini.
Source: §6 · p. 10Takeaways & Limitations
The 20-map suite provides a concrete way to investigate how controllers turn visual observations into coordinated cooking actions, with conclusions applying to this evaluated suite.
Source: §6 · p. 10Centralized control, top-down observations, engine navigation, and structured assistance limit the evaluation to planning through this combined interface rather than independent teammates or direct motor control.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Coordinating two players in a cooking task requires a vision-language agent to track changing scene states, sequence recipe steps, and allocate shared resources under a time budget. We introduce an Overcooked-inspired benchmark for evaluating these capabilities in Unreal Engine environments constructed using coding agents. A single vision-language model controls both players from top-down visual observations, selecting high-level actions while the engine executes navigation. The benchmark contains 20 maps in five categories. These categories support evaluation of centralized coordination under equipment contention, changing terrain, and handoff dependencies. A prop legend and pot ingredient-count indicators provide explicit visual cues, while grid-based destinations support spatial grounding. The evaluation compares four VLMs on all 20 maps, measuring task score, order completion, delivery precision, action acceptance rate, and model API expenditure. A memory ablation on the same 20 maps examines the effect of a shared, self-authored notebook while retaining six-response action history in both conditions. The strongest evaluated model, \texttt{gpt-6-astra}, achieves 26.2\% mean order completion. Notebook memory has mixed score effects across models, with positive mean differences for Claude and Qwen and negative differences for GPT and Gemini.
1 INTRODUCTION
DuoCook asks whether a vision-language model can coordinate interdependent cooking work from visual and structured information under order deadlines. It evaluates centralized two-player planning across varied kitchens, using a shared multimodal interface and performance, action-validity, and API-cost measures.
1 INTRODUCTION
- DuoCook evaluates a single VLM controlling two players across 20 Unreal Engine maps in five categories that introduce equipment contention, terrain changes, and required handoffs.The target is centralized coordination: assigning both players’ work to maximize delivery score under deadlines, time-dependent tips, and a streak bonus.Source: §1 · p. 1
- The controller selects actions and destination coordinates from kitchen images and structured task information, while the engine handles navigation.The introduction frames the task around tracking unfinished operations, assigning complementary work, and revising schedules as equipment and ingredients become available.Source: §1 · pp. 1–2
- The benchmark compares four VLMs on all 20 maps using total score, order completion, delivery precision, action acceptance, and API expenditure per episode.A same-map memory ablation removes only the shared, self-authored notebook while retaining other inputs.Source: §1 · p. 2
2 RELATED WORK
DuoCook connects cooking coordination, visual interaction, persistent context, and generated environments to evaluate centralized scheduling across varied kitchens. Its scope differs from work on independently controlled partners and benchmarks with different observation and control interfaces.
2.1 COOKING ENVIRONMENTS AND COOPERATIVE AGENTS
- Prior cooking-agent work studies human cooperation, teammate-intention inference, centralized scheduling, and interdependent collaboration; DuoCook instead evaluates one visually grounded controller scheduling both players.Its map variations expose resource and spatial dependencies, rather than inferring the intentions of an independently controlled partner.Source: §2.1 · p. 2
2.2 VISUAL INTERACTION AND GAME BENCHMARKS
- Visual-agent benchmarks cover games, embodied environments, and graphical interfaces, whereas DuoCook focuses on matched interventions within cooking workflows.Its engine-executed navigation, grid labels, and structured state assist the controller, so scores are not directly interchangeable with benchmarks using different interfaces.Source: §2.2 · p. 2
2.3 PERSISTENT CONTEXT FOR INTERACTIVE AGENTS
- Unlike action-history or cross-trial reflection approaches, DuoCook tests a shared notebook of self-authored observations with age-dependent retention.Removing the notebook while retaining six-response action history and current visual and structured inputs tests its effect on performance and inference cost.Source: §2.3 · p. 2
2.4 GENERATED ENVIRONMENTS FOR EVALUATION
- DuoCook uses coding agents to build Unreal Engine kitchens from reference layouts and instructions, followed by manual refinement and actor-contract verification.This construction process supplies environments for evaluating cooking controllers.Source: §2.4 · p. 3
3 SYSTEM AND TASK DESIGN
DuoCook combines agent-generated Unreal Engine kitchens with a centralized cooking controller and multimodal inputs to study coordination across map conditions. The design uses grid-based destinations and engine-managed navigation while evaluating score-based order completion.
3.1 MAP GENERATION WITH CODING AGENTS
- Coding agents construct kitchens in Unreal Engine from prompts and scene contracts; manual refinement and an actor-verification script follow construction.Manual edits address decoration, collisions, visibility, and camera configuration.Source: §3.1 · p. 3
3.2 MAP CONDITIONS
- The benchmark’s 20 maps vary equipment sharing, terrain changes, and item transfers, with separated workspaces requiring players to pass ingredients or plates across counters.Shared-equipment maps require station sharing, while dynamic maps change obstacles or traversable terrain every 1,000 steps.Source: §3.2 · pp. 3–4
3.3 COOKING TASK AND CONTROLLER OBJECTIVE
- A single VLM controls two players to prepare and serve orders; chopping and dish washing occupy a player, while soup boils unattended and permits concurrent work.Orders arrive every 1,000 steps, expire after 3,000, and may be fulfilled in any order; deliveries earn 20 points plus a waiting-time tip scaled by a combo multiplier.Source: §3.3 · p. 4
3.4 MULTIMODAL AGENT INPUTS
- Both memory conditions provide a kitchen image, map-specific rules and recipes, live-state information, legal operations, execution feedback, and six-response action history.The memory-enabled condition additionally supplies a shared, self-authored notebook.Source: §3.4 · p. 4, Figure 2 · p. 5
3.5 GRID-BASED NAVIGATION AND HIGH-LEVEL ACTIONS
- Grid coordinates ground destination selection, while the engine uses a navigation mesh to approach stations and returns errors for incompatible actions or unreachable destinations.State changes, including terrain changes, trigger model calls; the simulation pauses during inference, and terrain changes prompt destination reselection.Source: §3.5 · pp. 4–5
4 EXPERIMENTS
The experiments compare four VLMs across 20 maps and five categories, then test shared notebook memory on those same maps. The strongest model completes 26.2% of orders, while memory’s score effects vary across models.
4.1 EVALUATION SETUP
- The evaluation runs one episode per model and map, averaging metrics equally across the 20 maps; results therefore do not estimate variability across repeated model runs.The five metrics cover score, order completion, delivery precision, action acceptance, and API expenditure.Source: §4.1 · pp. 5–6, §4.3 · p. 6
4.3 MAIN BENCHMARK EVALUATION
- 26.2% of orders are completed by the strongest controller; the other models complete 8.8–11.2%, and gpt-6-astra leads on score, order completion, and action acceptance.Most episodes end with nothing delivered; Table 2 reports overall performance averaged equally across 20 maps, with one episode per model and map.Source: §4.3 · pp. 6–7, Table 2 · p. 7
- Terrain changes combined with shared equipment and handoffs across separated workspaces are the strongest barriers, with gpt-6-astra the main exception.Three of four controllers deliver nothing on terrain-change/shared-equipment maps; Claude and Gemini deliver nothing across their eight separated-workspace episodes.Source: §4.3 · p. 7
- 100% delivery precision for Claude, 93.8% for gpt-6-astra, 91.7% for Qwen, and 81.0% for Gemini apply only to maps with at least one delivery.Precision uses different map subsets, and action acceptance can rank models differently from task score; cost spans more than an order of magnitude and does not track performance.Source: §4.3 · p. 7
4.4 MEMORY ABLATION
- The notebook stores player-authored observations of grid cell, object kind, status, and optional notes; both players read it, and entries decay with age.Partner-authored hearsay decays twice as fast as first-hand observation, and the store is not validated against simulator state.Source: §4.4 · p. 8
- Shared memory has mixed score effects: Claude gains +4.8 points, the only model whose reported bootstrap interval excludes zero, while Gemini has the largest mean decrease at −8.1 points.The intervals resample maps and do not quantify repeatability across model runs; memory increases mean API expenditure for all four models.Source: §4.4 · p. 9
5 LIMITATIONS
The evaluation is bounded by its centralized, assisted interface, single-run comparisons, and generated-map scope; missing artifacts further constrain failure analysis and reproducibility.
5 LIMITATIONS
- The interface evaluates centralized planning with top-down observations, engine navigation, and structured assistance, not independent-teammate communication or direct motor control.The legend, pot indicators, live-state JSON, and legal-operation list also assist action selection.Source: §5 · p. 9
- The memory ablation adds a shared notebook while retaining action history, but memory can also change token usage and latency.Map-condition differences also do not isolate resource sharing, and the visual legend and pot indicators remain fixed.Source: §5 · pp. 9–10
- One run per model, map, and condition cannot quantify response variability or establish stable model rankings and memory effects.Fixed initial states and order sequences match scenarios but do not replace repeated runs.Source: §5 · p. 10
- Generated maps may repeat structural patterns, and new maps do not ensure recipes or task concepts are absent from pretraining.Generalization beyond the tested kitchen distribution, including transfer to real collaborative work, needs additional evidence.Source: §5 · p. 10
- Episode logs, per-map metric tables, and analysis scripts are absent, preventing trace-based failure diagnosis and independent recomputation from the artifacts.Source: §5 · p. 10
6 CONCLUSION AND FUTURE WORK
DuoCook provides a 20-map benchmark and memory ablation for studying centralized visual coordination in agent-generated kitchens. In one run per model, gpt-6-astra leads on mean score, order completion, and action acceptance, while memory has mixed score effects.
6 CONCLUSION AND FUTURE WORK
- The Overcooked-inspired benchmark evaluates visual planning and centralized coordination in agent-generated Unreal Engine kitchens, with 20 maps spanning five categories that cover independent work, shared equipment, changing terrain, and required item transfers.The protocol compares four VLMs on all 20 maps using task performance, action acceptance, and API cost, and includes a memory ablation on those same maps.Source: §6 · p. 10
- gpt-6-astra achieves the highest mean score (25.6), order completion (26.2%), and action acceptance (64.6); Claude has the highest delivery precision among episodes with deliveries.Performance remains low, especially on dynamic maps with shared equipment and separated-workspace maps; repeated runs are needed to assess stability.Source: §6 · p. 10
- The authors used claude-opus-5 to author scene-generation prompts and construct benchmark maps, and used AI tools to polish the manuscript.Source: AI USE STATEMENT · p. 10
A MAP CORRESPONDENCE
The appendix maps source levels to the evaluation subsets and clarifies that both experiments use the complete 20-map set.
A MAP CORRESPONDENCE
- Both the main benchmark and memory ablation use all 20 functional maps.Table 3 lists the source-level correspondence and evaluation membership for Main and Memory.Source: A MAP CORRESPONDENCE · p. 12, Table 3 · p. 12
- Evaluation identifiers replace source-level hyphens with underscores; Festive 1-6 and 1-7 and Kevin 1 have corresponding source-level names.Source: A MAP CORRESPONDENCE · p. 12
B SAMPLE SCENE GENERATION PROMPTS
The sample scene-generation prompts illustrate how map layouts encode coordination constraints, including cross-map handoffs, separated workspaces, and terrain bottlenecks.
B SAMPLE SCENE GENERATION PROMPTS
- The truck layout separates ingredients and preparation from cooking and serving, forcing dishes to cross two gangplanks at least twice.The design places all ingredients on the blue truck and all three stoves on the red truck.Source: B SAMPLE SCENE GENERATION PROMPTS · p. 13
- The space-station prompt keeps side modules docked and makes the east hatch nonfunctional, concentrating delivery and plate return in the west module.The north and south ring pods are decorative and unreachable.Source: B SAMPLE SCENE GENERATION PROMPTS · pp. 14–15
- The Frozen Brook Lodge divides the kitchen across a blocked brook, with movement limited to a bridge and an east-end strip.The halves mirror each other and connect through those crossings.Source: B SAMPLE SCENE GENERATION PROMPTS · pp. 15–16