Source-linked AI summary
MAGMA-Bench: Evaluating Long-Horizon Embodied Agents under Dynamic Tasks and Execution Failures
Loan Bernat, Matthieu Grard, Ariane Herbulot, Florent Lamiraux
TL;DR
Long-horizon embodied agents must preserve task information and coordinate heterogeneous robots as instructions and execution change, but existing benchmarks provide limited evaluation of these demands together. MAGMA-Bench evaluates persistent task reasoning in dynamic multi-robot missions through a structured skill-level interface and controlled perturbations. Results show that task-specific supervision can improve a small planner, while substantial weaknesses remain in multi-robot assignment and task resumption.
Problem
Existing benchmarks provide limited evaluation of persistent task reasoning across long-horizon interactive multi-robot missions, where instructions and execution can change.
Source: §1 · pp. 1–2Method
Results
Task-specific supervision can substantially improve a small planner, but pronounced weaknesses remain in multi-robot assignment and task resumption.
Source: §7 · p. 8Takeaways & Limitations
The findings motivate evaluating persistent task reasoning through both task outcomes and trace-level diagnostics of evolving commitments.
Source: §7 · p. 8The benchmark does not measure visual recognition or low-level motor control, so its results do not directly characterize end-to-end VLM or VLA policies.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Long-horizon embodied agents must maintain task-relevant information, coordinate heterogeneous robots, and adapt their decisions as execution unfolds. Existing benchmarks investigate dynamic task execution and embodied multi-agent coordination, but provide limited evaluation of their intersection. We introduce MAGMA-Bench, an architecture-agnostic benchmark for persistent task reasoning in dynamic multi-robot environments. MAGMA-Bench evaluates agents across increasing task horizons and controlled online perturbations, including execution failures, task interruptions, and mission updates, with scenarios requiring coordination across robots with heterogeneous capabilities. It uses a structured skill-level interface to separate task-level reasoning from perception and low-level control. We evaluate locally deployable open-weight planners, multi-agent state-tracking strategies, and frontier models. Even the strongest evaluated models fail on a substantial fraction of clean episodes, and their success rates decrease further under online perturbations. Supervised adaptation greatly improves a small model, but performance remains substantially lower on longer and multi-robot missions. We release the benchmark, 261,646 task-level training examples, and an architecture-agnostic expert-trace generation pipeline.
1 Introduction
Long-horizon multi-robot missions require agents to preserve and revise task state as instructions, execution, and robot availability change. MAGMA-Bench evaluates this ability through skill-level decisions and controlled online perturbations.
1 Introduction
- Systematic evaluation of persistent task reasoning in long-horizon interactive multi-robot missions remains limited, while perception and control can obscure task-level failures.The required capability includes tracking evolving goals, constraints, assignments, and progress across events.Source: §1 · pp. 1–2
- Dynamic missions require agents to preserve unfinished work while revising goals and robot assignments after interruptions or execution failures.A suspended delivery may need to resume after another request takes priority; isolated planning or replanning decisions do not capture this requirement.Source: §1 · pp. 1–2
- MAGMA-Bench evaluates persistent task reasoning across 12 scenario families, one to three robots, changing horizons, online perturbations, and generalization settings.Its skill-level interface leaves policy architecture unspecified and supports comparison of planners, state-tracking strategies, adapted models, and selected frontier systems.Source: §1 · p. 2
- The benchmark combines long-horizon task execution with controlled perturbations and heterogeneous multi-robot coordination in an architecture-agnostic, skill-level setting.The released resources include 261,646 task-level training examples and an expert-trace generation pipeline projectable into architecture-specific formats.Source: §1 · p. 2
2 Related Work
Prior work separately measures persistent interactions and embodied planning, memory, or coordination. MAGMA-Bench brings these demands together in controlled, skill-level evaluations of long-horizon multi-robot missions.
Memory and dynamic task execution in language agents
- Digital agentic benchmarks study stateful tool use and memory, while Multi-Mission Tool Bench, InterruptBench, and Gaia2 [3] [4] [5] add mission changes or dynamic environments.ToolSandbox [12], τ-bench [13], MemoryAgentBench, and MemoryArena [2] also measure persistent information or stateful interactions.Source: §2 · p. 2
Long-horizon reasoning in embodied agents
- Embodied benchmarks cover multi-step manipulation and robotic memory, while PARTNR [6] exposes high-level skills for heterogeneous human–robot coordination.CoMuRoS evaluates event-driven replanning after failures or changing intentions, whereas MAGMA-Bench emphasizes completion across extended interactive episodes.Source: §2 · p. 2
Positioning of MAGMA-Bench
- MAGMA-Bench combines persistent user-intent tracking with physical constraints, heterogeneous robot capabilities, and execution recovery in controlled skill-level evaluations.Unlike evaluations centered on fixed tasks or isolated replanning, it tests coherent execution as goals, rules, and conditions evolve.Source: §2 · p. 2
- MAGMA-Bench separates task-level decisions from perception and low-level control to compare reasoning architectures under reproducible perturbations.Related benchmarks provide complementary coverage of evolving intentions, physical constraints, heterogeneous capabilities, and recovery.Source: §2 · p. 2
3 Persistent Task Reasoning
The paper formalizes persistent task reasoning as maintaining task-level information through asynchronous robot execution and changing instructions. Agents infer a latent task state from event-driven observations and choose their own representation of interaction history.
3 Persistent Task Reasoning
- An agent is the complete decision-making system that maps observations to robot skill calls or user-facing responses, with its internal implementation unrestricted.It may use one model or multiple software agents, symbolic planners, and memory mechanisms.Source: §3 · p. 2
3.1 Problem Formulation
- Event batches group user and environment events by arrival time, and each new batch triggers an event-driven decision step.Skills may continue asynchronously while later events arrive, so actions need not complete in dispatch order.Source: §3.1 · p. 3
3.2 Latent Task State
- Persistent task reasoning retains, revises, and applies latent task information across separated instructions and execution events while robot skills may still be running.The task state distinguishes physical environment state from semantic task state; the latter includes current goals and active interaction-derived rules.Source: §3.2 · p. 3
- Agents infer task-relevant information from observations, permanent rules, and robot skill interfaces rather than receiving either underlying state directly.Current goals can include active and suspended objectives, priorities, assignments, and completion status; user instructions may revise or override goals and interaction-derived rules.Source: §3.2 · p. 3
- Agents choose how to represent past interactions, including retaining full history, truncating it, summarizing it, maintaining explicit task state, or retrieving external memory.Full-history context and computation costs increase as interactions progress, without guaranteeing effective access to task-relevant information.Source: §3.2 · p. 3
4 Persistent Task Reasoning Benchmark
MAGMA-Bench evaluates persistent task reasoning across long-horizon robotic tasks, online perturbations, and generalization regimes. Its task-level setup covers robotic interaction, benchmark composition, released training data, and metrics for success, partial completion, and robustness.
4 Persistent Task Reasoning Benchmark
- The benchmark varies execution horizon, online perturbations, and generalization to evaluate persistent task reasoning in robotics.Horizons range from short to extreme; perturbations include mission updates, execution failures, and interruptions.Source: §4 · p. 3
4.1 Task-Level Evaluation Setting
- Policies act through a common skill API, selecting robot skills, responding in text, or waiting for execution events.Instructions include persistent rules, updates, underspecified requests, embodied questions, and interruptions.Source: §4.1 · p. 3
4.2 Robotic Interaction Model
- Perception may miss objects and physical skills may fail, changing the environment and requiring agents to reason across later decisions.A failed grasp leaves an object unmoved, while a missed object may become observable later.Source: §4.2 · p. 4
4.3 Benchmark Composition
- The benchmark spans 12 scenario families, one to three robots, five task conditions, and in-domain, compositional, and held-out generalization.Execution perturbations are scheduled consistently across policies, but occur only if an agent reaches the targeted stage and invokes a compatible skill.Source: §4.3 · pp. 4–5
4.4 Released Dataset
- The released architecture-agnostic dataset contains 261,646 training rows and 13,574 validation rows, with programmatic expert-trace targets.It excludes stochastic skill failures and partial observations, so recovery from these is not directly supervised.Source: §4.4 · p. 5
4.5 Metrics
- Metrics report complete-task success, per-episode goal completion, and conditional retention of clean success under paired perturbations.Each episode contributes equally to Goal Completion, and Conditional Robustness must be read together with Success Rate.Source: §4.5 · p. 5
5 Benchmark Validation and Empirical Characterization
The benchmark is evaluated for whether performance depends on retained task state, whether supervision enables learning while preserving headroom, and whether horizon, perturbations, and multi-robot coordination expose distinct challenges.
5.1 VQ1: Can Zero-Shot Models Maintain Task State?
- Oracle task-state descriptions outperform a single-model policy, while generated reflection and explicit-state variants underperform it; truncating history also lowers clean SR.For gpt-oss-20b, clean SR falls from 5.7% with full available history to 3.0% with K=32 and 1.0% with K=8.Source: §5.1 · p. 6
5.2 VQ2: Is MAGMA-Bench Learnable While Retaining Headroom?
- Fine-tuning improves SR and GC in every horizon bucket, but performance still declines as horizon grows.The table compares Qwen3-4B before and after fine-tuning across horizons.Source: §5.2 · p. 6, Table 3 · p. 6
- Qwen3-4B-FT achieves SRs of 38.19% in-domain, 33.33% on compositional tasks, and 24.00% on the held-out Packaging family.The reported transfer does not isolate interface-compliance gains from task-behavior gains.Source: §5.2 · p. 7
- Fine-tuned Qwen approaches frontier-model performance on short tasks, but the gap widens on longer missions; Gemini’s success rate declines with horizon while GPT’s curve is non-monotonic.Figure 2 compares zero-shot performance on the full benchmark with frontier models and the best model on a shared 215-episode mini-benchmark.Source: Figure 2 · p. 6, §5.2 · p. 7
5.3 VQ3: Do Horizon and Online Perturbations Expose Distinct Weaknesses?
- Perturbation sensitivity differs across models: Gemini retains more clean successes under stochastic execution than interruptions, while fine-tuned Qwen retains a substantial fraction under stochastic execution.Under interruptions, Qwen has higher conditional retention than frontier models, although its absolute success rate remains below GPT’s and conditioning sets differ.Source: §5.3 · p. 7
- Across four models, macro-averaged goal completion decreases by 1.5–30.1 percentage points between the shortest and longest clean horizon buckets.Scenarios are equally weighted; within-scenario trends are not uniformly monotonic.Source: §5.3 · p. 7
- Text-for-action and No progress / timeout together account for 61.2–64.7% of failures, consistent with premature completion, failure to resume work, or non-contributing actions.The outcome categories are assigned deterministically and supplemented by a trace audit.Source: §5.3 · p. 7, Figure 4 · p. 7
5.4 VQ4: Does MAGMA-Bench Expose Multi-Robot Coordination Failures?
- Qwen3-4B-FT’s clean SR falls from 67.7% on single-robot tasks to 11.4% on multi-robot tasks, while GPT-6.1-SOL remains near-constant at 73.5% versus 74.3%.Trace audits also find Qwen assignment errors on 2.04% of evaluable calls versus 0.09% for GPT-6.1-SOL; none of Qwen’s 12 evaluable resume obligations succeeds, compared with 12 of 21 for GPT-6.1-SOL.Source: §5.4 · p. 8
6 Limitations
MAGMA-Bench evaluates task-level decisions through structured observations and skill calls, not visual recognition or low-level motor control. The paper characterizes selected centralized language-model approaches rather than determining the best architecture.
6 Limitations
- MAGMA-Bench does not measure visual recognition or low-level motor control, so its results do not directly characterize end-to-end VLM or VLA policies.Its evaluation uses structured observations and skill calls.Source: §6 · p. 8
- The evaluated persistence concerns goals, rules, assignments, and execution progress rather than spatial scene understanding.Source: §6 · p. 8
- The paper focuses on centralized language-model policies; identifying the best architecture is outside its scope, despite interface support for symbolic planners and decentralized architectures.Source: §6 · p. 8
7 Conclusion
MAGMA-Bench evaluates whether agents maintain and revise task commitments during long, asynchronous executions. The results show that supervision can substantially improve a small planner, while multi-robot assignment and resumption remain weak points.
7 Conclusion
- Task-specific supervision substantially improves a small planner, but pronounced weaknesses remain in multi-robot assignment and task resumption.Source: §7 · p. 8
- Frontier models retain substantial headroom and show different sensitivities to mission updates, interruptions, and execution failures.Source: §7 · p. 8
- Strong single-robot task performance does not necessarily extend to concurrent commitments under online task revisions.The paper motivates evaluating persistent task reasoning through task outcomes and trace-level diagnostics of evolving commitments, and releases the benchmark, training data, and expert-trace generation pipeline.Source: §7 · p. 8