PaperlayerView exact revision on arXiv

Source-linked AI summary

ST-BiBench: Benchmarking Multi-Stream Multimodal Coordination in Bimanual Embodied Tasks for MLLMs

Xin Wu, Zhixuan Liang, Yue Ma, Mengkang Hu, Zhiyuan Qin, Xiu Li

arXiv:2602.08392v2cs.ROcs.AIcs.CV

TL;DR

Bimanual embodied tasks require MLLMs to coordinate multiple perception and action streams, but a persistent gap remains between strategic reasoning and grounded execution. ST-BiBench evaluates this gap through strategic planning, spatial grounding, and fine-grained action control, finding pervasive grounding and synchronization failures despite strong high-level logic.

  • Problem

    Synchronized bimanual manipulation requires spatial grounding and temporal coordination across multiple streams, while prior MLLM capabilities leave consistent continuous-action grounding unresolved.

  • Method

    ST-BiBench evaluates zero-shot MLLM coordination through strategic planning, workspace-aware spatial grounding, and 16-Dim fine-grained action control.

  • Results

    Across over 30 MLLMs, frontier models show promising strategic logic but pervasive spatial hallucinations, strategy-actuation gaps, and multi-stream fusion failures.

  • Takeaways & Limitations

    The benchmark identifies cross-modal alignment and multi-stream fusion as critical bottlenecks in dual-arm manipulation.

  • Takeaways & Limitations

    ST-BiBench is simulation-based, confined to rigid-body tasks, and affected by high inference latency for real-time coordination.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have significantly advanced the landscape of embodied AI, yet transitioning to synchronized bimanual coordination introduces formidable challenges in multi-stream multimodal integration. We introduce ST-BiBench, a comprehensive multi-tier framework for evaluating spatio-temporal multimodal coordination. Our approach centers on Strategic Coordination Planning, assessing high-level cross-modal reasoning over multiple action and perception streams. To investigate the "proximity paradox"-where semantically coherent plans fail to align with spatially grounded visual inputs-we incorporate Foundational Spatial Grounding to verify workspace awareness and arm-selection logic. Furthermore, we probe model frontiers through Fine-Grained Action Control, investigating whether MLLMs can directly synthesize high-dimensional continuous action modalities (16-Dim) from complex multimodal metadata. Evaluating 30+ state-of-the-art MLLMs, we uncover a persistent and pervasive "coordination paradox"-a significant gap between high-level strategic reasoning and fine-grained physical execution. Results reveal that while frontier MLLMs excel at logic-driven strategy, they frequently suffer from perception-logic disconnection and multi-stream interference during multimodal fusion. ST-BiBench provides a platform for identifying critical bottlenecks in multi-stream multimodal fusion and cross-modal alignment for complex embodied tasks.

1. Introduction

ST-BiBench evaluates MLLMs for synchronized bimanual coordination across planning, spatial grounding, and fine-grained action control. Across more than 30 models, it exposes persistent disconnections between strategic reasoning, physical grounding, and continuous action synthesis.

  • Bimanual manipulation requires synchronized reasoning across parallel perception-action channels and enables interactions unavailable to single-arm systems.
  • ST-BiBench spans Strategic Coordination Planning, Foundational Spatial Grounding, and Fine-Grained Action Control.
  • Strategic Coordination Planning: The framework evaluates high-level cross-modal alignment by decomposing long-horizon instructions into atomic action primitives across multiple streams.
  • Foundational Spatial Grounding: The proximity paradox occurs when logically sound plans fail to align with spatially grounded inputs, including incorrect arm assignments within rigid workspaces.
  • Fine-Grained Action Control: Fine-Grained Action Control tests whether MLLMs can synthesize 16-Dim continuous actions from context-augmented multimodal metadata.
  • Evaluating over 30 state-of-the-art MLLMs reveals perception-logic disconnection, strategy-actuation disparity, and inefficient multi-stream fusion.

2. Related Work

Prior embodied benchmarks largely emphasize symbolic reasoning, perception-to-action pipelines, or trained dual-arm policies rather than zero-shot multi-stream coordination. ST-BiBench instead examines how general-purpose MLLMs align reasoning and execution in bimanual manipulation.

  • Earlier benchmarks such as ALFRED and ALFWorld focus on high-level symbolic reasoning while abstracting away physical execution.
  • VLABench and EmbodiedBench evaluate perception-to-action pipelines but mainly use single-stream or sequential interaction settings.
  • TWIN and RoboTwin provide high-fidelity dual-arm simulations primarily for reinforcement-learning policy training.
  • ST-BiBench evaluates the zero-shot capacity of off-the-shelf MLLMs rather than training task-specific policies.
  • Bimanual systems face spatial hallucinations, inconsistent role assignment, and cascading errors when perception, reasoning, and action streams misalign.

3. Problem Formulation

The problem formulation distinguishes goal success from spatial grounding quality in bimanual manipulation. It formalizes coordination failures and uses workspace-aware scoring to evaluate arm-selection decisions under kinematic constraints.

  • 3. Problem Formulation: The Coordination Paradox is defined as systemic disconnection between semantically coherent logic and spatially grounded execution.
  • 3. Problem Formulation: The Proximity Paradox describes failures in spatial-semantic mapping when models misjudge kinematic feasibility despite identifying the correct target.
  • 3.1. Success Rate for Manipulation Tasks: Strategic Planning and Fine-Grained Action Control are evaluated with Success Rate based on programmatic verification of terminal goal conditions.
  • 3.2. Gaussian-Weighted Spatial Score for Grounding Tasks: Spatial Grounding evaluates whether the model selects the left or right arm using visual cues, proximity, and kinematic reachability.
  • 3.2. Gaussian-Weighted Spatial Score for Grounding Tasks: The proximal arm is defined as optimal because distal cross-centerline reaching can approach singularities, reduce precision, and increase collision risk.
  • 3.2. Gaussian-Weighted Spatial Score for Grounding Tasks: The Gaussian-Weighted Spatial Score computes each target’s score from its distance to the workspace centerline.
  • 3.2. Gaussian-Weighted Spatial Score for Grounding Tasks: The metric gives boundary errors higher partial credit while penalizing distant incorrect-arm selections more heavily.

4. Benchmark Construction

ST-BiBench evaluates the coordination chain from high-level planning through spatial grounding to continuous action control without task-specific fine-tuning. Its construction combines multi-stream task modes, synchronized multimodal inputs, reasoning-driven planning, and adaptive execution feedback.

  • ST-BiBench probes spatio-temporal coordination across granularities and tests coordinated behavior without task-specific fine-tuning.
  • Strategic Coordination Planning: Strategic Coordination Planning evaluates temporal reasoning and decomposition through parameterized atomic primitives such as Grasp, Place, and Move.
  • Foundational Spatial Grounding: Foundational Spatial Grounding tests whether visual spatial perception is aligned with discrete arm-selection decisions under physical constraints.
  • Fine-Grained Action Control: Fine-Grained Action Control demands zero-shot synthesis of 16-Dim poses and gripper states from visual inputs and context-augmented metadata.
  • Task Modes: Tasks are categorized as independent parallel, sequential collaborative, or synchronous collaborative manipulation according to required spatio-temporal synchronization.
  • Agent Architecture: The agent architecture integrates multi-perspective visual streams, numerical metadata, language, and interaction history through synchronized multimodal processing.
  • Agent Architecture: A Describe-Reason-Plan pipeline and multi-step action chunking align high-level thinking with low-level action streams while maintaining temporal consistency.
  • Dynamic Recalibration: Task-Adaptive Execution Truncation executes only the first k actions before re-observing, balancing long-horizon consistency with dynamic perceptual grounding.

5. Experiments

Across three evaluation tiers, ST-BiBench exposes a persistent gap between strategic reasoning, spatial grounding, and precise bimanual actuation. Performance depends on model capacity, architecture, and robustness to visual complexity, with perception–logic disconnection and multi-stream interference recurring as central failure patterns.

  • Model Scaling: Scaling is non-monotonic: mid-sized models can outperform 70B+ or 200B+ models, while larger models may suffer more multi-stream interference.The reported efficiency peak attributes embodied-task performance more to architectural efficiency and instruction-following alignment than raw scale.
  • Model Scaling: A reasoning threshold separates 7B models from the 30B+ tier, yet even 30B+ models retain an alignment gap in strict cross-arm temporal synchronization.Qwen2.5-VL-7B nearly collapses on collaborative tasks when a second action stream and temporal dependencies are introduced.
  • Foundational Spatial Grounding: Closed-source models dominate spatial grounding, while mid-sized open-source models can outperform larger counterparts when vision-language alignment is stronger.Gemini-2.5-Pro and Gemini-2.0-Flash exceed 90% accuracy; Ovis2-34B reaches 92.67% and Qwen3-VL-32B reaches 94.00%.
  • Strategic Coordination Planning: Incorrect arm assignment remains a bottleneck, with InternVL3-78B dropping from 97.07% to 90.16% under clutter.The result indicates that logically sound plans can still misalign spatial feasibility when multimodal inputs become complex.
  • Fine-Grained Action Control: GPT-5 leads fine-grained action control at 66.80%, while Gemini-2.5-Pro reaches 60.20% and InternVL3-78B reaches 27.60%.The reported gap separates proprietary and open-source performance in precise continuous action synthesis.
  • Multi-Stream Fusion: High-capacity models benefit from auxiliary views in some tasks, but less robust models often experience information overload and performance degradation.Gemini-2.5-Pro improves on Place9 from 39% to 55%, whereas Gemma3-27B drops on Place8 from 13% to 4%.

6. Conclusion and Limitations

ST-BiBench evaluates coordination planning, spatial grounding, and action control, revealing brittle reasoning, pervasive arm-proximity hallucinations, and persistent synchronization challenges. The benchmark is simulation-based and limited to rigid-body tasks.

  • ST-BiBench evaluates MLLMs across coordination planning, spatial grounding, and action control.
  • All evaluated models exhibit stochastic spatial hallucinations and frequently misjudge arm proximity, disrupting strategic plans and causing dual-arm execution failures.
  • Fine-Grained Action Control remains difficult because models struggle with multi-stream synchronization despite reasoning-supportive metadata.
  • Limitations: ST-BiBench is simulation-based and confined to rigid-body tasks, limiting its reflection of real-world sensory nuances.

A. Future Research Directions

Future work targets the gap between simulated rigid-body evaluation and robust real-world bimanual coordination. Proposed directions include sim-to-real robustness and broader multimodal, deformable-object settings.

  • Bridging the Multimodal Sim-to-Real Gap: Future research should bridge the multimodal sim-to-real gap through cross-domain robustness, sim-to-real adaptation, and domain randomization.
  • From Rigid Bodies to Multi-sensory Integration: The benchmark should expand beyond rigid bodies to deformable-object manipulation and incorporate tactile sensing alongside vision.
  • Addressing these directions could bridge high-level strategic planning and precise physical actuation in bimanual embodied agents.

B. Model Details

The evaluation covers proprietary and open-source MLLMs spanning multiple families and parameter scales. Model selection includes GPT, Gemini, Claude, Qwen-VL, InternVL, Llama, Gemma, Ovis, and GLM series.

  • The study selects diverse MLLMs ranging from state-of-the-art proprietary systems to open-source models across parameter scales.
  • Closed-Source Model Series: Proprietary evaluations include GPT-5, GPT-4.1, GPT-4o, Gemini-2.5-Pro, Gemini-2.5-flash, Gemini-2.0-flash, and Claude Sonnet variants.
  • Qwen-VL Series: The Qwen-VL evaluation spans Qwen3-VL and Qwen2.5-VL generations, including large, medium, lightweight, and mixture-of-experts models.
  • InternVL Series: InternVL coverage includes InternVL3, InternVL3.5, and InternVL2.5 variants at 78B, 38B, and 8B scales.
  • Llama, Gemma, Ovis and GLM Series: Additional open-weight models include Llama, Gemma, Ovis, and GLM families.

C. Task Details

The task details section presents high-, average-, and low-quality reasoning examples. These examples respectively illustrate strong reasoning, spatial ambiguity, and significant visual hallucination.

  • Figure 1 presents a high-quality reasoning example.
  • Figure 2 presents an average-quality reasoning example involving spatial ambiguity.
  • Figure 3 presents a low-quality reasoning example with significant visual hallucination.

C.2. Strategic Coordination Planning

Strategic Coordination Planning evaluates whether MLLMs can decompose manipulation instructions into executable, temporally consistent action sequences while assigning arms appropriately. An interface guardrail checks reachability and truncates clearly invalid assignments so planning logic can be assessed separately from basic spatial errors.

  • Planning formulation: Strategic planning converts task instructions into ordered primitives such as grasp, place, and move actions.The planner must maintain temporal consistency and cross-modal role assignment across complex dependencies.
  • Execution interface: The allocation mechanism validates each predicted arm tag against kinematic constraints and object reachability before execution.Invalid actions are truncated and returned to the model with textual feedback rather than immediately ending the episode.
  • Evaluation design: The guardrail is intended to separate high-level planning assessment from failures caused solely by one-time spatial arm-selection errors.This enables evaluation of logical dependencies in tasks such as handover and placing bread in a skillet.
  • Scope and caveat: The mechanism does not catch nuanced coordination errors or subtle workspace overlaps, so some end-effector allocation errors remain in the final analysis.Its coverage is primarily limited to clear kinematic violations and reachability failures.
  • Action primitives: The benchmark includes manipulation skills for grasping, placing, relative and absolute motion, gripper control, and home reset.These skills expose arm tags, poses, offsets, orientations, and gripper states as action parameters.
  • Illustrative example: A handover example sequences left-arm grasping, lifting, transfer to a middle pose, right-arm takeover, and final placement on a blue pad.The plan keeps the gripper closed during transfer and returns the inactive arm to origin before final placement.

C.3. Fine-Grained Action Control

Fine-Grained Action Control tests whether MLLMs can directly produce continuous dual-arm actions from visual and numerical context. The stacking example illustrates detailed pose, height, orientation, and collision-avoidance planning, while the benchmark frames this tier as a demanding zero-shot fusion stress test.

  • Task set: The benchmark contains five Fine-Grained Action Control manipulation tasks.These tasks include stacking two blocks and placing a burger and fries on a tray.
  • Example: The successful GPT-5 example is presented as a Fine-Grained Action Control demonstration for stacking two blocks.The figure caption identifies the model, tier, and task.
  • Stacking example: A stacking plan assigns the left arm to the green block and the right arm to the red block, then places them at center with distinct target heights.It specifies bottom and top gripper heights of approximately 0.88649 and 0.93649 after accounting for a 0.162 m offset.
  • Failure example: A failed InternVL3-78B example concerns placing a burger and fries with dual-arm manipulation.The task requires picking both food items and placing them on a tray.

D. Error Analysis

The error analysis separates failures into environmental errors and model output errors.

  • Errors are divided into environmental errors and model output errors.

D.1. Environmental Errors

Environmental errors arise from simulation-interface behavior and scene randomization rather than the decision-making model alone. These problems can prevent task completion and may affect models similarly within the same task.

  • Interface errors: Simulation-interface errors occur when robotic behavior does not match API documentation, independently of the decision-making model within a task.Their probability varies across tasks because tasks call different environment interfaces.
  • Grasp failures: A grasp can fail after collision changes a symmetrical object’s orientation because the simulator’s grasp function implicitly depends on orientation.
  • Scene randomization: Scene randomization can place interfering objects in inappropriate locations, preventing task completion for all models using the same episode seed.

D.2. Model Output Errors

The benchmark categorizes model output failures into perceptual reasoning, strategic planning, and format errors, while using task-specific visual inputs and structured prompts to assess dual-arm decisions. The analysis focuses on perceptual and strategic failures because format errors mainly occur in smaller models.

  • Error taxonomy: Model output errors comprise Perceptual Reasoning Error, Strategic Planning Error, and Format Error.The analysis focuses on the first two categories because Format Error usually occurs in models with fewer than 10B parameters.
  • Perceptual reasoning: Perceptual Reasoning Error reflects failure to infer the current task state from visual observations, including state-estimation misjudgment and physical-attribute misreasoning.Physical-attribute misreasoning mainly appears in tasks requiring block-size determination.
  • Perceptual reasoning: Incorrect end-effector allocation can remain among Strategic Coordination Planning errors because interface truncation does not capture every such case.The interface usually truncates infeasible allocation cases and gives feedback until the model selects the correct end-effector.
  • Strategic planning: Strategic Planning Error includes incorrect action sequencing and bimanual conflict or collision during execution.Examples include failing to raise the end effector before placement or return, and failing to account for conflicts between the two arms.
  • Observation settings: Strategic Coordination Planning receives both ego-centric and third-person images, whereas Fine-Grained Action Control primarily uses the ego-centric view.The third-person view provides global context and is used to mitigate occlusion in Strategic Coordination Planning and selected Fine-Grained Action Control tasks.
  • Prompt structure: The dual-arm spatial-reasoning prompt requires visual-state description, object identification, and left-or-right arm assignment in JSON format.The task materials also specify collision avoidance, returning an unused arm to its origin, and ordered block placement procedures.
Loading 2602.08392v2…