Source-linked AI summary
SleepWalk: A Three-Tier Benchmark for Stress-Testing Instruction-Guided Vision-Language Navigation
Niyati Rawal, Sushant Ravva, Shah Alam Abir, Saksham Jain, Aman Chadha, Vinija Jain, Suranjana Trivedy, Amitava Das
TL;DR
It remains unclear whether VLMs can reliably translate natural-language instructions into spatially grounded, executable behavior in 3D environments. SleepWalk introduces a controlled benchmark for explicit trajectory prediction and finds that current VLMs struggle with precise spatial grounding, especially for occluded and multi-step tasks.
Problem
It remains unclear whether VLMs can reliably translate natural-language instructions into spatially grounded, executable behavior in 3D environments.
Method
SleepWalk evaluates whether VLMs map language and visual context into explicit trajectories in controlled single-scene 3D environments, using structured judge-based scoring.
Results
Current VLMs still struggle with spatial grounding, compositional instructions, and executable path generation, especially in occluded and multi-step settings.
Takeaways & Limitations
SleepWalk provides a controlled diagnostic testbed for assessing grounded spatial reasoning rather than exhausting all forms of embodied evaluation.
Takeaways & Limitations
The manuscript does not provide human-agreement studies, multi-judge comparisons, or prompt-sensitivity analyses for its evaluation protocol.
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) have advanced rapidly in multimodal perception and language understanding, yet it remains unclear whether they can reliably ground language into spatially coherent, plausibly executable actions in 3D digital environments. We introduce SleepWalk, a benchmark for evaluating instruction-grounded trajectory prediction in single-scene 3D worlds generated from textual scene descriptions and filtered for navigability. Unlike prior navigation benchmarks centered on long-range exploration across rooms, SleepWalk targets localized, interaction-centric embodied reasoning: given rendered visual observations and a natural-language instruction, a model must predict a trajectory that respects scene geometry, avoids collisions, and terminates at an action-compatible location. The benchmark covers diverse indoor and outdoor environments and organizes tasks into three tiers of spatial and temporal difficulty, enabling fine-grained analysis of grounding under increasing compositional complexity. Using a standardized pointwise judge-based evaluation protocol, we evaluate three frontier VLMs on 2,472 curated 3D environments with nine instructions per scene. Results reveal systematic failures in grounded spatial reasoning, especially under occlusion, interaction constraints, and multi-step instructions: performance drops as the difficulty level of the tasks increase. In general, current VLMs can somewhat produce trajectories that are simultaneously spatially coherent, plausibly executable, and aligned with intended actions. By exposing failures in a controlled yet scalable setting, SleepWalk provides a critical benchmark for advancing grounded multimodal reasoning, embodied planning, vision-language navigation, and action-capable agents in 3D environments.
1 Introduction
SleepWalk addresses whether VLMs can translate language into spatially grounded, executable behavior in 3D environments, focusing on localized interaction rather than long-range exploration. It introduces a three-tier benchmark and standardized evaluation to expose persistent grounding failures.
- Motivation: VLMs must connect language, space, and action by localizing themselves, reasoning about geometry, avoiding collisions, and reaching interaction-compatible locations.Embodied competence requires more than semantic recognition or text-pixel alignment.
- Motivation: Existing benchmarks often emphasize long-horizon room or building navigation and endpoint success, providing only partial visibility into localized trajectory grounding.SleepWalk targets continuous, spatially coherent, executable trajectories within a single 3D scene.
- Benchmark: SleepWalk evaluates continuous paths that respect scene layout, avoid collisions, and terminate at locations compatible with intended actions such as approaching, picking up, or sitting.The benchmark assesses full spatial and temporal coherence rather than symbolic planning or endpoint proximity alone.
- Benchmark: SleepWalk contains 2,472 curated 3D environments with nine instructions per scene across easy, medium, and hard tiers.The tiers support controlled analysis under increasing compositional and interaction complexity.
- Evaluation: A standardized judge-based protocol compares heterogeneous VLM trajectory outputs while emphasizing spatial feasibility, temporal coherence, and interaction compatibility.The protocol is designed to reveal failures in occluded, compositional, and interaction-heavy scenarios.
2 SleepWalk: A Three-Tier Benchmark for Grounded Trajectory Reasoning
SleepWalk constructs navigable single-scene 3D worlds from language, adds tiered instructions, and evaluates continuous trajectories against geometry and action constraints. Its pipeline combines visual observations, zero-shot VLM prediction, structured judging, and qualitative humanoid execution checks.
- Scene Construction: SleepWalk reconstructs navigable indoor and outdoor single-scene environments from natural-language descriptions using Hunyuan3D-3.0.The construction uses 1,200 filtered or rewritten scene descriptions spanning diverse layouts and object configurations.
- Instruction Generation: Each scene receives nine Qwen3-8B-VL-generated instructions across easy, medium, and hard tiers using top-down and oblique views.Difficulty increases from short-range localization to structured dependencies and multi-step interaction goals.
- Trajectory Prediction: Given visual observations and an instruction, a model predicts a continuous trajectory represented by ordered 3D waypoints.The trajectory must remain feasible within the environment, avoid collisions, and end where the intended action can be executed.
- Trajectory Prediction: SleepWalk evaluates the entire path rather than rewarding only proximity to the final goal.This stricter formulation tests consistency with scene geometry, object affordances, and temporal ordering.
- Evaluation: A structured vision-language judge scores trajectories pointwise on start-location consistency, goal satisfaction, obstacle avoidance, and trajectory efficiency.The judge receives the top-down scene, instruction, and trajectory overlay; scores are normalized and aggregated across tiers with equal default weighting.
- Embodied Validation: Selected outputs are converted into humanoid control signals and full-body motions for qualitative validation rather than primary scoring.This visualization can expose collisions, awkward stopping positions, and motion patterns incompatible with intended interactions.
3 Evaluating Grounded Trajectory Reasoning in 3D Scenes
SleepWalk evaluates whether frontier VLMs can convert instructions into spatially grounded, collision-aware, interaction-compatible trajectories in 3D scenes. Across qualitative and quantitative analyses, performance declines with compositional difficulty, while precise goal grounding remains a central weakness.
- Evaluation Setup: The evaluation tests Qwen3-VL, Gemini Robotics ER-1.5, and GPT-5-mini on trajectory prediction across easy, medium, and hard tasks.Models are evaluated under standardized conditions using deterministic, zero-shot predictions and judge-based scoring.
- Qualitative Results: Easy instructions preserve broad semantics, but all models can still produce trajectories that pass through or approach objects too closely, creating collision risks.GPT-5-mini selects the more plausible target lamp, whereas Gemini Robotics ER-1.5 ends closer to the farther lamp.
- Qualitative Results: Medium instructions expose compositional grounding failures: Gemini Robotics ER-1.5 identifies the final tree more accurately but misses the start region, while GPT-5-mini captures both references.The task requires resolving two references while preserving temporal order.
- Qualitative Results: Hard multi-step interaction instructions cause compounded errors, although GPT-5-mini produces the strongest trajectory by aligning the start, goal, and route.The hard example requires picking up a tray, walking to a table, and placing it there.
- Quantitative Results: GPT-5-mini performs best across all four reported factors, while obstacle avoidance is relatively stronger than goal grounding for the evaluated models.The factor analysis covers start-location consistency, goal satisfaction, obstacle avoidance, and trajectory efficiency.
- Quantitative Results: Performance drops from easy to medium to hard tiers, with Qwen3-VL degrading more sharply on goal grounding and trajectory efficiency.The steepest decline occurs on tasks requiring multi-step reasoning, ordered execution, and interaction-aware endpoint selection.
- Embodied Execution: Humanoid animation provides an additional qualitative check of whether top-down paths remain physically plausible during execution.This check can reveal collisions, awkward stopping positions, and unnatural motion transitions without replacing the primary judge-based evaluation.
4 Conclusion and Future Avenues
SleepWalk isolates grounded, interaction-centric trajectory reasoning in 3D scenes and shows that current VLMs still struggle with spatial grounding, compositional instructions, and executable paths. The benchmark is positioned as a scalable testbed for advancing embodied planning and action-capable vision-language systems.
- Conclusion: SleepWalk finds that current VLMs struggle with spatial grounding, compositional instructions, and executable path generation, especially in occluded and multi-step settings.The benchmark focuses on localized, interaction-centric reasoning to expose the gap between perception and action.
- Future Avenues: Future directions include richer observations, structured 3D reasoning, tighter path-to-control coupling, and physics-simulator extensions for embodied training and sim-to-real transfer.These directions are proposed as extensions to the benchmark and its downstream embodied applications.
- Conclusion: SleepWalk provides a scalable testbed for grounded multimodal reasoning, embodied planning, and action-capable vision-language systems.Its contribution is framed as a benchmark for advancing these research areas.
5 Frequently Asked Questions (FAQs)
SleepWalk’s primary contribution is a controlled benchmark for testing whether VLMs convert language into continuous, spatially grounded, executable trajectories. Its results show systematic degradation as compositional, interaction, and planning demands increase, revealing grounding failures beyond coordinate formatting.
- Benchmark contribution: SleepWalk defines a controlled benchmark for evaluating continuous, spatially grounded, and plausibly executable trajectories in single-scene 3D environments.The benchmark formulation is the central contribution; the generation pipeline instantiates it at scale.
- Benchmark contribution: The benchmark combines single-scene interaction-centric evaluation, scalable scene and instruction construction, and trajectory scoring under a standardized protocol.It reconstructs scenes, generates tiered instructions, evaluates predicted paths, and studies the resulting failures empirically.
- Empirical findings: Current frontier VLMs degrade substantially on compositional, interaction-heavy, and multi-step instructions, even though GPT-5-mini performs best across all four reported factors.The reported factors are start-location consistency, goal satisfaction, obstacle avoidance, and trajectory efficiency.
- Scope and interpretation: SleepWalk should be interpreted as a controlled diagnostic testbed rather than an exhaustive form of embodied evaluation.Its synthetic, standardized design measures a currently under-evaluated capability without claiming to replace real-world embodied evaluation.
- Empirical findings: The failures reflect grounded planning problems rather than merely output-format errors, including mislocalized starts, incorrect goals, broken temporal order, and physically unsafe trajectories.The evidence identifies substantive grounding and planning failures in explicit trajectory prediction.
- Empirical findings: Obstacle avoidance can be comparatively stronger than goal grounding, indicating that models may generate navigable-looking paths without reaching the correct interaction-compatible endpoint.The key bottleneck is binding instructions to the right object, preserving action sequence, and stopping at the right place.
- Empirical findings: Performance drops from easy to medium to hard tasks, while harder tiers expose sharper degradation in goal grounding, trajectory efficiency, and interaction-heavy reasoning.The tiering functions as a diagnostic axis for increasing compositional and interaction demands.
A.1 Benchmark overview
SleepWalk evaluates whether VLMs can transform natural-language instructions and rendered scene observations into executable trajectories in single-scene 3D environments.
- SleepWalk evaluates spatially coherent, executable trajectories generated from natural-language instructions and rendered scene observations.
- The benchmark emphasizes localized, interaction-centric reasoning rather than long-range room-to-room exploration.
- Each task is situated inside a single-scene 3D environment.
A.2 Scene generation and curation
SleepWalk constructs curated single-scene 3D environments and tiered instructions for localized, interaction-centric navigation. Scenes and tasks are filtered and represented to support grounded trajectory evaluation under geometric and interaction constraints.
- Scene generation: Scenes begin as textual descriptions converted into single coherent indoor or outdoor 3D environments with Hunyuan3D-3.0.
- Scene curation: Generated scenes are manually filtered for coherence, navigability, object recognizability, and geometric plausibility.Excluded scenes include disconnected spaces, insufficient free space, unidentifiable targets, and severely broken geometry.
- Scene representation: Each accepted environment provides top-down and oblique views, combining route and free-space structure with object identity and interaction context.
- Instruction generation: Nine Qwen3-8B-VL-generated instructions per scene are grouped into easy, medium, and hard tiers targeting progressively harder embodied reasoning.Instructions are rejected when they reference nonexistent objects, ambiguous global references, or visually unassessable actions.
- Trajectory evaluation: Predicted trajectories are represented as waypoint sequences, projected onto the top-down view, and judged for start validity, collision avoidance, and action-compatible termination.
- Task tiers: Tasks range from object-to-object navigation to simple interaction and multi-object or multi-state interaction sequences.
C Experimental Settings and Reproducibility
Experiments use a zero-shot, evaluation-only setup with identical inputs and formatting across models. The paper reports factor-score aggregation and plans to release the dataset for reproducibility.
- Experimental protocol: All experiments are zero-shot and evaluation-only, with no fine-tuning, adaptation, or benchmark-specific gradient updates.
- Experimental protocol: Each evaluated model receives the same rendered views, natural-language instruction, and output-formatting requirement.
- Evaluation aggregation: Normalized factor scores are averaged within tiers and then across tiers with equal tier weight.
- Reproducibility: The authors plan to release the SLEEPWALK dataset containing 3D environments and corresponding instructions upon acceptance.
D Additional Qualitative Examples
Additional examples illustrate SleepWalk’s three instruction tiers through increasingly involved navigation and interaction tasks. The examples span simple movement, object placement, and multi-step manipulation.
- Qualitative examples: The appendix presents qualitative examples generated by GPT5-mini across Easy, Medium, and Hard SleepWalk tiers.
- Easy: The Easy example asks an agent to walk from a trash can to a whiteboard without interaction.
- Medium: The Medium example requires placing a green book from a shelf onto a teacher’s desk.
- Hard: The Hard example combines movement with picking up a book and placing it on a teacher’s podium.
E.1 Benchmark limitations
SleepWalk evaluates an intermediate capability rather than fully interactive embodied behavior. Its reconstructed scenes, judge-based scoring, and top-down overlays each constrain what the benchmark captures.
- Benchmark limitations: Reconstructed single-scene environments approximate object dynamics and low-level contact mechanics instead of modeling fully interactive physics-based worlds.
- Benchmark limitations: The primary evaluation relies on a strong judge model rather than human annotation for every sample, which may introduce scoring bias.
- Benchmark limitations: Top-down trajectory overlays simplify comparison but do not capture every detail of embodied execution.
E.2 Observed failure modes
Across models, SleepWalk reveals recurring failures in starting-location grounding, goal grounding, and geometrically safe execution. These failures motivate diagnosing grounded spatial reasoning before real-world deployment, while recognizing that improved language-conditioned navigation may be dual-use.
- Models sometimes start from the wrong region despite correctly identifying the overall target object.
- Paths can be plausible yet terminate at the wrong object, the wrong side, or a non-interaction-compatible location.
- Semantically sensible trajectories may pass through clutter, clip furniture, or stop where a humanoid agent could not realistically execute them.
- The benchmark diagnoses grounded spatial-reasoning failures to support safer and more reliable embodied multimodal systems before real-world deployment.
- Better language-conditioned navigation may also improve autonomous capabilities in ways that are dual-use.