PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboTALES: Learning Reasoning-Guided Robot Policies via Task-Aligned Simulated Futures

Hanan Gani, Tejal Kulkarni, Madhoolika Chodavarapu, Nicklas Hansen, Manmohan Chandraker

arXiv:2607.06018v1cs.RO

TL;DR

Imagined futures from video generative models can drift from task intent and lack reliable action conditioning, limiting their use for robot planning and policy extraction. RoboTALES combines hierarchical subgoal planning with VLM-critic feedback in single-stage training to produce task-aligned futures and policies, outperforming strong baselines across challenging manipulation benchmarks, especially for long-horizon tasks.

  • Problem

    Robot agents need hierarchical, goal-directed imagination and semantically faithful futures to reliably use visual rollouts for long-horizon action generation.

  • Method

    RoboTALES uses an LLM planner for structured subgoals, a VLM critic for semantic reward feedback, and coupled training of video and action models.

  • Results

    RoboTALES outperforms strong baselines across task success, demonstration efficiency, long-horizon ability, and robustness on RoboCasa and LIBERO manipulation tasks.

  • Takeaways & Limitations

    Reasoning-guided, critic-shaped visual representations support smoother and more purposeful robot actions during long-horizon manipulation.

  • Takeaways & Limitations

    The current setup is purely simulation-based, limiting its demonstrated scope to simulated manipulation tasks.

Abstract

from arXiv · show

Pretrained video generative models are promising backbones for visuomotor control, but their imagined futures often drift from task intent and are not reliably action-conditional. As a result, these models can be difficult to use for planning or policy extraction. To address these limitations, we propose RoboTALES, a single-stage framework that learns task-aligned simulated futures and uses them to train robot policies. Our approach introduces two key innovations: (1) a hierarchical LLM-based planner that breaks complex tasks into a sequence of subgoals to guide the model's imagination; and (2) a VLM-based critic that evaluates these ``imagined'' futures and uses reward-based feedback to keep the model's internal representations focused on the goal. By anchoring the video generator in abstract reasoning, we produce temporally consistent rollouts and more coherent actions. We evaluate RoboTALES on diverse manipulation tasks from RoboCasa and LIBERO10, and show that our method consistently outperforms existing methods, especially in long-horizon tasks. Our code and models are publicly available at https://github.com/hananshafi/RoboTALES.

1 Introduction

RoboTALES addresses the weak task alignment of imagined video futures by combining hierarchical language-guided planning with vision-language critique. These components produce structured, semantically faithful predictive features for more precise long-horizon robot actions.

  • Overview: RoboTALES jointly uses an LLM planner and VLM critic to make video-generated futures goal-conditioned and improve robot action generation for long-horizon manipulation.The planner constrains which futures are imagined, while the critic ensures they remain worth acting upon.
  • Motivation: Video generators can provide strong priors for policy learning, but visual realism and reconstruction objectives leave their simulated futures weakly grounded for control.Existing approaches use generated rollouts to supervise or condition actions, yet video generation alone does not constitute a reliable world model.
  • Method: The VLM critic supplies language-aligned reward feedback through differentiable reinforcement learning to steer the video generator toward semantically faithful futures.This directly refines the generator’s latent dynamics rather than leaving language and prediction as loosely coupled modules.
  • Method: The LLM planner decomposes task instructions into ordered temporal subtasks, imposing hierarchical structure that organizes diffusion-generated rollouts around task milestones.This converts rollout generation from an undifferentiated prediction problem into structured, milestone-based imagination.
  • Results: Conditioning action generation on reasoning-aligned predictive features yields smoother, more purposeful, precise, and semantically consistent robot trajectories.The resulting features link robot actions directly to the reasoned imagined futures.

2 Related Work

Prior work uses diffusion models for temporally consistent, multimodal policy learning and video generators as world models for robot control. Language-based planners provide abstract task reasoning, but RoboTALES integrates that reasoning into the generative process rather than leaving it decoupled from predictive representations.

  • Diffusion Policy Models: Diffusion policies address over-conservatism, limited expressiveness, and compounding errors by modeling trajectories jointly across timesteps and capturing multimodal action distributions, [26].Diffuser introduced trajectory-level diffusion to jointly predict all plan timesteps, improving temporal consistency and reducing compounding errors.
  • Video Generation for Robot Actions: Large-scale video generative models are increasingly used as robot-control world models because video prediction captures physical dynamics, object interactions, and causal structure [41], [28].VideoPolicy [35] builds on Stable Video Diffusion (SVD) [8] and conditions a 1D action UNet on hidden embeddings extracted from the video UNet’s decoder.
  • Planning via Language: LLMs and VLMs support embodied-agent planning through task decomposition, affordance grounding, environment-feedback loops, and visual waypoint generation [3],,.These approaches demonstrate that language can serve as a medium for abstract task reasoning.
  • Planning via Language: Unlike prior planners whose language reasoning remains decoupled from predictive representations, RoboTALES conditions the generative process itself to embed structured task reasoning into latent visual dynamics.Prior plans influence action selection but do not shape what the agent imagines about the future.

3 RoboTALES

RoboTALES addresses long-horizon visuomotor control by decomposing language goals into hierarchical subgoals, generating task-aligned future states, evaluating them with a VLM critic, and jointly training video and action models. Action-level feedback shapes the video generator’s representations for control rather than leaving the policy to adapt to fixed features.

  • Framework: RoboTALES combines an LLM planner, video generator, action policy, and RL-based VLM critic to produce task-aligned imagined futures and actions.The planner supplies semantic subgoals, the generator predicts future states, the critic evaluates alignment with the task, and the policy uses hierarchical trajectory representations.
  • Hierarchical Planner: The LLM planner decomposes compressed instructions into context-grounded, semantically coherent subgoals that provide dense milestones for long-horizon visual prediction.The resulting roadmap makes implicit task knowledge explicit and guides futures toward causal task progression rather than visual plausibility alone.
  • VLM Critic: A frozen VLM critic scores imagined rollouts for semantic alignment, realism, and goal satisfaction, and its reward optimizes targeted video-generator parameters.The reward is applied through differentiable policy optimization to reinforce denoising transitions that yield high-reward visual states.
  • Joint Training: Targeted updates preserve the pretrained visual prior while adapting planner-conditioned cross-attention modules and decoder layers that provide features to the action UNet.The trainable parameters are restricted to interfaces where task semantics enter the model and where decoder representations condition action generation.
  • Joint Training: Joint single-stage training sends action-level gradients through the policy’s conditioning features into trainable video-generator decoder layers, shaping futures for action prediction.This removes the stop-gradient used by decoupled approaches and encourages latent representations that encode spatial, temporal, semantic, and control-relevant structure.

4 Experimentation

RoboTALES achieves the strongest reported performance across RoboCasa and LIBERO10, with gains especially evident in long-horizon manipulation. Ablations and qualitative analyses attribute these gains to planner-grounded imagination, critic-shaped representations, and improved execution stability.

  • Main Results: RoboTALES attains the highest average success on RoboCasa and improves consistently across task categories, including structure-sensitive door and drawer tasks.Table 1 evaluates 50 rollouts per task using 50 demonstrations.
  • Main Results: 97% mean success across all 10 LIBERO10 tasks demonstrates RoboTALES’s generalization beyond RoboCasa.The reported LIBERO10 result is summarized in Table 2.
  • Qualitative Comparison: Qualitatively, RoboTALES preserves environmental structure and successfully completes object placement, whereas VideoPolicy exhibits semantic drift during long-horizon transitions.The comparison focuses on task-execution stability and semantic grounding in RoboCasa manipulation tasks.
  • Effect of the Planner: Planner conditioning improves performance only after video-generator training, while naively adding structured subgoals to the baseline degrades success.The ablation compares inference-time planner augmentation, planner-conditioned training without the Critic, and the full model across three RoboCasa tasks.
  • Demonstration Efficiency: The Planner adds approximately 1s of execution overhead, indicating that reasoning-guided conditioning remains computationally efficient.Demonstration-efficiency results are averaged over four RoboCasa tasks and compare RoboTALES with two baselines.
  • Effect of Policy Optimization: The VLM Critic improves representation stability and downstream control, producing lower end-effector jerk and more physically plausible actions than pixel-level prediction alone.These results indicate that critic-based refinement shapes internal representations rather than merely improving visual fidelity.

5 Conclusion and Future Work

RoboTALES couples hierarchical reasoning, video generation, and action generation for long-horizon robotic manipulation. It uses structured sub-goals and a frozen VLM Critic to guide task-aligned visual imagination beyond pixel-level fidelity.

  • Conclusion and Future Work: RoboTALES tightly couples hierarchical reasoning, video generation, and action generation for long-horizon robotic manipulation.
  • Conclusion and Future Work: An LLM Planner decomposes compressed task instructions into structured sub-goals that condition the video model.
  • Conclusion and Future Work: A frozen VLM Critic steers the video generator toward semantically faithful futures, enabling task-aligned visual imagination beyond pixel-level fidelity.

RoboTALES: Learning Reasoning-Guided Robot Policies via

This section concerns task-aligned simulated futures.

  • The section focuses on task-aligned simulated futures.

A Appendix

The appendix contains quantitative LIBERO10 results, additional ablations and qualitative results, planner details, and limitations.

  • A.1 Quantitative Results on LIBERO10 Benchmark: The appendix reports quantitative results on the LIBERO10 benchmark.
  • A.2 Additional Ablations and Analysis: It presents additional ablations and analysis.
  • A.3 Additional Qualitative Results: It includes additional qualitative results.
  • A.4 Planner Details: It provides details of the planner.
  • A.5 Limitations: It discusses the method’s limitations.

A.1 Quantitative Results on Libero10 Benchmark

RoboTALES substantially outperforms all baselines on the LIBERO10 benchmark, achieving perfect or near-perfect success on most of its 10 tasks.

  • Quantitative Results: RoboTALES outperforms all baselines by a substantial margin in mean success rate across all 10 LIBERO10 tasks [36].The comparison is reported in Table 2 of the main paper.
  • Per-task Results: RoboTALES achieves perfect 1.0 success on both turning on the stove and placing the moka pot, and placing the alphabet soup and cream cheese box in the basket.These results are among the majority of tasks with perfect or near-perfect success, based on 50 trials per task.

A.2 Additional Ablations and Analysis

Additional analyses show that RoboTALES’ critic is calibrated and benefits from trajectory-level averaging, while task geometry affects critic choice and action horizon. The experiments also support joint training and planner-guided decomposition quality, alongside evaluation of retained video-generation capabilities.

  • Effect of Critic: Critic rewards are well calibrated: success rises from 0.40 in the lowest reward quartile to 0.88 in the highest across 360 rollouts.The overall rollout success rate was 0.63, and higher predicted reward reliably corresponded to greater success probability.
  • Effect of Critic: Mean trajectory rewards achieve AUROC 0.72 versus 0.59 for single-frame maxima, supporting sustained trajectory agreement over isolated reward peaks.This motivates using mean-aggregated rewards for critic feedback.
  • Effect of different VLM critic choices: SBERT is selected as the primary critic because critic performance varies by task geometry and semantic complexity, while SBERT has the higher overall average success rate.CycleReward slightly leads on Coffee SetupMug, whereas SBERT significantly outperforms on PnP CounterToSink and PnP MicrowaveToCounter.
  • Effect of changing action horizon: Action horizon 16 performs best across three tasks, while horizons 8 and 32 reduce performance, indicating a tradeoff between re-planning and action commitment.Short horizons can become overly reactive, whereas long horizons increase open-loop drift and error accumulation.
  • Joint Training vs. Stage-wise Training: Planner decompositions outperform prompt-only generation, with 60% versus 38% wins and fewer steps, 2.80 versus 5.28, indicating greater efficiency.The analysis supports concise, useful decompositions rather than longer plans that merely game the judge.
  • Retaining and Improving General Video Generation Capabilities: The study evaluates whether joint task-aligned training preserves video-generation capabilities using matched UCF101 conditioning samples and 30 denoising steps.Generated futures are compared with ground-truth future clips in feature space.

A.3 Additional Qualitative Results.

Extended qualitative results support RoboTALES’ robustness, showing goal-directed behavior over long horizons and successful execution across diverse complex kitchen manipulation tasks.

  • A.3 Additional Qualitative Results.: Figure 13 shows RoboTALES maintaining goal-directed behavior over long horizons compared with the baseline.The comparison is presented as an extended qualitative visualization of long-horizon behavior.
  • A.3 Additional Qualitative Results.: Figure 14 showcases successful execution across a diverse array of complex kitchen manipulation tasks.These examples further demonstrate the framework’s robustness qualitatively.
  • A.3 Additional Qualitative Results.: The section provides extended qualitative comparisons and visualizations to further demonstrate the robustness of the framework.The results complement the long-horizon and kitchen-task examples with additional visual evidence.

A.4 Details about Planner

The planner uses Gemini 2.5 Pro to decompose high-level robot instructions into 2–5 ordered, observable sub-goals for video rollout. These decompositions are cached to support computational efficiency and consistency during training and inference.

  • A.4 Details about Planner: Gemini 2.5 Pro converts high-level RoboCasa instructions into 2–5 ordered actions representing meaningful, observable environment-state transitions.The planner follows VLWM-style abstraction and constrains each instruction to 2–5 action steps so policies can learn meaningful sub-goals.
  • A.4 Details about Planner: Planner decompositions are pre-generated and cached by instruction and model-version hash, avoiding redundant API calls and stabilizing downstream training.The persistent local cache also ensures consistent decompositions across inference rollouts.
  • A.4 Details about Planner: Table 4 illustrates Gemini 2.5 Pro decompositions for Libero10 instructions, grounding complex requests in manageable intermediate world-state transitions.The examples demonstrate sequential sub-goals under the 2–5-step granularity constraint.

A.5 Limitations

RoboTALES remains limited by its frozen VLM Critic, expensive DDPO-based optimization, and purely simulation-based setup. The authors identify adaptive reward modeling and broader deployment beyond simulation as future work.

  • Limitations: The frozen VLM Critic supplies a coarse, task-level reward signal, limiting feedback richness, especially for contact-rich tasks.The authors suggest a learnable reward model that co-adapts with the video generator and action policy.
  • Limitations: DDPO-based policy optimization is computationally expensive because each training step requires multiple denoising rollouts to estimate gradients.
  • Limitations: The current RoboTALES setup is purely simulation-based, and extending it beyond simulation remains ongoing and future work.
Loading 2607.06018v1…