PaperlayerView exact revision on arXiv

Source-linked AI summary

This&That: Language-Gesture Controlled Video Generation for Robot Planning

Boyang Wang, Nikhil Sridhar, Chao Feng, Mark Van der Merwe, Adam Fishman, Nima Fazeli, Jeong Joon Park

arXiv:2407.05530v2cs.ROcs.AIcs.CV

TL;DR

Simple verbal descriptions can be verbose and ambiguous, while pointing with “this” or “that” can communicate tasks clearly. This&That generates language-gesture-conditioned video plans and uses behavior cloning to produce robot actions; the authors report that videos and rollouts align exceptionally well with user intentions.

  • Problem

    Simple verbal descriptions can be verbose and ambiguous, while pointing with “this” or “that” can communicate targets clearly.

    Source: §I · p. 1

  • Method

    This&That conditions video generation on language and gestures, then uses a behavior-cloning policy to translate video plans, live images, and end-effector pose into action chunks.

    Source: §I · p. 1, §V · p. 4

  • Results

    The authors report that generated videos and subsequent rollouts align exceptionally well with users’ intentions.

    Source: §VIII · p. 7

  • Takeaways & Limitations

    The authors present This&That as a direction toward multi-task human-robot collaborations.

    Source: §VIII · p. 7

    Video-based behavioral-cloning experiments are limited to simulation; real-robot testing was unavailable because the WidowX 250 arm was unavailable at the authors’ facility.

    Source: §VII · p. 7

Abstract

from arXiv · show

Clear, interpretable instructions are invaluable when attempting any complex task. Good instructions help to clarify the task and even anticipate the steps needed to solve it. In this work, we propose a robot learning framework for communicating, planning, and executing a wide range of tasks, dubbed This&That. This&That solves general tasks by leveraging video generative models, which, through training on internet-scale data, contain rich physical and semantic context. In this work, we tackle three fundamental challenges in video-based planning: 1) unambiguous task communication with simple human instructions, 2) controllable video generation that respects user intent, and 3) translating visual plans into robot actions. This&That uses language-gesture conditioning to generate video predictions, as a succinct and unambiguous alternative to existing language-only methods, especially in complex and uncertain environments. These video predictions are then fed into a behavior cloning architecture dubbed Diffusion Video to Action (DiVA), which outperforms prior state-of-the-art behavior cloning and video-based planning methods by substantial margins.

I. INTRODUCTION

This&That uses language and pointing gestures to generate video plans that reflect user intent, then uses behavior cloning to translate those plans into robot actions. The paper positions this approach against prior video-based planning and goal-conditioned imitation-learning methods.

I. INTRODUCTION

  • Pointing with “this” or “that” can communicate a target more clearly and concisely than a verbose verbal description.The introduction motivates this form of instruction as practical for controlling robots across tasks.Source: §I · p. 1
  • Language-gesture conditioning combines deictic words and gesture locations in the first frame to generate videos aligned with human intent, including in complex, uncertain scenes.The generated video serves as a plan for robot execution.Source: §I · pp. 1–2
  • The video generator acts as a generalizable planner, envisioning environmental changes across tasks, while the execution module follows the predicted video.This connects video prediction to robot actions.Source: §I · pp. 1–2
  • Unlike prior video-based methods that simplify the action space or use special inverse-dynamics models, This&That integrates video prediction into behavior cloning.Its behavior-cloning module cross-attends to video frames.Source: §I · p. 2

II. RELATED WORK

  • Prior video-based robotics approaches use inverse dynamics, predicted visual sub-goals, or optical-flow-derived actions; the paper identifies ambiguous-scene prediction as an under-explored challenge.Related work also notes that many motion-control methods require dense spatial or temporal information, whereas This&That uses two points.Source: §II · p. 2

III. OVERVIEW

This&That combines language-gesture-conditioned video generation with video-based robot planning. Its video generator builds on Stable Video Diffusion, fine-tuned on robotics videos and conditioned on language and an initial frame.

III. OVERVIEW

  • The framework has two components: language-gesture-conditioned video generation and video-based robot planning that converts generated plans into low-level actions.The paper evaluates whether generated videos align with user intent and can support simulated tasks.Source: §III · p. 2

IV. LANGUAGE-GESTURE CONDITIONED VIDEO DIFFUSION MODELS

  • Stable Video Diffusion is fine-tuned first with text and initial frames, then modified and fine-tuned with gesture conditioning to predict video frames from the initial frame and both conditions.The model is built on a large-scale, open-vocabulary video diffusion model.Source: §IV · p. 3

A. Language-Conditioned Finetuning

  • For language-conditioned fine-tuning, the model incorporates language and the first frame, modulating intermediate features using parameters derived from cross-attention between language and image tokens.Training adds Gaussian noise to video frames and optimizes noise reconstruction conditioned on the first frame and text.Source: A. Language-Conditioned Finetuning · p. 3

B. Gesture-Conditioned Training and Inference

Gesture conditioning adds sparse pointing locations to language-conditioned video generation, while DiVA uses video plans and live observations to predict robot actions. The section describes both the gesture-conditioning approach and the video-conditioned behavior-cloning policy.

B. Gesture-Conditioned Training and Inference

  • Gestures are represented as 2D coordinates relative to the first frame and added through a supplementary conditioning branch alongside the diffusion UNet.The diffusion UNet is frozen while the new gesture branch is trained.Source: B. Gesture-Conditioned Training and… · p. 3
  • Naïve ControlNet conditioning struggles with gestures given as two 1-pixel coordinates only in the first two frames, and gesture alone does not specify the task.The method concatenates first-frame latents, noisy video frames, and sparse gesture images before the branch's first convolution to provide dense signals.Source: B. Gesture-Conditioned Training and… · p. 4
  • Because robotics video datasets lack ground-truth gestures, the authors detect and track the gripper and relevant objects to annotate target points.For tasks such as opening a door, they use only one gesture point.Source: B. Gesture-Conditioned Training and… · p. 4

V. VIDEO-CONDITIONED BEHAVIORAL CLONING

  • DiVA predicts action chunks from the live image, end-effector pose, and a subset of video-plan frames using a Transformer encoder-decoder policy.It compresses image embeddings to 16 tokens per image and cross-attends between live-image and video-plan tokens.Source: §V · p. 4
  • DiVA trains on evenly spaced demonstration images as goals, with temporal noise added to address errors from small timing misalignments between video plans and actions.The authors attribute its success to image-token compression, cross-attention between observation and goal tokens, and temporal noise when sampling training goals.Source: §V · p. 4

VI. EXPERIMENTS

The experiments assess video realism and user alignment, as well as the use of predicted video plans in downstream policy learning. The supplied passages identify the behavior-cloning architecture and video-quality metrics, but do not provide comparative result values.

VI. EXPERIMENTS

  • The experiments assess realistic, user-aligned video generation, language-gesture conditioning, and the integration of predicted video plans into downstream policy learning.The stated evaluation spans video generation and translation of plans into robot actions.Source: §VI · p. 4
  • DiVA uses a Transformer encoder-decoder to convert video plans into executable robot actions, compressing image embeddings with TokenLearner and cross-attending to video-plan tokens.The architecture combines video plans with live observations.Source: Figure 4 · p. 5
  • Bridge video quality is assessed using FID and FVD, where lower is better, PSNR and SSIM, where higher is better, and LPIPS, where lower is better.The table caption states that the best result in each metric column is highlighted.Source: Table II · p. 5

A. Video Generation Experiments and Comparisons

Language–gesture conditioning improves generated video quality and alignment with user intent, particularly when scenes are ambiguous. In simulation, the full video-generation and behavior-cloning pipeline outperforms the cited baselines, while language and gesture address different ambiguities.

A. Video Generation Experiments and Comparisons

B. Synthetic Rollout Experiments

  • The full language–gesture VDM + DiVA pipeline outperforms ACT, AVDC + DiVA, and language-conditioned VDM in synthetic rollouts.The comparison includes in-distribution and out-of-distribution scenes; Fig. 5 qualitatively compares the pipeline with AVDC.Source: Table III · p. 6, B. Synthetic Rollout Experiments · p. 6, Figure 5 · p. 6
  • Synthetic rollout models are trained from scratch on scripted pick-and-place trajectories, then tested on out-of-distribution scenes with identical blocks.Source: B. Synthetic Rollout Experiments · p. 6
  • Language-only instructions remain inadequate for ambiguous blocks, while gesture improves success; a language cue resolves 3D ambiguity in gesture-only input.The authors stress-test with identical blocks and note that even spatial adjectives do not make language alone sufficient.Source: B. Synthetic Rollout Experiments · p. 6, Figure 6 · p. 7

VII. LIMITATIONS

The authors identify shape changes, short modular task predictions, gesture-only 3D ambiguity, and the lack of real-robot testing as limitations. They conclude that This&That’s generated videos and rollouts align with user intentions.

VII. LIMITATIONS

  • Predictions are limited to short, modular tasks, and object shapes may change over time; gesture alone can also fail under 3D ambiguity.Source: §VII · p. 7
  • Real-robot behavioral-cloning experiments were not conducted because the WidowX 250 arm was unavailable at the authors’ facility.The authors cite prior Bridge-data research reporting successful transfer and close correlation between real-world and simulated performance.Source: §VII · p. 7

VIII. CONCLUSION

  • The authors conclude that generated videos and subsequent rollouts align well with user intentions in This&That’s video-planning and imitation-learning framework.Source: §VIII · p. 7

APPENDIX

The appendix adds qualitative comparisons, four VDM design ablations, and a resolution-matched comparison with AVDC. Across these analyses, This&That videos align better with instructions, while several design choices materially affect performance.

1) Qualitative Comparison with Contemporary Video Generative Models:

  • DragAnything, StreamingT2V, and SVD fail to follow commands; AVDC is semantically correct but visually deficient, while This&That produces high-quality, intention-aligned video.AVDC's output has low spatial and temporal resolution and microwave-door artifacts that hinder interpreting robot and environment states.Source: 1) Qualitative Comparison with… · p. 9
  • In test-split examples without corresponding ground-truth videos, language-only models struggled with nuanced geometry, and AVDC and a language-only baseline failed a user study example.AVDC also failed to follow a straightforward closing prompt, likely due to overfitting on training data.Source: 1) Qualitative Comparison with… · p. 9

2) VDM Ablation Study:

  • Replacing the proposed pre-trained VAE encoding and gesture/noise concatenation with naïve ControlNet conditioning significantly reduced metrics and gesture adherence.Visual inspection confirmed that the modified model did not accurately follow gesture cues.Source: 2) VDM Ablation Study · p. 9
  • SAM masks did not improve numerical performance; broad scene segments added extraneous pixels, while 2D dilation from a point yielded better results.SAM sometimes selected a desk rather than the target cup.Source: 2) VDM Ablation Study · p. 9
  • Removing Layer Normalization after concatenating CLIP embeddings for language and image inputs worsened performance across all numerical metrics.Source: 2) VDM Ablation Study · p. 9
  • Gesture regions made twice or half as large as the 10x10-pixel block were not helpful.Source: 2) VDM Ablation Study · p. 9

3) Additional Quantitative Comparison Considering Image Resolution:

  • At AVDC's native 48x64 resolution, This&That still outperformed AVDC across all metrics after resizing the comparison images.The authors could not retrain AVDC at their VDM's 256x384 default resolution because of memory demands and a 48GB device limit.Source: 3) Additional Quantitative Comparison… · pp. 9–10

4) Additional Qualitative Results from Simulation Rollouts:

Simulation results show that gesture conditioning addresses scene ambiguity left unresolved by language-only prompts, and DiVA translates generated plans into robot actions. The section also reports DiVA ablations and details the video model's base and fine-tuning choices.

4) Additional Qualitative Results from Simulation Rollouts:

  • Language-only baselines still failed on ambiguous identical-block scenes after test-time prompts added location adjectives, while This&That generated high-quality, intent-aligned plans.The reported comparisons include generated video plans and DiVA-executed rollouts.Source: 4) Additional Qualitative Results from… · p. 10
  • DiVA's success rose roughly linearly as goal frames increased from 1 to 15, then plateaued through 25; performance was nearly zero with one frame.The authors hypothesize more frames may help complex tasks, leaving those experiments for future work.Source: 4) Additional Qualitative Results from… · p. 10
  • Adding temporal noise by sampling one demonstration image from each of N consecutive groups made DiVA more accurate and robust.The study compares pick and overall success rates with and without temporal noise.Source: 4) Additional Qualitative Results from… · p. 10

1) Base Architecture:

  • The video diffusion model builds on Stable Video Diffusion, whose denoising algorithm was modified using an open-source codebase because SVD training code was unavailable.Source: 1) Base Architecture · p. 10

2) UNet Finetuning Details (Stage 1):

  • For robotics sequences, the model fixes the motion bucket ID at 200 and noise augmentation at 0.1 during training and inference to override pretrained motion control.Source: 2) UNet Finetuning Details (Stage 1) · pp. 10–11
  • A fixed 0.1 noise augmentation replaces the small random noise traditionally added to the conditioning frame.The change is described as enhancing VDM stability.Source: 2) UNet Finetuning Details (Stage 1) · pp. 11–12
  • Compared with contemporary video generators, This&That followed user intention with high-quality video, whereas DragAnything, StreamingT2V, and SVD failed to follow language or gesture commands.AVDC produced a reasonable result on a scene from its training data but lacked visual quality.Source: Figure 9 · p. 12
  • Layer-normalizing concatenated CLIP text and first-frame embeddings improved visual and quantitative outcomes.The concatenated sequence has 78 tokens with feature dimension 1024.Source: 2) UNet Finetuning Details (Stage 1) · p. 12

3) Gesture Conditioning Branch Details (Stage 2):

The gesture branch adds temporally distributed, spatially expanded gesture cues to the video diffusion model. For real-data training, an automated pipeline detects and tracks gripper and object locations, with low-accuracy cases excluded.

3) Gesture Conditioning Branch Details (Stage 2):

  • Across T target frames, two gesture points appear in randomly positioned frames, with the red point preceding the green; other frames use zero-valued placeholders.Each point is a 10-pixel square centered on its coordinate, expanded using 2D Gaussian dilation.Source: 3) Gesture Conditioning Branch Details… · p. 12
  • The gesture-branch input concatenates noise, repeated initial-frame latents, and gesture-condition latents into a 12-channel representation.The input dimensions are (B × T) × 12 × H × W.Source: 3) Gesture Conditioning Branch Details… · p. 12
  • A newly trained, zero-initialized convolution handles the gesture branch's differing input dimensions; latent masking is applied only on this branch.Masking the UNet side caused performance degradation.Source: 3) Gesture Conditioning Branch Details… · pp. 12–13
  • The gesture branch encodes its input and supplies skip connections to the UNet decoder, while both branches use FiLM parameters obtained through cross-attention to CLIP embeddings.Source: 3) Gesture Conditioning Branch Details… · p. 13

4) Automatic Gesture Labeling on Real Data:

  • Automatic real-data labeling detects gripper boxes using a YOLOv8 detector trained on 450 annotated images, then tracks target objects to obtain interaction points.Tracking captures object trajectories, including objects released mid-air and their landing points; coordinates are expanded to 10x10-pixel blocks.Source: 4) Automatic Gesture Labeling on Real… · p. 14
  • The annotation pipeline is imperfect for complex interactions, and videos with tracking accuracy below acceptable thresholds are excluded.Source: 4) Automatic Gesture Labeling on Real… · p. 14

D. DiVA Model Implementation Details

DiVA conditions action-chunk prediction on the current image observation, end-effector pose, and selected goal images from a generated video. The authors contrast this design with an inverse-dynamics alternative that was impractical for variable-length demonstrations.

D. DiVA Model Implementation Details

  • DiVA models action chunks from the current image, end-effector pose, and a subset of video goal images; each goal image is encoded into a latent representation.Goal and observation embeddings are tokenized and processed with transformer attention; the experiment executes chunks of 10 actions before querying again.Source: D. DiVA Model Implementation Details · p. 15

1) Alternative Method: Inverse Dynamics Model:

  • The authors' inverse-dynamics alternative failed to produce reasonable actions because fixed-length generated videos did not match variable-length demonstrations.Their approach avoids the extra temporal-interpolation generative step and is described as robust to input-image counts and temporal misalignment.Source: 1) Alternative Method: Inverse… · pp. 15–16

D. DiVA Model Implementation Details

  • At AVDC's native low resolution, This&That outperformed AVDC across all VDM comparison metrics.Source: Table V · p. 16

E. Experiment Details

The experiments specify how the video model was trained and evaluated, including human assessments of user alignment and tests of deictic prompts. They also report occasional changes in moved-object appearance.

1) VDM Training Details:

  • Bridge VDM training uses 8 L40S GPUs for 99K UNet iterations, followed by 4 GPUs for 30K gesture-conditioning iterations.Horizontal flipping augments the data with probability 0.45, except when prompts contain positional terms such as left or right.Source: 1) VDM Training Details · p. 16

2) VDMs Visual-Quality Experiment Details:

  • Bridge video-quality comparisons use a 10% V1/V2 test split, five metrics, and standardized 256×384, 14-frame outputs.The reported metrics are FID, FVD, PSNR, SSIM, and LPIPS; all methods’ outputs are resized to the same resolution and frame count.Source: 2) VDMs Visual-Quality Experiment… · p. 16

3) User Alignment Experiment Details:

  • Three robotics-experienced evaluators independently assessed 24 cases across four task types, with prompts and gesture conditions varied across methods.The study also altered prompts and gesture points to test novel objects and placements; participants judged whether videos matched the stated intention.Source: 3) User Alignment Experiment Details · p. 16
  • The VDM occasionally changes a moved object’s appearance over time despite closely following user intent.The authors attribute this to limits of representing 3D geometry in 2D video.Source: Figure 13 · p. 17

4) Preparing Deictic Language Prompts:

  • Deictic prompts are formed by adding the first action verb to “this to there” and are tested zero-shot using weights trained on regular text conditions.Examples include “put this to there,” “fold this to there,” and “stack this to there.”Source: 4) Preparing Deictic Language Prompts · p. 17

5) Training Details of the VDM Baselines: •

In simulation, the vision-plus-language-plus-gesture model outperforms the other baselines, especially in ambiguous and out-of-distribution scenes. The section also describes the training and evaluation setups and notes occasional object-appearance artifacts.

5) Training Details of the VDM Baselines: •

  • The VDM baselines comprise vision-only, vision-plus-language, vision-plus-gesture, and vision-plus-language-plus-gesture variants, alongside pretrained AVDC inference.The variants use Stage 1 UNet weights and, where applicable, the Stage 2 gesture-conditioning branch.Source: 5) Training Details of the VDM… · p. 17

6) DiVA Training Details:

  • DiVA is trained on 900 instances and tested on 100, using ground-truth visual goals rather than generated videos.The authors report retaining high performance with ground-truth goals and infer little domain gap from generated videos to those goals.Source: 6) DiVA Training Details · p. 17

7) Simulation Experiment Details:

  • The simulation dataset uses four tabletop blocks with varied shapes, colors, and spatial relations, plus OOD scenes with identical objects and colors.Pick and place success are assessed with scripted criteria; near-task placement tolerances are more lenient than stack-task tolerances because 2D gestures have some 3D ambiguity.Source: 7) Simulation Experiment Details · p. 17

8) Baselines in Simulated Experiments:

  • Simulation comparisons include vanilla ACT, language-conditioned ACT, language-and-gesture ACT, retrained AVDC, and a language-only VDM.The ACT variants concatenate encoded language and gesture with current-image tokens; AVDC and the VDM are trained on simulation demonstrations.Source: 8) Baselines in Simulated Experiments · p. 18
  • Vision+Lang.+Gesture outperforms all other baselines on regular and OOD tests, especially in ambiguous scenes.Generated videos are fed to DiVA to produce robot actions; the comparisons are reported in Figs. 10 and 11.Source: 8) Baselines in Simulated Experiments · p. 18

F. Limitation: Changes in Object Appearance Over Time

  • The VDM occasionally changes the shape of moved objects over time, which the authors suspect reflects limits of representing 3D geometry in 2D video.They suggest these artifacts may not hinder action translation when the end-effector remains clear and discernible.Source: F. Limitation: Changes in Object… · p. 18
Loading 2407.05530v2…