Source-linked AI summary
TASTE-Rob: Advancing Video Generation of Task-Oriented Hand-Object Interaction for Generalizable Robotic Manipulation
Hongxiang Zhao, Xingchen Liu, Mutian Xu, Yiming Hao, Weikai Chen, Xiaoguang Han
TL;DR
Existing datasets such as Ego4D have inconsistent camera views and misaligned interactions, while video models struggle with accurate task-oriented hand-object interaction generation. TASTE-Rob introduces a large, instruction-aligned dataset and a three-stage pose-refinement pipeline. The authors report improved HOI video generation and robotic manipulation generalization, while noting degraded generation quality for objects that undergo significant transformations.
Problem
Ego4D's inconsistent camera views and misaligned interactions limit its suitability for precise imitation-learning demonstrations, and video models struggle with task-oriented HOI generation.
Source: §1 · p. 1, §4.2 · p. 6Method
TASTE-Rob provides 100,856 instruction-aligned egocentric videos, and a three-stage pipeline refines hand poses before regenerating videos.
Source: §1 · p. 2Results
The authors report that the dataset and pose-refinement pipeline enhance HOI video generation and improve robot manipulation generalization to novel scenes through imitation learning.
Source: §6 · p. 8Takeaways & Limitations
The work presents a diverse egocentric HOI dataset and pose-refinement pipeline for generating task-oriented videos and improving manipulation generalization.
Source: §6 · p. 8Generation quality degrades when objects undergo significant transformations, such as rotation and opening.
Source: §6 · p. 8
Abstract
from arXiv · showhide
We address key limitations in existing datasets and models for task-oriented hand-object interaction video generation, a critical approach of generating video demonstrations for robotic imitation learning. Current datasets, such as Ego4D, often suffer from inconsistent view perspectives and misaligned interactions, leading to reduced video quality and limiting their applicability for precise imitation learning tasks. Towards this end, we introduce TASTE-Rob -- a pioneering large-scale dataset of 100,856 ego-centric hand-object interaction videos. Each video is meticulously aligned with language instructions and recorded from a consistent camera viewpoint to ensure interaction clarity. By fine-tuning a Video Diffusion Model (VDM) on TASTE-Rob, we achieve realistic object interactions, though we observed occasional inconsistencies in hand grasping postures. To enhance realism, we introduce a three-stage pose-refinement pipeline that improves hand posture accuracy in generated videos. Our curated dataset, coupled with the specialized pose-refinement framework, provides notable performance gains in generating high-quality, task-oriented hand-object interaction videos, resulting in achieving superior generalizable robotic manipulation. The TASTE-Rob dataset is publicly available to foster further advancements in the field, TASTE-Rob dataset and source code will be made publicly available on our website https://taste-rob.github.io.
1. Introduction
Video demonstrations can help robots generalize beyond recorded environments, but existing robot datasets are limited and current video models struggle with accurate, task-oriented hand-object interactions. TASTE-Rob pairs a large, consistently recorded and instruction-aligned dataset with pose refinement to improve generated interactions and robotic manipulation.
1. Introduction
- Existing video models struggle to generate high-quality, task-oriented hand-object interaction videos with accurate task understanding.Robot-object interaction video generation is constrained by limited robot datasets, motivating the use of human hand-object interaction videos.Source: §1 · p. 1
- Ego4D's inconsistent camera perspectives and clips containing partial, overlapping interactions misalign videos with instructions and reduce generation quality.Ego4D contains 83,647 egocentric hand-object interaction videos.Source: §1 · p. 2
- TASTE-Rob contains 100,856 egocentric videos, each capturing one complete instruction-matched interaction from a fixed camera viewpoint.The collection is designed to support task-oriented video generation and reliable imitation-learning demonstrations.Source: §1 · p. 2
- A three-stage pipeline generates a coarse video, refines hand poses with a Motion Diffusion Model, then regenerates the video using those poses.The pipeline addresses unnatural grip changes that remain after fine-tuning a video diffusion model on TASTE-Rob.Source: §1 · p. 2
- The dataset and pose-refinement pipeline reportedly surpass state-of-the-art methods in generated interaction-video quality and achieve accurate robotic manipulation.The authors also describe the dataset and pipeline as contributions to generalizable robotic manipulation.Source: §1 · p. 2
2. Related Works
Prior work explores video demonstrations and generative models for robotic imitation learning, but robot video datasets are limited and effective hand-object interaction generation remains a challenge. This work combines video and motion diffusion models and introduces TASTE-Rob for task-oriented hand-object interaction video generation.
2. Related Works
- Video demonstrations, optical flows, and images have been used to improve generalization, but robot video generation is limited by small robot datasets and human-hand video generation lacks effective HOI models.Imitation-learning methods using demonstrations from environments identical to execution have limited generalization to novel scenes and tasks.Source: §2 · p. 2
- Existing egocentric HOI datasets have limitations in resolution, scale, or scene diversity; Ego4D's varied camera views and imprecise action-language alignment complicate its use for IL demonstrations.TASTE-Rob contains 100,856 videos specifically designed for task-oriented HOI video generation.Source: §2 · p. 2
- Video diffusion models generate realistic colors and temporally coherent videos but struggle with significant motion variation, which motion diffusion models address using motion-control parameters.The paper combines the models to generate HOI videos with realistic grasping poses.Source: §2 · pp. 2–3
3. Building TASTE-Rob Dataset
TASTE-Rob is a large-scale egocentric HOI dataset designed around single, instruction-aligned actions, fixed views, varied environments and tasks, and diverse hand poses. Its collection protocol and distinctive dataset features target the requirements of task-oriented video generation and imitation-learning demonstrations.
3. Building TASTE-Rob Dataset
- TASTE-Rob contains 100,856 video-instruction pairs, with each video recorded from a static perspective and containing a single action closely aligned with its instruction.The dataset is also intended to cover diverse environments, tasks, and hand poses.Source: §3 · p. 3
3.1. Collection Strategy and Camera Setting
- Collection uses fixed egocentric camera views and recordings under eight seconds, with collectors starting and stopping each clip around one instructed action.The camera perspective is aligned with the head-mounted setup of Ego4D.Source: §3.1 · p. 3
3.2. Data Diversity
- The dataset covers varied locations and tasks, including 75,389 single-hand videos and 25,467 double-hand videos.Tasks include picking, placing, pushing, and pouring; the collection also analyzes palm orientation and grasping-pose distributions.Source: §3.2 · pp. 3–4
3.3. Comparisons with Other Datasets
- TASTE-Rob is designed specifically for task-oriented HOI generation and distinguishes itself through fixed views, precise language-action alignment, and descriptive object determiners.The dataset is also proposed as a resource for imitation-learning demonstrations.Source: §3.3 · p. 4, Table 2 · p. 4
4. Method
The method combines a task-conditioned coarse video generator with hand-pose refinement and pose-conditioned video regeneration to target accurate task understanding and feasible hand-object interaction. Its diffusion-model components generate video latents and hand-motion sequences under task and environment conditioning.
4. Method
- The three stages target accurate object-and-task understanding and feasible, consistent grasping poses in the final HOI video.The coarse video supports task understanding; pose refinement and conditioned regeneration target improved grasping fidelity.Source: §4 · pp. 4–5
4.1. Preliminary: Diffusion Models
- The method uses diffusion models to denoise video latents conditioned on task text and an environment image, while MDM generates clean motion sequences.The preliminary describes denoising from Gaussian noise for diffusion models and clean-sequence generation for MDM.Source: §4.1 · p. 5
4.2. Stage I: A Coarse Action Planner
- Fine-tuned DynamiCrafter generates a coarse HOI video from the task description and environment image, but its grasping poses can vary inconsistently over time.Fine-tuning updates the image-context projector and spatial layers of the denoising U-Net.Source: §4.2 · pp. 5–6
4.3. Stage II: Revising Hand Pose Sequences
- An Image-to-Hand-Params MDM refines key-point sequences extracted from the coarse video, conditioning on task text and environment imagery.Refinement starts from the coarse sequence to combine its spatial awareness with physically plausible motion.Source: §4.3 · p. 6
4.4. Stage III: Regenerating with Refined Pose
- A frame-wise pose encoder injects visualized refined-hand-pose images into video features, while training freezes the video model and updates the encoder.The injected features are processed similarly to ControlNet.Source: §4.4 · p. 6
- Generated fine HOI videos used as imitation-learning demonstrations confirm their effectiveness for enabling robot manipulation.The manipulation policy is Im2Flow2Act.Source: §4.4 · p. 6
5. Experiments
Experiments evaluate video generation, grasping-pose consistency, and robot manipulation, comparing TASTE-Rob with existing video diffusion models and testing dataset and pose-refinement choices.
5.1. Implementation
- Evaluation uses TASTE-Rob-Test and 50 sampled MuJoCo tasks, with video quality, pose consistency, and robot-manipulation success as evaluation aspects.The implementation fine-tunes the coarse action planner, trains the MDM, and fine-tunes the pose encoder in three stages.Source: §5.2 · pp. 6–7, §5.1 · p. 6
5.3. Baselines and Comparisons.
- TASTE-Rob outperforms four general video diffusion model baselines on video generation quality and generalization.The baselines are DynamiCrafter, consistI2V, Open-Sora Plan, and CogVideoX; the authors report that these models fail to perform manipulation tasks well.Source: §5.3 · p. 7, Table 3 · p. 7
5.4. Ablation Study
- Coarse-TASTE-Rob achieves superior video-generation performance to Ego4D-Gen when the two planners are trained on different datasets.The comparison changes only the training dataset: Ego4D-Gen uses 83,647 Ego4D videos, while Coarse-TASTE-Rob uses TASTE-Rob.Source: §5.4 · p. 7, Table 4 · p. 7
- TASTE-Rob improves video generation, grasp-pose consistency, and robot manipulation over Coarse-TASTE-Rob across the reported metrics.The comparison also reports more precise target placement and higher manipulation success rates.Source: Table 5 · p. 7, §5.4 · pp. 7–8
- Increasing pose-refinement denoising steps improves pose consistency but degrades spatial awareness, while video quality first improves and then deteriorates.The authors select Nrv = 10 as a trade-off.Source: §5.4 · pp. 7–8
6. Discussion and Conclusion
The paper concludes that its dataset and three-stage pose-refinement pipeline improve task-oriented hand-object interaction video generation and robot-manipulation generalization, while noting a limitation under large object transformations.
6. Discussion and Conclusion
- The collected ego-centric HOI dataset and three-stage pose-refinement pipeline enhance video generation and manipulation generalization to novel scenes through imitation learning.Source: §6 · p. 8
- Generation quality degrades when objects undergo significant transformations, including rotation or opening.The authors identify addressing this limitation as future work.Source: §6 · p. 8
8. Details of TASTE-Rob dataset
TASTE-Rob addresses Ego4D's camera-viewpoint and action-language alignment issues and uses more specific instructions to identify targets and actions across diverse object scenarios.
8.1. Comparisons between Ego4D and TASTE-Rob
- Ego4D's seven head-mounted cameras produce varied viewpoints, and clips extracted from long recordings can include actions misaligned with their instructions.The paper gives an example of a person mixing food while the instruction says to put down a bowl.Source: §8.1 · p. 12
8.2. Design of Language Instruction in TASTE-Rob
- TASTE-Rob instructions use distinctive object attributes and spatial destinations to identify the target object and intended action more precisely.Examples include distinguishing a red bowl or a bowl beside a kettle, and specifying where to place an object.Source: §8.2 · p. 12
8.3. Diversity of Manipulated Objects in TASTE-Rob
- Manipulated objects vary by scenario, with kitchen utensils common in cooking activities and clothing in bed environments, alongside objects such as cups and phones across scenarios.Figure 11 visualizes object distributions across six scenes.Source: Figure 11 · p. 12, §8.3 · p. 13
9. Analyzing Details of hand poses
The hand-pose analysis uses MANO parameters and examines palm orientation, finger spread, and finger curvature distributions across sampled TASTE-Rob frames.
9.1. MANO parametric hand model
- HaMeR reconstructs 3D hand meshes from RGB images using MANO pose and shape parameters; the analysis samples 16 frames per video.Pose parameters comprise global orientation and articulation parameters for 15 hand joints.Source: §9.1 · p. 13
9.2. Calculating Details of analyzing Distribution
- Palm orientation is measured by projecting the palm normal onto the frame plane, measuring its angle from the positive X-axis, and binning the normalized angle into four 90-degree intervals.Source: §9.2 · p. 13
- Finger-spread distributions use angles between thumb-index and index-middle directions, while curvature distributions use bending angles between adjacent finger-joint segments.Both sets of angles are plotted as probability-density curves.Source: §9.2 · pp. 13–14
- Figure 14 provides additional video-generation comparisons between TASTE-Rob and CogVideoX.Source: Figure 14 · p. 14
10. Our Introduced Metrics
The introduced metrics assess grasp-pose consistency, grasp-type agreement, and hand-movement direction. GPV visualizations show more consistent grasping after pose refinement than with the coarse action planner.
10. Our Introduced Metrics
- GPV measures variance in hand-pose parameters across frames with grasping poses; higher values indicate less grasp-pose consistency.For two-hand videos, GPV is computed independently for each hand and then averaged.Source: §10 · p. 14, Figure 15 · p. 14
- Figure 16 contrasts no refinement (Nrv = 0) with direct pose-sequence generation from Gaussian noise (Nrv = 50).The figure visualizes different pose-refinement denoising steps.Source: Figure 16 · p. 14
- Comparing two generated videos, the complete pose-refinement pipeline maintains consistent grasping poses, while the coarse action planner’s poses vary significantly from pinching to holding, yielding a higher GPV value.Each generated video consists of L = 16 frames.Source: §10 · p. 14
- GTCE evaluates generated poses by comparing their grasp-type classifications against ground truth across 8 manipulation frames and 17 grasping types.HaMeR predicts pose parameters before classification.Source: §10 · p. 15
- HMDA measures the average angle difference between generated and ground-truth hand movement vectors derived from consecutive-frame keypoints.The keypoints are extracted with MediaPipe.Source: §10 · p. 15
11. More Qualitative Results
The additional results show generated hand-object interaction videos across environments and tasks, and report accurate robot manipulation in simulation. Pose refinement uses a 10-step denoising process at inference.
11. More Qualitative Results
- At inference, pose refinement uses a 10-step denoising process starting from coarse generated hand-pose sequences.Figure 16 visualizes different denoising steps.Source: §11 · p. 15
- TASTE-Rob generates high-quality hand-object interaction videos across various environments and tasks.The paper also provides corresponding videos in the supplementary materials.Source: §11 · p. 15
- TASTE-Rob generates high-quality hand-object interaction videos on a robot simulation platform and achieves accurate robot manipulation.Figure 17 shows examples of video generation and manipulation in simulation.Source: §11 · p. 15
12. More Implementation Details
The implementation uses full-video sampling for the coarse action planner, an MDM to revise pose sequences, and a frame-wise adapter to regenerate videos from refined poses.
12.1. Stage I: A Coarse Action Planner
- Stage I fine-tunes DynamiCrafter and samples 16 frames across the entire video to preserve complete actions.Fine-tuning uses a batch size of 16, learning rate 5×10^-5, and 30K steps; inference uses a 50-step DDIM sampler.Source: §12.1 · pp. 15–16
12.2. Stage II: Revising Hand Pose Sequences
- Stage II trains an MDM on 60-frame MediaPipe hand-pose coordinates conditioned on environment images and language instructions, then uses it to refine coarse pose sequences.Training runs for 500K steps; inference uses 50-step DDIM generation from Gaussian noise and 10-step refinement from coarse poses.Source: §12.2 · p. 16
12.3. Stage III: Regenerating with Refined Pose
- Stage III fine-tunes a Stable Diffusion 2 frame-wise adapter to regenerate videos conditioned on refined-pose images, environment images, and language instructions.Training uses 16-frame visualized hand-pose images; inference applies 50 denoising steps.Source: §12.3 · p. 16