Source-linked AI summary
YUBI-STAG: Contact and Semantic-Rich Alignment for VLAs via Automated Video-Language Grounding
Masatoshi Tateno, Takehiko Ohkawa, Yueh-Hua Wu, Hanlong Li, Tatsuya Matsushima, Yoichi Sato, Kei Ota
TL;DR
Coarse robot-task labels omit how grippers interact with objects, limiting the fine-grained language available to elicit pretrained VLA capabilities. YUBI-STAG and its distilled YUBI-VLM recover contact-grounded annotations from demonstrations, and post-training with these annotations improves fine-grained manipulation, language following, and compositional generalization.
Problem
Robot demonstrations often repeat coarse task-level labels that omit gripper actions, object contacts, and other execution details needed for fine-grained language alignment.
Source: §1 · pp. 1–2Method
YUBI-STAG combines contact-object segmentation with staged VLM annotation, and YUBI-VLM distills this capability to annotate raw, unsegmented wrist-view video.
Source: §1 · p. 2, (ii) VLM-based annotation pipeline · p. 5Results
Post-training with rich annotations improves fine-grained manipulation, language following, and compositional manipulation beyond the original coarse labels.
Source: §1 · p. 3, §7 · p. 12Takeaways & Limitations
Contact-aware annotations add control over object identity, acting gripper, target location, and spatial relations absent from the original labels.
Source: §1 · p. 3
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models acquire broad manipulation capabilities via large-scale pretraining, yet eliciting them through language requires fine-grained alignment between instructions and physical interactions. Existing robot demonstrations typically provide only coarse task descriptions, omitting how actions are executed, including which gripper acts, which object is contacted, and how it is grasped and moved. We introduce YUBI-STAG, a framework for Spatio-Temporal Annotation and Grounding that automatically enriches manipulation demonstrations with interaction-rich semantics to align pretrained VLAs with fine-grained manipulation language. Combining contact-object segmentation with vision-language models, YUBI-STAG annotates object identities, attributes and states, per-gripper actions, bimanual coordination, and spatially grounded interactions. To address YUBI-STAG's reliance on localized sequences and multi-stage VLM inference, we distill it into YUBI-VLM. YUBI-VLM directly recovers action structure and annotations from raw, unsegmented video in few inference calls and operates from wrist views alone. We evaluate both frameworks on YUBI-STAG-Bench across temporal, semantic, and spatial grounding tasks. YUBI-VLM retains much of YUBI-STAG's annotation accuracy with fewer inference calls and shorter runtime while generalizing to unseen manipulations. Finally, post-training VLA policies on these annotations aligns them with fine-grained language and contact-aware structure. Bimanual experiments demonstrate improved performance and instruction following, including control over object identity, acting gripper, target location, and spatial relations absent from original labels.
1. Introduction
Coarse task labels omit the interaction details needed to align pretrained VLAs with fine-grained manipulation language. YUBI-STAG and YUBI-VLM recover contact-grounded annotations, and post-training with these annotations improves fine-grained manipulation and language following.
Motivation
- Coarse episode labels omit which gripper acts, which object it contacts, and how it is grasped, handed over, or released.This matters because language is the channel through which pretrained VLA capabilities are elicited, while datasets often repeat one task-level label across demonstrations.Source: §1 · pp. 1–2
Approach
- YUBI-STAG uses contact-object segmentation and staged VLM annotation to recover object identities, per-gripper actions, bimanual coordination, and state changes; YUBI-VLM annotates raw, unsegmented wrist-view video.YUBI-VLM is distilled from YUBI-STAG to reduce reliance on localized sequences and multiple model calls.Source: §1 · p. 2
Experiments
- Bimanual Franka experiments report improved fine-grained language following, contact-robust manipulation, and compositional generalization after alignment with rich annotations.The annotations include object identities, attributes, target locations, spatial relations, handedness, and contact phases.Source: §1 · p. 3
Contributions
- The paper introduces YUBI-STAG-Bench to evaluate structured annotation quality on unseen bimanual tasks and a contact- and semantic-rich post-training recipe for a π0.5-based policy.The post-training recipe adds handedness, contact phases, and contact-aware sampling.Source: §1 · p. 3
2. Related Work
Scalable manipulation data collection increases the need for annotation of noisy, unsegmented demonstrations. Prior annotation approaches leave gaps in visual-temporal contact grounding, while VLA post-training efforts lack explicit contact-transition and coordination supervision.
2.1. Scalable Collection of Unconstrained Manipulation Data
- In-the-wild, unsegmented demonstrations add clutter, camera motion, and operator variation, making curation and fine-grained annotation a bottleneck for scalable robot learning.Low-cost bimanual systems, teleoperation, and handheld grippers have enabled collection of diverse demonstrations at scale.Source: §2.1 · p. 3
2.2. Video-Language Grounding and Automated Data Annotation
- Language relabeling and image-grounded boxes, traces, or contact points do not jointly recover gripper-object contact in pixels and time; proprioceptive phase inference can miss non-grasping contacts.YUBI-STAG detects contact visually and recovers per-gripper contact intervals, object masks, and coordination from moving wrist views; YUBI-VLM annotates raw wrist-only video.Source: §2.2 · p. 3
2.3. Data-Centric Post-Training Alignment in VLA Policies
- Existing VLA alignment efforts optimize preferences, curate demonstrations, or pair trajectories with fine-grained instructions, but provide no explicit supervision at contact transitions or bimanual coordination moments.Source: §2.3 · pp. 3–4
3. Spatio-Temporal Interaction Annotation
YUBI-STAG grounds structured annotations in contact tracks, while YUBI-VLM distills this capability for raw wrist-view video. The pipeline resolves scene context, action structure, coordination, and object changes before producing grounded descriptions.
3. Spatio-Temporal Interaction Annotation
- YUBI-STAG enriches temporally localized primitive-action labels with grounded structure, while YUBI-VLM distills the pipeline to annotate raw videos at lower cost and support wrist-only inference.Primitive actions are composite manipulation units whose constituent actions are called PA stages.Source: §3 · p. 4, §3.1 · p. 4
(ii) VLM-based annotation pipeline
- Contact tracks anchor four annotation stages: scene inventory with persistent object IDs, PA stages with per-gripper actions and coordination, object changes, and unified grounded PA descriptions.The pipeline uses appearance and temporal continuity to stabilize object identities, then compares objects before contact and after release to record spatial and state changes.Source: Figure 2 · p. 4, (ii) VLM-based annotation pipeline · p. 5
3.1. YUBI-STAG
- Contact-object segmentation visually infers contact intervals and contacted-object masks, including non-grasping contacts that gripper-aperture proxies can miss.The segmentation module predicts contact state and pixel-level object masks from wrist-camera streams.Source: §3.1 · p. 4
3.2. Training YUBI-VLM
- YUBI-VLM is trained to predict ten annotation modes from YUBI-STAG annotations, with wrist-frame sampling concentrated around contact intervals.Training uses 2,477 sessions from 129 long-horizon tasks, and excludes all ten benchmark tasks to test generalization to unseen tasks.Source: §3.2 · pp. 5–6
4. YUBI-STAG-Bench
YUBI-STAG-Bench evaluates structured annotation for long-horizon manipulation, covering contact grounding and semantic understanding from video, task descriptions, and contact tracks.
Benchmark construction
- YUBI-STAG-Bench contains 100 manually annotated episodes from 10 tasks, with annotations following the YUBI-STAG schema.The benchmark includes 510 objects, 970 primitive actions, and 2,201 interaction intervals.Source: §4 · p. 6
Contact-object grounding
- The benchmark evaluates contact-interval detection and contacted-object segmentation from wrist video, with masks and temporal intervals annotated at 10 fps.Source: §4 · p. 6
Semantic annotation
- Semantic tasks evaluate object identity and attributes, PA localization and generation, bimanual coordination, spatial relations, and object states and transition times.Inputs include video, task description, and contact tracks.Source: §4 · p. 6
5. Training VLA Policies
The policy keeps its pretrained architecture and action setup while post-training adds object- and location-aware instructions, time-varying gripper labels, and contact phases. Training also prioritizes contact-transition chunks without changing the objective.
5. Training VLA Policies
- The π0.5-based policy is pretrained on 8,434 hours of YUBI data and post-trained on the same 10–20 hours of demonstrations under different supervision.The ablations differ only in supervision.Source: §5 · p. 6
5.1. YUBI VLA and Action Representation
- At each step, the policy uses two wrist images, language, and proprioception to predict a horizon of bimanual actions with 6-DoF end-effector motion and gripper commands.The action expert uses conditional flow matching, following the standard π0.5 recipe without changing the network or objective.Source: §5.1 · p. 6
5.2. Contact- and Semantic-Rich Post-Training
- Primitive-action instructions are enriched with task-relevant object attributes, identities, target locations, and spatial relations.For part sorting, descriptions specify the part’s color and destination bin cell.Source: §5.2 · pp. 6–7
- Handedness labels identify the acting gripper or handover direction, while contact-phase labels mark approach, grasp, hold, or release over time.Grasp covers ±0.2 seconds around contact onset, and release covers the final 0.2 seconds of a contact segment.Source: §5.2 · p. 7
- Post-training keeps observations, actions, architecture, and objective fixed while adding supervision for interaction semantics and temporal contact structure.Contact-prioritized sampling increases the frequency of contact phases in training chunks.Source: §5.2 · p. 7
- Contact-prioritized sampling raises the representation of sparse transitions, such as sampling a grasp-onset phase from 5% of demonstrations as 30% of chunks.The annotations also support curation of instruction–trajectory mismatches and more diverse language descriptions.Source: §5.2 · p. 7
6. Experiments
Across annotation benchmarks and robot tasks, YUBI-STAG and YUBI-VLM support richer temporal, semantic, and spatial grounding than coarse labels alone. Fine-grained supervision improves manipulation and language control, while generated primitive actions support composition on unseen tasks.
6.1. Results on YUBI-STAG-Bench
- With matched PA input, YUBI-VLM preserves most YUBI-STAG capabilities and matches or exceeds it on several categories, though YUBI-STAG remains stronger on attributes and states.YUBI-STAG also faces challenges in appearance, relative location, and state changes when visual details are missing from the views.Source: §6.1 · p. 8, Table 2 · p. 9
- YUBI-VLM substantially improves temporal, coordination, spatial, and state understanding over the untrained backbone, recovers PAs from raw video, and retains most performance using wrist views alone.Relative-location and state-timing drops indicate the value of the top view for global context; RoboFine-VLM remains well below YUBI-VLM and tends to over-segment PAs.Source: §6.1 · pp. 8–9
6.2. Results on Fine-grained Manipulation
- Contact-aware supervision raises full success from 25% to 60% on YUBI part sorting and from 20% to 40% on fuel-bracket picking; neither coarse-label baseline completes either task.On fuel-bracket picking, pickup success rises from 35% to 55%; the authors report sharper motions under fine-grained supervision.Source: Figure 4 · p. 9, §6.2 · p. 10
6.1. Results on YUBI-STAG-Bench
- YUBI-VLM processes 100 episodes in less than half YUBI-STAG’s wall-clock time, using 33.0–34.1 versus 154 VLM requests per episode.It generates 3.3–3.4k versus 14.6k output tokens per episode and reaches 3.5–3.7× real-time throughput.Source: §6.1 · p. 9
6.4. Action Composition from YUBI-VLM-Generated PAs
- YUBI-VLM-generated PA supervision improves the action-success funnel on BED and enables task-level placement and completion never achieved by the task-level baseline; gains also transfer to unseen BUS.On BUS, it enables successful placement and completion of individual actions that the baseline never achieves.Source: Figure 7 · p. 11, §6.4 · p. 12
6.3. Fine-grained Language Steerability
- YUBI-STAG improves color accuracy to 82.9% from 46.4% and follows specified grippers in all completed trials; it also largely retains target-bin accuracy on unseen bin orders.The baseline’s target-bin accuracy falls from 100% to 53.3% on unseen orders.Source: §6.3 · p. 11, Figure 6 · p. 11
- YUBI-STAG achieves 50.0% correct relational placement versus 11.1% for the baseline, and 41.7% exact ordinal selection versus 12.5%.For ordinal selection, 91.7% of YUBI-STAG selections are within one position of the target; its MAE is 0.67 positions versus 2.26 for the baseline.Source: §6.3 · p. 11, Figure 6 · p. 11
- YUBI-STAG enables language control over semantic, embodiment, and spatial concepts absent from the original coarse PAs, with remaining color and spatial errors generally structured or near the intended target.The reported color errors reflect annotation confusions, while spatial errors include reference-color confusion, left/right reversal, or placement on the reference.Source: §6.3 · p. 11
7. Conclusions
YUBI-STAG annotates bimanual demonstrations with contact-anchored spatio-temporal structure, while YUBI-VLM directly annotates raw, wrist-only video. Post-training VLAs with these annotations improves fine-grained language following and manipulation beyond coarse labels.
7. Conclusions
- YUBI-STAG provides contact-anchored spatio-temporal annotation of bimanual demonstrations, and YUBI-VLM directly annotates raw, wrist-only video with capability validated on unseen manipulations.Post-training YUBI-pretrained VLAs with these annotations improves fine-grained language following and compositional manipulation beyond original coarse labels.Source: §7 · p. 12