PaperlayerView exact revision on arXiv

Source-linked AI summary

One-shot Human Video-Guided Adaptation for Vision-Language-Action Models

Qi Chen, Yuwei Wu, Quan Chen, Kunpeng Gao, Jin Chen, Che Sun, Zhi Gao, Yunde Jia

arXiv:2610.19746v1cs.RO

TL;DR

Rapid adaptation of VLA models to unseen manipulation tasks remains limited, while human videos offer task-specific information about how tasks are carried out. CapPro uses one human demonstration to provide phase-aware guidance through a pretrained policy's conditioning context. It improves average performance across unseen target-task groups while maintaining comparable source-task performance.

  • Problem

    VLA models remain limited in rapidly adapting to manipulation tasks beyond their training distributions, and human videos offer task-specific information about interaction patterns and task execution.

    Source: §I · p. 1

  • Method

    CapPro extracts phase-aware representations from a single human video and incorporates them into the conditioning context of a pretrained VLA policy as task-specific guidance.

    Source: §I · pp. 1–2

  • Results

    CapPro improves average success rate across unseen target-task groups while maintaining comparable performance on source tasks, and real-world experiments demonstrate cross-task adaptation.

    Source: §I · p. 2

  • Takeaways & Limitations

    RoboTwin results indicate that effective human-video-guided adaptation uses structured representation and integration rather than simply concatenating video tokens into the policy prefix.

    Source: §V · p. 6

    Performance remains limited on tasks with large object, interaction, and compositional shifts.

    Source: §V · p. 6

Abstract

from arXiv · show

Vision-language-action (VLA) models have recently achieved remarkable progress by leveraging large-scale robot datasets and foundation models for general-purpose robot manipulation. However, their ability to rapidly adapt to novel manipulation tasks beyond the training distribution remains limited. Existing approaches often require task-specific robot demonstrations and additional training, which are labor- and compute-intensive, limiting the scalability of VLA adaptation to open-world scenarios. To reduce this dependence on target-task robot data, we propose CapPro, a framework that enables a VLA policy to adapt to previously unseen manipulation tasks using only a single human demonstration video. CapPro consists of two components. First, a phase-aware human video representation module constructs a compact video representation through learned visual token selection and learns task-relevant query representations under phase-caption supervision. Second, a video-conditioned adaptation module jointly incorporates the compact video and query-derived representations into the action-conditioning context of the pretrained VLA policy. % This design introduces task-specific guidance while largely preserving the original VLA action-generation backbone. In this way, the human demonstration provides task-specific execution guidance rather than an embodiment-aligned trajectory to imitate, without requiring target-task robot demonstrations or target-task fine-tuning. Experiments across source and unseen target tasks show that CapPro improves average performance across unseen target groups while maintaining comparable performance on the source tasks. These results demonstrate the effectiveness of human-video guidance for cross-task VLA adaptation.

I. INTRODUCTION

CapPro uses a single human demonstration video to guide a pretrained VLA policy on unseen manipulation tasks, addressing the need for task-specific knowledge beyond pretraining without target-task robot demonstrations or fine-tuning. The introduction motivates this approach by noting both the difficulty of selecting useful information across video phases and the gap between human demonstrations and robot execution.

I. INTRODUCTION

  • VLA models have shown broad manipulation capabilities, but their rapid adaptation to tasks beyond their training distributions remains limited.Source: §I · p. 1
  • Existing adaptation often relies on robot demonstrations and task-specific fine-tuning, which add substantial data and computational requirements.Source: §I · p. 1
  • Human videos can convey interaction patterns, execution procedures, and goals without requiring the task to be demonstrated on a specific robot platform.Source: §I · p. 1
  • Human demonstrations are visually redundant, convey different information at different execution stages, and differ from robot execution in embodiment, viewpoint, and action space.Source: §I · p. 1
  • CapPro improves average success rate across unseen target-task groups while maintaining comparable performance on source tasks, with real-world experiments demonstrating cross-task adaptation.The evaluation covers 26 unseen RoboTwin tasks organized into three groups with increasing task shift.Source: §I · p. 2
  • CapPro uses a single human demonstration as task-specific execution guidance, without target-task robot demonstrations or target-task fine-tuning.It combines phase-aware video representations with video-conditioned adaptation rather than directly transferring human behavior to robot actions.Source: §I · pp. 1–2

II. RELATED WORK

The related work distinguishes CapPro from pretrained VLA development, robot-policy adaptation, and learning from human videos. CapPro uses one human demonstration to guide a pretrained VLA policy without target-task robot demonstrations, fine-tuning, or explicit trajectory conversion.

A. Vision-Language-Action Models

  • Pretrained VLA research expands manipulation capabilities through large-scale pretraining, whereas this work studies acquiring task-specific knowledge at deployment from a single human video.Source: A. Vision-Language-Action Models · p. 2

B. Robot Adaptation

  • Robot adaptation methods use demonstrations, retrieved trajectories, skill representations, specialized modules, or deployment-time optimization, but task-specific information commonly still comes from robot demonstrations or additional optimization.Source: B. Robot Adaptation · p. 2

C. Learning from Human Videos

  • Human-video approaches translate motion into robot-compatible trajectories or extract higher-level representations; CapPro instead conditions a pretrained policy without explicit trajectory conversion or target-task robot demonstrations.Source: C. Learning from Human Videos · p. 2

III. METHOD

CapPro frames one-shot adaptation as using a human video for task-specific guidance rather than direct imitation. Its two modules extract execution information in the context of the current robot observation and incorporate that information into the pretrained VLA policy; phase-caption supervision is used during training, not inference.

A. Problem Formulation

  • At deployment, the policy receives one human video of an unseen target task that provides execution information but contains no robot actions or embodiment-aligned trajectory; no target-task robot demonstrations or fine-tuning are used.Source: A. Problem Formulation · p. 2

B. Framework Overview

  • The phase-aware representation module extracts task-relevant information from the human video in the context of the current robot observation, and the adaptation module adds it to the pretrained policy's conditioning context.Phase-level textual supervision is an auxiliary training signal; phase captions are not required at inference.Source: B. Framework Overview · p. 3, Figure 1 · p. 3

C. Phase-Aware Human Video Representation

The phase-aware representation module creates compact human-video guidance and observation-conditioned query representations. It uses phase-caption supervision during training to encode execution-stage information, while phase captions and robot progress are not required at inference.

b) Phase-Aware Context Representation

  • The module outputs compact video representation ZH and observation-conditioned query representation SH_t as guidance for video-conditioned adaptation.At inference, SH_t is computed directly from the human video and current robot observation without phase captions, teacher-forcing tokens, or robot execution progress.Source: C. Phase-Aware Human Video… · pp. 3–4

a) Compact Human Video Representation

  • Learnable spatial selection extracts informative visual features from uniformly sampled human-video frames, and video-progress embeddings preserve their temporal ordering.The resulting compact representation retains visual and temporal information from the demonstration.Source: C. Phase-Aware Human Video… · p. 4

b) Phase-Aware Context Representation

  • Task-specific phase captions supervise execution stages, with shared descriptions but episode-specific boundaries to account for different execution rates.Robot execution progress selects the phase caption during training and is never provided to the policy.Source: C. Phase-Aware Human Video… · p. 4
  • Learnable queries interact with human-video and robot visual context under a mask that prevents them from attending to teacher-forcing caption tokens.Caption prediction is conditioned on the query representations and preceding caption tokens, avoiding direct exposure of the target caption to the queries.Source: C. Phase-Aware Human Video… · p. 4

D. Video-Conditioned Adaptation

The adaptation module combines compact video and observation-conditioned query representations with the original conditioning context to form the action prefix. This adds human-video task guidance while retaining the base VLA policy's action-generation mechanism.

D. Video-Conditioned Adaptation

  • The module incorporates compact human-video and context-dependent query representations into the action-conditioning context of the policy.A learnable gate adaptively scales query-derived guidance before it enters the action prefix.Source: D. Video-Conditioned Adaptation · p. 4
  • The augmented prefix is used for action generation, introducing human-demonstration guidance while retaining the base VLA policy's action-generation mechanism.The original context includes robot-observation representation and task/state representation constructed from the language instruction and robot state.Source: D. Video-Conditioned Adaptation · pp. 4–5

E. Training Objective

CapPro jointly optimizes robot action generation under human-video conditioning and an auxiliary phase-caption objective. The objectives train action generation and execution-stage-aware query representations, respectively.

Training Objective

  • Joint optimization combines the action-generation objective with auxiliary phase-caption supervision for learning execution-aware query representations.The action objective trains robot action generation under human-video conditioning.Source: E. Training Objective · p. 5

a) Action Generation Objective

  • The action-generation objective retains the base VLA policy's flow-matching action objective and conditions it on the video-augmented prefix.It uses a ground-truth action chunk, Gaussian noise, and flow time to form the noisy action state.Source: E. Training Objective · p. 5

b) Phase-Caption Objective

  • The auxiliary phase-caption objective trains the model to predict caption tokens for the phase assigned to each training sample.This objective encourages query representations to encode execution-stage-aware information.Source: E. Training Objective · p. 5

A. Experimental Setup

The simulation setup uses RoboTwin source and target splits to assess one-shot adaptation under different task shifts, alongside a real-world evaluation. Target policies receive one human video at inference, without target-task robot demonstrations or fine-tuning.

A. Experimental Setup

  • The four RoboTwin splits contain 39 tasks: 13 Source, 6 Target-A1, 10 Target-A2, and 10 Target-B.Source tasks train the policy; the target splits represent near-task variations, larger object, interaction, or affordance changes, and compositional manipulation, respectively.Source: A. Experimental Setup · p. 5
  • For each target task, the policy receives one egocentric human video at inference, with no target-task robot demonstrations or fine-tuning.The video provides task-level guidance rather than a direct robot demonstration.Source: A. Experimental Setup · p. 5
  • Simulation tasks use 100 rollout episodes per clean and randomized setting, with task success rate as the reported metric.The matched π0.5 baseline and Direct Video Prefix baseline are included in the evaluation.Source: A. Experimental Setup · p. 5
  • The real-world evaluation covers seven tasks: three Source, two Target-A, and two Target-B tasks, with five trials per task.Trials use predefined task-specific initializations and the same conditions for CapPro and matched π0.5; success requires completing the full task objective.Source: A. Experimental Setup · p. 5

B. Implementation Details

CapPro is initialized from pretrained π0.5 parameters and trained with LoRA and AdamW under a cosine learning-rate schedule.

B. Implementation Details

C. Main Results

CapPro improves success rates over matched π0.5 across RoboTwin task groups, while directly concatenating human-video tokens performs worse. Across all 39 tasks, it achieves the highest reported overall average.

C. Main Results

  • Direct Video Prefix lowers overall average success from 28.4% to 19.1% relative to matched π0.5, with similar degradation on Target-A2 and Target-B.This comparison indicates that simply adding human-video context is insufficient for effective cross-task adaptation.Source: C. Main Results · p. 6
  • CapPro improves over matched π0.5 in every task group: Source 68.9% vs 64.4%, Target-A1 44.8% vs 34.5%, Target-A2 29.0% vs 2.5%, and Target-B 20.7% vs 4.0%.The gains are particularly pronounced on Target-A2 and Target-B, which represent larger task shifts.Source: C. Main Results · p. 6
  • Across all 39 tasks, CapPro reaches 42.6% average success, versus 28.4% for matched π0.5 and 19.1% for Direct Video Prefix.The reported improvements are 14.2 and 23.5 percentage points, respectively.Source: C. Main Results · p. 6

D. Ablation Studies

The ablations vary individual CapPro components under a controlled training setup; the conclusion presents CapPro’s cross-task results and notes that performance remains limited under larger shifts.

D. Ablation Studies

  • The ablations modify one component at a time while holding the pretrained model, source data, optimization settings, and training budget fixed.Variants cover GridS sampling, phase-aware captioning, caption loss, query-to-action guidance, human-video progress, and the direct video path.Source: D. Ablation Studies · p. 6

V. CONCLUSION

  • CapPro uses one egocentric human video as task-level guidance rather than directly imitating human motion or requiring target-task robot demonstrations.Its phase-aware representation and video-conditioned adaptation extract compact task-relevant visual and semantic information while preserving the original action-generation capability.Source: §V · p. 6
  • On RoboTwin, CapPro improves over matched π0.5 across source and target splits, whereas directly concatenating human-video tokens performs substantially worse.The conclusion attributes effective guidance to structured representation and integration rather than simply adding visual context.Source: §V · p. 6
  • Performance remains limited on tasks with large object, interaction, and compositional shifts.The authors identify stronger representations and adaptation mechanisms for challenging shifts and broader real-world deployment as future work.Source: §V · p. 6
Loading 2610.19746v1…