PaperlayerView exact revision on arXiv

Source-linked AI summary

Frame-Level Temporal Alignment for Human-to-Robot Visual Adaptation

Xizhe Zhang, Jingfeng Zhang, Zirun Zhou, Hong Jia

arXiv:2610.04372v1cs.ROcs.LG

TL;DR

Human and robot demonstrations can differ in execution rate and non-key-frame proportion, making same-time frames represent different task stages. The paper uses two temporal priors to learn frame correspondences without annotations; it reports higher success across encoder initializations and improved real-world results.

  • Problem

    Paired human and robot videos can have different temporal structures, so frames sampled at the same relative time may represent different task progress and video-level alignment does not establish frame correspondences.

    Source: §1 · pp. 1–2

  • Method

    The framework adapts pretrained visual encoders using global progress and local temporal order priors to learn frame correspondences without frame-level annotations.

    Source: §1 · p. 2

  • Results

    The method achieves higher zero-shot success than HRP and HRAlign across four encoder initializations; with D4R-ViT, it reports a 65.96% relative improvement over HRP.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Updating 53.1K normalization-affine parameters achieves higher success rates than the late-adapter variant despite using approximately 29.8× fewer trainable parameters.

    Source: §1 · p. 3

    With 40 demonstrations per task and a single camera view, training data may not cover deployment object positions, and small or occluded objects can contribute to grasping failures.

    Source: §5 · p. 10

Abstract

from arXiv · show

Transferring visual representations pretrained on human videos to robot manipulation requires learning reliable correspondences between human and robot demonstrations. However, paired demonstrations can differ in execution rate and in the proportion of non-key frames that do not directly reflect task progress. Frames at the same relative timestamp may therefore represent different task stages, which can cause correspondence learning to fail. To address these issues, we propose Frame-Level Temporal Alignment (FLTA), a framework that uses two temporal priors to adapt visual encoders pretrained on human videos for robot manipulation. It learns shared task-progress representations without frame-level correspondence annotations, allowing frames at different relative temporal positions to match. A global progress prior combines normalized temporal positions with visual similarity to construct a soft correspondence target. A local temporal order prior penalizes backward transitions while allowing stays and varying forward rates to accommodate execution-rate differences. With ResNet-50 and ViT encoders, our method achieves relative improvements of 46.93% and 65.96%, respectively, over the best baselines in average simulation success rates and achieves higher task success rates on real-world manipulation tasks. These results also suggest that effective human-robot adaptation depends less on the number of parameters updated than on which parameters are selected. Our project page is available at https://rtx5090ultra.github.io/FLTA-Project-Page/.

1 Introduction

Human videos offer a scalable source of task-relevant visual experience, but human–robot domain and timing differences make frame-level correspondence important. The paper introduces temporal alignment to learn those correspondences without frame-level annotations.

1 Introduction

  • Human videos provide a relatively inexpensive, widely available source of manipulation experience that can support transferable visual representations for downstream policy learning.Robot demonstrations are costly to collect and often cover a limited range of tasks.Source: §1 · p. 1
  • Frames at the same relative time can show different task stages because paired videos differ in non-key-frame proportions and execution rates.Video-level contrastive alignment brings representations together but does not explicitly identify corresponding human and robot frames.Source: §1 · pp. 1–2
  • The framework combines a global progress prior and visual similarity for soft correspondences with a local order prior that penalizes backward matches while allowing stays and forward transitions.Adaptation updates only normalization-layer affine parameters, without frame-level correspondence annotations.Source: §1 · p. 2
  • 48.27% relative improvement in average success rate over HRAlign with R3M-RN50 initialization; D4R-ViT achieves a 65.96% relative improvement over HRP.With D4R-ViT, success rates also improve on all four real-world tasks compared with Frozen and HRP.Source: §1 · pp. 2–3

2 Human-to-Robot Domain Adaptation

The section frames human-to-robot adaptation as adapting a human-video-pretrained encoder with paired demonstrations that lack frame-level correspondence annotations. Its alignment framework combines visual similarity with global progress and local temporal-order priors before freezing the adapted encoder for policy learning.

2 Human-to-Robot Domain Adaptation

  • Frame-level temporal alignment adapts a pretrained visual encoder using paired human–robot videos without frame-level correspondence annotations, then freezes it for downstream policy learning.The adaptation addresses the human-to-robot domain gap in demonstrations with different temporal structures.Source: §2 · p. 3

2.1 Problem Formulation

  • The downstream policy uses the adapted encoder’s representation of robot observations as its visual state input for predicting actions.The formulation describes the encoder as first pretrained on human videos and then adapted using the paired demonstration dataset.Source: §2.1 · p. 3

2.2 Framework Overview

  • The framework learns frame-level temporal correspondences using visual similarity, a global progress prior, a local temporal order prior, and auxiliary frame discrimination.These components guide cross-domain frame matching.Source: §2.2 · p. 4

2.3 Frame-Level Temporal Alignment

  • At each iteration, the method samples eight ordered frames by default from each video and computes cross-domain visual similarities between their representations.The sampled frames include non-key frames; the temporal indices refer to positions in the sampled sequences.Source: §2.3 · p. 4
  • The global progress prior combines normalized sampled-position distance with visual matching cost, then Sinkhorn normalization converts the combined costs into correspondence weights.Larger differences between sampled frame indices incur higher costs; ρ controls the prior’s weight and larger ϵ values spread weights more evenly.Source: §2.3 · p. 4
  • The local temporal order prior penalizes backward correspondence transitions, with larger backward distances receiving higher penalties, while allowing stays and forward moves.The training loss matches predicted probabilities to the soft target, discourages backward transitions, and averages losses in both alignment directions.Source: §2.3 · pp. 4–5
  • An auxiliary InfoNCE loss uses augmented views of the same frame as positives and other same-domain batch frames as negatives; training updates only normalization-layer affine parameters.The total objective weights the InfoNCE and temporal-alignment losses, while other encoder parameters remain fixed.Source: §2.3 · p. 5

3 Experiments

The experiments evaluate manipulation success, human-to-robot task-progress retrieval, temporal-prior contributions, and adaptation under different data budgets and strategies. Results favor frame-level alignment across simulation, real-world tasks, and retrieval settings.

3 Experiments

  • Simulation uses 18 training task categories and 23 unseen evaluation categories; real-world evaluation covers four WidowX tasks with 40 training demonstrations per task.The simulation evaluates L1 and L2 generalization, while real-world policies use ACT with the evaluated frozen encoder.Source: §3 · p. 6

3.1 Downstream Language-Conditioned Policy Evaluation

  • 65.96% over HRP with D4R-ViT and 48.27% over HRAlign with R3M-RN50; the method has the highest average success rates on L1 and L2 across all four initializations.The evaluation covers 23 unseen AGNOSTOS tasks.Source: §3.1 · p. 6, Table 1 · p. 7

3.2 Human-to-Robot Task-Progress Retrieval

  • Our method achieves the most correct task-progress retrievals with both encoders in both settings, with gains of 22 and 21 queries on RH20T and 9 and 18 on self-collected data.Each gain is relative to HRP with D4R-ViT and HRAlign with R3M-RN50, respectively; each setting uses 100 queries.Source: Table 3 · p. 7, §3.2 · p. 8

3.1 Downstream Language-Conditioned Policy Evaluation

  • 56.25% average real-world success with D4R-ViT and 40 demonstrations per task is a 73.08% relative improvement over HRP.Success rates improve over Frozen and HRP on all four tasks, though lid opening remains challenging with a single diagonal camera and partial lid occlusion.Source: Table 2 · p. 7, §3.1 · p. 7

3.3 Ablation Studies

  • Removing both temporal priors produces the largest component-ablation drop; removing Q hurts more than removing D, and removing LInfoNCE lowers success for both encoders.Success falls from 18.67% to 16.64% with D4R-ViT and from 17.97% to 16.93% with R3M-RN50 when LInfoNCE is removed.Source: §3.3 · p. 8, Table 4 · p. 9
  • The proposed alignment objective achieves higher overall success than TCC and Soft-DTW with both encoders.The authors note that noisy frames can diffuse TCC soft-nearest-neighbor weights, while Soft-DTW boundary constraints can misalign sequences whose endpoints show different progress.Source: §3.3 · p. 8, Table 4 · p. 9

3.4 Additional Analysis

  • 14.74% with late-adapter fine-tuning exceeds HRAlign's 12.12%; BN-affine tuning reaches 17.97% with 53.1K parameters, about 29.8× fewer than the late adapter.The late-adapter comparison uses the same initialization, downstream configuration, data, hyperparameters, and training iterations.Source: Figure 5 · p. 9, §3.4 · p. 9
  • Success rises monotonically with paired-data fraction, from 12.28% at 10% of the data to 16.48% at 75% and 17.97% with the full dataset.Other configurations are held fixed while the data fraction varies.Source: Figure 5 · p. 9, §3.4 · p. 9

4 Related Work

Prior work spans human-video visual pretraining, human-to-robot transfer, and temporal correspondence learning. FLTA focuses on frame-level cross-domain correspondences to adapt pretrained visual encoders for manipulation.

4 Related Work

5 Conclusion and Limitations

The framework aligns human and robot frames using global-progress and local-order priors, without frame-level correspondence annotations. The paper reports stronger simulation and real-world results, while noting limitations in real-world data coverage and paired-video visibility.

Conclusion

  • The framework combines global progress and local temporal order priors with frame discrimination to learn human–robot correspondences without frame-level labels.It updates only normalization-layer affine parameters; task-progress retrieval provides additional evidence of learned cross-domain correspondences.Source: §5 · p. 10

Limitations of Real-World Transfer

  • With 40 demonstrations per task, training data may not cover deployment object positions, and small or occluded objects are difficult to localize from one external camera.The authors plan larger, more diverse robot datasets and complementary camera views.Source: §5 · p. 10

Quality of Paired Videos

  • When key actions are not visible in RH20T videos, visual similarity is less reliable for matching task progress, making temporal alignment more difficult.The authors plan to explore shared human–robot signals that reduce reliance on visual similarity.Source: §5 · p. 10

6 Impact Statement

The impact statement describes limited use of language models for editorial refinement and identifies the research's focus and possible application.

Language-model use

  • LLMs were used for grammatical refinement, word choice, and expression.The statement says they enhanced grammatical accuracy, textual clarity, and lexical selection.Source: §6 · pp. 10–11
  • The research design, implementation, data analysis, and conceptual development did not rely on LLMs.Source: §6 · p. 11

Potential impact

  • The work studies adapting visual representations for manipulation and may support more effective reuse of human demonstrations for robot learning.Source: §6 · p. 11

A.1 Temporal Differences in Paired Human–Robot Videos

The appendix reports temporal differences in paired human–robot videos and details the alignment objective, frame-discrimination loss, and pretraining setup.

A.1 Temporal Differences in Paired Human–Robot Videos

  • 90.1% of RH20T pairs have longer robot execution; median robot-to-human execution-time ratio is 2.12, while non-key frames average 31.7% for humans and 2.6% for robots.Human videos have more non-key frames in 95.8% of pairs; both differences hold across all 145 tasks using within-task medians.Source: §A.1 · p. 14, Figure 6 · p. 14

A.2 Notation

A.3 Details of the Bidirectional Alignment Objective

  • The soft target T is constructed from visual similarity and the global progression prior, normalized with 20 Sinkhorn iterations, then matched by row-wise cross-entropy.A final row normalization produces T, and gradients are stopped through the target.Source: §A.3 · p. 15
  • The local prior penalizes backward transitions but allows stays and forward transitions; the reverse-direction objective is averaged with the forward loss.Source: §A.3 · p. 15

A.4 Frame Discrimination Loss

  • InfoNCE distinguishes frames using augmented views of the same frame as positives and representations from other frames as negatives.Source: §A.4 · p. 15

A.5 Pretraining Details

  • Pretraining used RH20T human–robot pairs on one NVIDIA RTX 5090; ViT took about 14.10 hours and RN50 about 12.97 hours.The dataset contained approximately 64K video pairs.Source: §A.5 · pp. 15–16

A.5.1 Number of Sampled Frames

Eight sampled frames are the default: adding frames gave only small further success-rate gains while increasing compute and memory costs.

A.5.1 Number of Sampled Frames

  • 17.97% average success with eight frames was close to 18.05% with nine and 18.10% with ten, so eight was adopted as the practical default.Three frames achieved 11.65%; nine or ten frames required reducing batch size from 20 to 16 and took 15–17 hours versus 12.97 hours for eight.Source: §A.5.1 · p. 16, Figure 7 · p. 17

B.1 Downstream Policy Architectures and Hyperparameters

The section specifies lightweight downstream policies and their training and evaluation setups for simulation and real-world manipulation. Simulation policies use frozen visual encoders, while the real-world setup uses ACT with the evaluated encoder frozen.

B.1 Downstream Policy Architectures and Hyperparameters

  • RVT-2 uses an 8-layer attention network for coarse localization and a second 8-layer network for local prediction; the modified version changes only the second stage.The policy uses the second stage to predict end-effector position, rotation, gripper state, and collision control state.Source: §B.1 · p. 17
  • Simulation policies use frozen visual encoders and are trained on 18 RLBench tasks with 100 demonstrations per task for 25,000 iterations.The global batch size is 3; checkpoints are evaluated in three runs using 25 randomly selected test demonstrations per task.Source: §B.1 · pp. 17–18
  • The real-world policy follows the ACT training pipeline, replacing its default ResNet-18 encoder with the evaluated frozen encoder while keeping other configurations unchanged.This setup isolates the effect of the learned visual representations from end-to-end adaptation.Source: §B.1 · p. 17, Table 8 · p. 18

B.2 Human-to-Robot Task-Progress Retrieval

The section describes task-progress retrieval evaluations, an unseen-task simulation benchmark, and real-world manipulation experiments. Retrieval examples illustrate correspondences across human and robot videos, while the simulation and real-world passages specify their evaluation settings.

B.2 Human-to-Robot Task-Progress Retrieval

  • Retrieval evaluation uses 100 randomly sampled human query frames and counts a retrieval as correct when query and retrieved frames belong to the same manually annotated task stage.RH20T queries retrieve from their paired robot video; self-collected-task queries retrieve from a shared pool of five robot videos.Source: §B.2 · p. 19, Table 10 · p. 21, Table 11 · p. 22
  • In selected examples, the method retrieves robot frames from the same annotated progress segment as the human query, while Frozen R3M and HRAlign retrieve outside the press-task action interval.The retrieval examples complement quantitative results reported in Table 3.Source: §B.2 · p. 19, Figure 8 · p. 19, Figure 9 · p. 20

B.3 Simulation Environment

  • AGNOSTOS trains on the standard 18 RLBench task categories and tests on 23 unseen categories, with Level-1 and Level-2 separating partial semantic overlap from previously unseen object and action semantics.The benchmark evaluates transfer to tasks without downstream demonstrations.Source: §B.3 · p. 23, Table 12 · p. 23

B.4 Real World Environment

  • Real-world evaluation uses four WidowX AI tasks covering pick-and-place, button pressing, goal-directed pushing, and object opening.Each task uses 40 teleoperated demonstrations and 20 evaluation episodes with object positions resampled for evaluation.Source: §B.4 · pp. 23–24, Figure 10 · p. 24
Loading 2610.04372v1…