PaperlayerView exact revision on arXiv

Source-linked AI summary

Time-Contrastive Networks: Self-Supervised Learning from Video

Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine

arXiv:1704.06888v3cs.CVcs.RO

TL;DR

The paper tackles how robots can learn task-relevant, viewpoint-invariant representations and imitate humans from unlabeled video across different embodiments. It trains Time-Contrastive Networks with simultaneous-view positives and temporal-neighbor negatives, then uses the representations for reward-based robotic imitation and direct human-pose imitation. The reported demonstrations include real-robot pouring, simulated manipulation, and real-time pose imitation.

  • Problem

    Robots need representations that capture interactions, object attributes, and pose across viewpoints and embodiments without impractical task-specific labeling.

  • Method

    Time-Contrastive Networks use multi-view metric learning to attract simultaneous viewpoints and repel temporally different frames, learning functional features while ignoring nuisance variation.

  • Results

    The representation supports reward-based robotic skill learning from raw video and direct human-pose imitation without explicit joint-level correspondence.

  • Takeaways & Limitations

    Robotic imitation can be learned from raw human videos across differing viewpoints, agents, and embodiments using self-supervised representations.

Abstract

from arXiv · show

We propose a self-supervised approach for learning representations and robotic behaviors entirely from unlabeled videos recorded from multiple viewpoints, and study how this representation can be used in two robotic imitation settings: imitating object interactions from videos of humans, and imitating human poses. Imitation of human behavior requires a viewpoint-invariant representation that captures the relationships between end-effectors (hands or robot grippers) and the environment, object attributes, and body pose. We train our representations using a metric learning loss, where multiple simultaneous viewpoints of the same observation are attracted in the embedding space, while being repelled from temporal neighbors which are often visually similar but functionally different. In other words, the model simultaneously learns to recognize what is common between different-looking images, and what is different between similar-looking images. This signal causes our model to discover attributes that do not change across viewpoint, but do change across time, while ignoring nuisance variables such as occlusions, motion blur, lighting and background. We demonstrate that this representation can be used by a robot to directly mimic human poses without an explicit correspondence, and that it can be used as a reward function within a reinforcement learning algorithm. While representations are learned from an unlabeled collection of task-related videos, robot behaviors such as pouring are learned by watching a single 3rd-person demonstration by a human. Reward functions obtained by following the human demonstrations under the learned representation enable efficient reinforcement learning that is practical for real-world robotic systems. Video results, open-source code and dataset are available at https://sermanet.github.io/imitate

I. INTRODUCTION

The paper addresses learning task-relevant representations and robotic behaviors from unlabeled video, including imitation across viewpoints and embodiments. Its self-supervised multi-view approach learns viewpoint- and agent-invariant features that support reinforcement-learning and direct pose imitation.

  • Motivation: Robots need to learn interaction attributes from observation despite variation in backgrounds, objects, viewpoints, and salient task properties.Pouring may depend on hand contact, container tilt, liquid amount, and viscosity, making exhaustive supervision impractical.
  • Approach: The paper uses unlabeled multi-view interaction videos to learn representations that disentangle functional attributes such as pose from viewpoint and agent identity.These representations are then linked to motor commands through reinforcement learning or direct regression.
  • Representation: The proposed embedding is sensitive to object interactions and pose while ignoring nuisance variables such as viewpoint and appearance.It is built on semantically relevant ImageNet features and trained as a metric embedding.
  • Demonstrated uses: The representation supports reward-based robotic skill learning from raw video and direct human-pose imitation without explicit joint-level correspondence.Experiments cover real-robot pouring, simulated dish-rack manipulation, and real-time human-pose imitation.
  • Imitation setting: Third-person imitation is motivated by the need to transfer behaviors across contexts and embodiments without requiring demonstrations in the learner’s context.This could make acquiring broad repertoires of robotic skills more scalable than kinesthetic or teleoperated demonstrations.
  • Novelty: The method combines temporal contrast with multi-view correspondence to learn meaningful interaction attributes while remaining invariant to nuisance variation.Unlike approaches using random negatives from other videos, it uses temporal neighbors as negatives and simultaneous views as positives.

III. IMITATION WITH TIME-CONTRASTIVE NETWORKS

TCNs learn from simultaneous multi-view frames by pulling corresponding views together and pushing temporally different frames apart. This combination encourages viewpoint invariance while preserving changes that reveal pose, task progress, and object interactions.

  • Method: The imitation pipeline first learns abstract visual representations from passive sensory observation, then uses them to guide robotic imitation and task learning.The approach avoids requiring labels for every observed behavior and retains scalability as a design goal.
  • Training signal: TCN training uses multi-view metric learning in which co-occurring frames are positives and temporally different frames are negatives.The model is trained to distinguish what is shared across different views from what changes across nearby times.
  • Learned invariances: Cross-view correspondence encourages invariance to viewpoint, scale, occlusion, motion blur, lighting, and background.The same subject appears across views with changes in these factors, supplying correspondence without manual labels.
  • Temporal structure: Temporal competition encourages the embedding to represent changing attributes such as liquid amount, hand pose, object pose, and interactions.Visually similar temporal neighbors must be distinguished despite their appearance similarity.
  • Data collection: Two-operator smartphone capture provides synchronized multi-view data with varied scale, viewpoint, occlusion, blur, and background conditions.The authors characterize this setup as cheap and practical relative to human labeling, though more cumbersome than single-view capture.
  • Single-view variant: A single-view TCN samples positives near an anchor and negatives outside a margin range, remaining useful when multi-view data is unavailable.The reported configuration uses a positive range of 0.2s and a margin range twice that size.

B. Learning Robotic Behaviors with Reinforcement Learning

For third-person demonstrations, the robot compares demonstration and execution embeddings rather than raw pixels, then optimizes an embedding-based reward with reinforcement learning. The TCN embedding supplies a low-dimensional visual state that supports trajectory optimization across differing embodiments and contexts.

  • Problem and representation: Third-person human demonstrations differ from the robot’s embodiment and context, so direct pixel tracking is not a sensible imitation signal.TCN features instead provide invariance to camera angle and manipulated objects while representing physical interactions.
  • Training alternative: Single-view TCN uses nearby positives and distant temporal negatives, providing an alternative training signal when multi-view data is unavailable.The multi-view version is reported to perform best, while the single-view variant still improves over ImageNet-trained features.
  • Embedding-based reward: The method represents each demonstration and robot-observation frame as TCN embeddings and defines reward from their squared Euclidean distance with a Huber-style loss.The reward compares corresponding demonstration and execution embeddings over the task sequence.
  • Reward behavior: The squared-distance term provides stronger gradients when embeddings are far apart, while the Huber-style term dominates near convergence for precision and motion fine-tuning.The two terms therefore address early policy updates and late-stage execution refinement, respectively.
  • Policy optimization: The embedding-based reward is optimized with reinforcement learning using PILQR, which combines model-based LQR updates, fitted linear dynamics, and model-free corrections.The TCN representation is 32-dimensional in the experiments and is included in the robot’s system state.

C. Direct Human Pose Imitation

The paper studies direct human-pose imitation by combining a shared TCN representation with self-regression, allowing robots to map human and robot poses without explicit joint correspondences.

  • Direct pose imitation: Direct pose imitation maps human observations to robot behavior without explicitly specifying joint-level correspondences.The TCN is used to model pose-sensitive representations and implicit human–robot correspondence.
  • Self-regression: Self-regression trains the robot to predict its internal joint state from an image of itself.The robot first observes itself and humans performing random motions, then learns the image-to-joint mapping.
  • Evaluation: Experiments evaluate TCN representations for pose modeling, object interactions, and implicit human–robot correspondences.The broader experimental program includes real-time human-pose imitation alongside manipulation tasks.
  • Training signals: Pose imitation uses three training signals: time-contrastive learning, self-regression, and human supervision.The time-contrastive signal learns representations of humans or robots, while the other signals train joint prediction from robot images or human attempts to imitate robot poses.

3) Model selection:

The evaluation compares representation models using alignment and classification measures, with model selection based on validation loss or a small labeled validation set. Multi-view TCN outperforms the baselines while requiring substantially less training time than single-view models.

  • Model selection: Models selected by validation classification score perform better on classification, while validation-loss selection performs only slightly worse.The paper reports both selection approaches because unsupervised training raises model-selection questions.
  • Model comparison: Multi-view TCN outperforms single-view models while requiring 15x less training time on the same dataset.The authors attribute the improvement to exploiting temporal correspondences.
  • Evaluation metrics: The alignment metric measures semantic correspondence between videos, while the classification metric measures pouring-related attributes.Evaluation uses nearest neighbors in embedding space; classification covers attributes relevant to pouring behavior.
  • Model comparison: Multi-view TCN outperforms all baselines, whereas single-view TCN and shuffle & learn are comparable on classification but not alignment.General off-the-shelf Inception features significantly underperform the other baselines.
  • Tasks: The object-interaction experiments include simulated plate transfer and real-robot pouring learned from third-person demonstrations.The simulated task uses a dish-rack environment and the real task uses human demonstrations.

1) Task Setup:

The paper evaluates reinforcement-learning imitation on simulated plate transfer and real-robot pouring, using multi-view videos to train the TCN reward representation. The multi-view model enables successful pouring after few iterations, unlike the tested alternatives.

  • Simulated dish-rack task: The simulated dish-rack task requires moving plates between racks through reaching, grasping, carrying, and placing.Human demonstrations are recorded in VR with first-view and third-person cameras.
  • Real pouring task: The real pouring dataset combines human pouring videos, non-pouring cup interactions, and robot-arm manipulation videos.The additional robot data helps the TCN represent human and robot arms and implicitly place them in correspondence.
  • Real pouring task: The real pouring experiment uses a TCN reward function to learn from a single third-person human demonstration.Performance is measured by the weight of beads in the receiving container across repeated rollouts.
  • Results: After 10 iterations, the multi-view TCN policy converges to consistently successful pouring behavior.The robot pours a significant amount after the first several iterations, while other models fail to accomplish the task.
  • Results: Multi-view correspondences provide useful reward guidance and gradients for reinforcement learning in the pouring task.The experiment suggests that single-view representations are insufficient for correctly modeling third-person object interaction here.

3) Qualitative Evaluation:

The TCN representation supports efficient robotic imitation from raw human videos, both as a reinforcement-learning reward and as a direct human-to-robot pose mapping.

  • Reinforcement-learning imitation: 10 iterations with 10 roll-outs per iteration suffice for the dish-rack task, whose policy learns reaching, grasping, carrying, and placement.The authors characterize the TCN reward as dense and smooth enough to guide a complex imitation policy.
  • Reinforcement-learning imitation: In pouring, the robot progresses from spilling beads during exploration to consistently pouring most beads after several iterations.The task involves non-linear transitions of granular media, which are difficult for conventional state estimation techniques.
  • Direct pose imitation: The multi-view TCN also maps human observations to robot poses in real time without explicit human-pose representation or joint-level correspondence.The resulting joints vector can be used by the robot stack to update its joints directly.
  • Direct pose imitation: Training with all supervision signals performs best, while the time-contrastive signal consistently improves pose-imitation performance.The self-supervised TC+self model outperforms the human-supervised model, although joint-angle evaluation is limited by subjective imitation and body differences.
  • Overall result: The method enables robotic imitation from raw human videos after training on multi-viewpoint video, using only a single raw demonstration for each imitation task.The conclusion covers both reward-based object manipulation and direct human-pose imitation.

APPENDIX

The reinforcement-learning appendix defines trajectory costs and policy optimization, then applies PILQR with time-varying linear-Gaussian controllers for efficient real-robot learning.

  • Trajectory formulation: A trajectory is a sequence of states and actions, and its cost is the sum of per-step costs c(x_t, u_t).The policy is optimized with respect to expected trajectory cost under the policy-induced trajectory distribution.
  • Trajectory formulation: The cost-function formulation is used because it is more common in trajectory-optimization theory than a reward-function formulation.This is a notation choice rather than a change in the underlying reinforcement-learning objective.
  • Policy optimization: PILQR learns time-varying linear-Gaussian controllers for real-robot reinforcement learning.The controller combines model-based and model-free policy updates for tasks with complex system dynamics.
  • Policy optimization: PILQR uses KL-constrained optimization and separates a fast model-based update from optimization of the residual cost-to-go.The model-based step uses LQR-FLM, while the residual is optimized separately.

C. Objects Interaction Analysis

The object-interaction analysis finds that multi-view TCN embeddings separate combinations of task-relevant attributes while remaining robust to visual nuisance factors.

  • Embedding analysis: t-SNE shows more separation of five-attribute combinations for multi-view TCN than for ImageNet-Inception.Each color represents a unique combination of five ground-truth attribute values.
  • Embedding analysis: The representation is intended to capture object interactions and functional attributes while discarding nuisance variation such as viewpoint and appearance.The method combines simultaneous multi-view signals with a task-focused image embedding.
  • Pouring evaluation: The table evaluates pouring alignment and five classification categories, including hand contact, pouring distance, container angle, liquid flow, and recipient fullness.Models are selected using classification score on a small labeled validation set and evaluated on the full test set.
  • Pouring representation: The multi-view TCN distinguishes hand poses and poured-liquid amounts from unsupervised observations while remaining invariant to viewpoint, background, objects, subjects, motion blur, and scale.The comparison uses nearest neighbors for pouring test images across multi-view TCN, shuffle-and-learn, and ImageNet-Inception models.

D. Pose Imitation Analysis

The pose-imitation analysis combines weak self-supervision, human supervision, and multi-view data to map human observations to robot joints without explicit human-pose representation.

  • Pose-imitation data: Approximately 2 hours of unlabeled human imitation and 3 hours of random robot motion provide time-contrastive supervision, compared with 40 minutes of expensive human supervision.Training and evaluation use human subjects and clothing pairs, with held-out pairs for validation and testing.
  • Representation analysis: t-SNE attribute-combination visualizations show better separation for TCN embeddings than for ImageNet-Inception embeddings.The visualization colors embeddings by unique combinations of five attribute values.
  • Evaluation caveat: Pose-imitation evaluation reports normalized joint-distance error, but perfect imitation is impossible because human supervision is noisy and human and robot bodies differ physiologically.The metric compares predicted joint angles with the angles the human was attempting to imitate.
  • Pose-imitation model: The pose model adds a joint decoder trained with L2 regression and directly feeds predicted joints into the robot stack at test time.This produces end-to-end imitation from pixels to joints without explicitly representing human pose.
  • Task learning: A single third-person human demonstration can support learning the pouring task with the multi-view TCN model after 20 iterations.The figure reports recipient weight in grams and standard deviation over 10 rollouts per iteration, with a maximum weight of 189g.
  • Supervision comparison: Self-supervised imitation outperforms human-supervised imitation in the comparison of supervision types.The comparison is presented as TCN with self-supervision versus human supervision.

3) Supervision Analysis:

The supervision analysis finds that adding temporal-contrastive supervision consistently reduces imitation error, while TC+Self can outperform full human supervision despite using no labels. Most human-supervision gains are concentrated in predicting the shoulder-pan joint, and performance improves with more unsupervised data.

  • Supervision combinations: TC+Self significantly outperforms the fully supervised Human model despite receiving no labels.Adding TC supervision to other signals also significantly decreases imitation error.
  • Joint-wise error: Human supervision improves overall imitation mainly by correcting shoulder-pan prediction.For every other joint, TC+Self performs almost as well as TC+Human+Self; on shoulder pan, TC+Self performs worse than Random.
  • Qualitative evaluation: Qualitative evaluations show that TC+Self associates poses across unseen humans and backgrounds while maintaining viewpoint, scale, and translation invariance.The evaluations include k-nearest neighbors, imitation strips, and t-SNE visualizations.
  • Data scaling: Increasing the number of unsupervised sequences decreases imitation error for both models.The reported trend supports using relatively large amounts of cheap weakly supervised data alongside smaller amounts of expensive human supervision.
  • Invariance: The model remains competitive under viewpoints and camera scales not seen during training, although imitation error increases.Tests use viewpoints of 30°, 90° and 150° instead of 0°, 60° and 120°, plus a camera distance about halfway to the subject.

F. Imitation Examples

The imitation examples evaluate the learned embedding with t-SNE visualizations and robot-joint error breakdowns, showing reduced organization by agent or view after training. Self-supervised imitation remains approximately effective for unseen subjects and altered capture conditions, with errors increasing but staying competitive under viewpoint and scale changes.

  • Joint-error analysis: The joint-error breakdown reports eight Fetch joints, their average, and the average excluding shoulder pan.This breakdown isolates the joint for which self-supervised imitation is weakest.
  • Embedding visualizations: After training, t-SNE points from the same agent or view spread across the manifold, indicating agent and viewpoint invariance.Before training, points form concentrated agent-specific and view-specific clusters.
  • Scale invariance: When the camera moves to about half the training distance, error increases but remains competitive and below the human-supervised baseline.This tests scale changes relative to the training capture conditions.
  • Imitation examples: TC+Self approximately imitates human subjects unseen during training without human supervision.The examples include discovering a mapping from many human joints controlling crouching to the robot’s torso joint.
Loading 1704.06888v3…