PaperlayerView exact revision on arXiv

Source-linked AI summary

LIV: Language-Image Representations and Rewards for Robotic Control

Yecheng Jason Ma, William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, Dinesh Jayaraman

arXiv:2306.00958v1cs.ROcs.AIcs.LG

TL;DR

Robotic control needs vision-language representations that generalize across homes, capture task progress, and adapt with scarce domain-specific data. LIV unifies multi-modal representation and reward learning from action-free, text-annotated videos, then supports domain fine-tuning and dense goal-conditioned rewards. Across simulated and real-world settings, LIV outperforms prior approaches in representation-based imitation learning and reward specification.

  • Problem

    General-purpose robots need representations that align language with visual context, capture task-directed progress, and adapt to new domains despite scarce robotics data.

  • Method

    LIV jointly learns vision-language representations and rewards from action-free text-annotated videos, using the same objective for generic pre-training and fine-tuning on small in-domain robot datasets.

  • Results

    LIV models consistently outperform prior input-state representations for imitation learning and reward-specification methods for policy synthesis across simulated and real-world robotic settings.

  • Takeaways & Limitations

    LIV provides one control-oriented vision-language framework for pre-training, domain-specific language grounding, and language-conditioned task reward specification.

  • Takeaways & Limitations

    Pre-training assumes access to language-annotated videos, and out-of-domain datasets such as human videos provide no action labels.

Abstract

from arXiv · show

We present Language-Image Value learning (LIV), a unified objective for vision-language representation and reward learning from action-free videos with text annotations. Exploiting a novel connection between dual reinforcement learning and mutual information contrastive learning, the LIV objective trains a multi-modal representation that implicitly encodes a universal value function for tasks specified as language or image goals. We use LIV to pre-train the first control-centric vision-language representation from large human video datasets such as EpicKitchen. Given only a language or image goal, the pre-trained LIV model can assign dense rewards to each frame in videos of unseen robots or humans attempting that task in unseen environments. Further, when some target domain-specific data is available, the same objective can be used to fine-tune and improve LIV and even other pre-trained representations for robotic control and reward specification in that domain. In our experiments on several simulated and real-world robot environments, LIV models consistently outperform the best prior input state representations for imitation learning, as well as reward specification methods for policy synthesis. Our results validate the advantages of joint vision-language representation and reward learning within the unified, compact LIV framework.

1. Introduction

LIV addresses the need for control-aware vision-language representations that align language and vision, encode task progress, and adapt from generic video to specific robotic domains. It unifies representation and reward learning, enabling pre-training, fine-tuning, and language-conditioned control applications.

  • LIV targets representations that align language with visual observations and capture task-directed progress for intermediate learning signals.
  • The method supports extensive domain-generic pre-training followed by domain-specific fine-tuning on small robotics datasets.
  • LIV learns joint vision-language representations and rewards from arbitrary text-annotated videos, including action-free human activity datasets.
  • LIV generalizes VIP’s visual goal-conditioned value learning to multi-modal vision-language values and representations.
  • The same objective yields temporal coherence and semantic alignment in a cross-modal embedding for diverse visuomotor control applications.
  • Experiments evaluate LIV for language-conditioned behavior cloning and visual reward specification, including zero-shot dense rewards on unseen robot videos.

2. Related Work

LIV extends prior representation-learning and language-conditioned manipulation work with a control-centric, multi-modal objective that supports pre-training, fine-tuning, and reward learning. It addresses the costly data demands of demonstration-based policy learning while targeting stronger robotic control representations.

  • Unlike VIP, which focuses on visual pre-training, LIV learns multi-modal vision-language values and representations.
  • LIV is presented as the first multi-modal vision-language objective simultaneously suitable for pre-training, fine-tuning, and reward learning in language-conditioned robotic control.
  • Prior language-conditioned manipulation methods use pre-trained representations but do not learn better control-centric representations from large out-of-domain text-annotated videos.
  • LIV extends beyond language-conditioned behavior cloning by supporting visual state-based rewards for alternative policy-synthesis algorithms.
  • Demonstration-based language-conditioned behavior cloning requires expensive collection and text labeling that can take months.

3. Preliminaries & Problem Setting

The paper builds on VIP’s action-free visual value learning and formulates a language-annotated video setting with shared vision-language embeddings. Contrastive alignment supplies semantic correspondence, while the learned value representation supports control and dense reward specification.

  • VIP learns an optimal goal-conditioned value function from videos through a dual goal-conditioned reinforcement-learning formulation.
  • VIP parameterizes value as embedding-space similarity, making the same model both a visual representation and a goal-conditioned reward-learning algorithm.
  • Because VIP does not use actions, it can be pre-trained on large-scale human video datasets and applied to unseen robot tasks.
  • The assumed dataset contains language-annotated videos whose final frames represent described outcomes, without requiring action labels for out-of-domain human videos.
  • The vision and language encoders map images and natural-language strings into a shared K-dimensional representation space.
  • InfoNCE aligns matching image-text pairs while repelling mismatched pairs through a cross-modal joint embedding.

4. LIV: Language-Image Value Learning

LIV extends value learning to language and image goals, combining temporal value learning with vision-language contrastive alignment. The resulting objective supports both control-oriented representation learning and domain-specific fine-tuning.

  • Objective: LIV extends VIP from image goals to multimodal language and image goals by learning separate language- and image-conditioned value objectives.The two components are denoted VIP-L and VIP-I.
  • Objective: Without cross-modal alignment, semantically equivalent image and language goals may remain distant, undermining language-grounded reward specification.LIV addresses this alignment problem through its connection to contrastive learning.
  • Theoretical analysis: For degenerate videos of repeated text-aligned frames, the VIP-L objective is equivalent to InfoNCE up to a constant.This establishes LIV as a temporal, decision-making generalization of CLIP-style contrastive learning.
  • Fine-tuning: The same LIV objective can fine-tune pre-trained contrastive vision-language models on in-domain, text-annotated robot demonstrations.The authors report that this is more effective than fine-tuning with the CLIP objective because LIV uses sequential, goal-directed video data.
  • Objective: LIV combines a value-learning objective in one modality with a vision-language InfoNCE objective to induce semantic alignment between goals.The implementation samples initial, intermediate, and final frames with text annotations and jointly updates the vision-language architecture.
  • Implementation: LIV is implemented near the CLIP architecture and pre-trained on EpicKitchen, a text-annotated egocentric dataset of humans performing household tasks.This supports comparison with pre-trained CLIP while using large annotated human-video data.

5. Experiments

Experiments evaluate LIV for zero-shot reward prediction, language-conditioned behavior cloning, fine-tuning, and reward-based skill synthesis across simulated and real robotic environments. LIV tracks task progress without robot pre-training data, improves policy learning and fine-tuning results, and supports reward-based acquisition of language-conditioned skills.

  • Experimental scope: The experiments test whether LIV provides multimodal rewards, enables effective vision-language control representations, and fine-tunes existing vision-language models.Evaluation includes zero-shot rewards, representation learning for behavior cloning, and downstream fine-tuning.
  • Zero-shot reward: LIV embedding distances remain informative of task progress on unseen robot videos, including a cost bump corresponding to an unnecessary action.The pre-trained models had never encountered robot data, yet the curves transferred across viewpoints and embodiments.
  • Zero-shot reward: LIV’s zero-shot reward capability is largely absent in CLIP, while LIV can distinguish opposite actions in untrimmed robot videos.The authors also report failure cases in additional examples.
  • Pre-trained representation: LIV achieves the highest average success rates across three distinct environments for language-conditioned behavior cloning.The pre-trained LIV model performs best in all evaluated environments without in-domain fine-tuning.
  • Fine-tuning: LIV fine-tuning substantially improves policy success rates across environments with differing in-domain dataset sizes and qualities.It remains effective even when improving an already strong base LIV model on RealRobot.
  • Fine-tuning: CLIP and VIP fine-tuning alone are insufficient for manipulation, while TCN+CLIP produces mixed results and can reduce performance on smaller datasets.The reported ablations indicate that temporal or semantic fine-tuning alone does not reproduce LIV’s results.
  • Fine-tuning: LIV fine-tuning improves temporal coherence and semantic alignment, whereas CLIP fine-tuning damages representations of earlier frames by over-aligning the goal frame-text pair.The qualitative analysis examines multimodal cost curves over in-domain demonstrations.
  • Reward-based synthesis: LIV’s dense goal-conditioned rewards support direct acquisition of new language-conditioned skills through language-reward model predictive control.Across both planning benchmarks, fine-tuned LIV achieves the best overall performance and improves over base pre-trained LIV and CLIP models.

6. Conclusion

The paper presents LIV as a unified algorithm for control-oriented vision-language pre-training, domain-specific fine-tuning, and language-conditioned reward generation. Trained on human videos and adapted with small robotics datasets, it outperforms prior approaches across three evaluation settings and operates on real-world robotic tasks.

  • Conclusion: LIV unifies control-oriented vision-language pre-training, domain-specific language grounding, and language-conditioned task reward generation.The method is trained on large generic human-video datasets and can be fine-tuned on small robotics datasets.
  • Conclusion: LIV outperforms state-of-the-art approaches across three distinct evaluation settings and successfully operates on real-world robotic tasks.

A. Proof of Proposition 1

The proof shows that under degenerate repeated-frame videos aligned with their text annotations, the VIP-L objective becomes InfoNCE up to a constant. The implementation scales cosine similarity to match the valid value range and uses CLIP components for LIV.

  • Under repeated-frame videos aligned with text, VIP-L is equivalent to InfoNCE up to a constant.
  • The proof rewrites the VIP-L objective by assuming identical observations in the log-sum-exp term and substituting goal-frame distributions.
  • LIV uses CLIP’s modified ResNet50 vision encoder and CLIP Transformer language encoder without architectural modifications.
  • Scaling cosine similarity by 1/(1−γ) maps CLIP’s similarity range to the valid state-value range, making the InfoNCE term precisely CLIP’s objective.
  • LIV is pre-trained on EPIC-KITCHENS-100 using RGB frames and text annotations.

C. Environment Details

The experiments span simulated and real robotic environments with language-conditioned tasks, including tabletop manipulation and household kitchen interactions. The section also reports that LIV benefits substantially from language task specifications.

  • Environment Details: MetaWorld uses a Sawyer robot interacting with four tabletop objects across procedurally language-labeled episodes collected with a random policy.
  • Environment Details: FrankaKitchen uses a Franka robot in a kitchen scene with five tasks and 50 demonstrations per task.
  • Environment Details: RealRobot contains nine fruit-placement tasks using three fruits, three containers, two cameras, and 100 human-teleoperated demonstrations per task.
  • Evaluation: Each evaluation task in the imitation-learning experiments is rolled out for 50 episodes to record success rate.
  • Language Task Encoding: LIV-EPIC benefits more from language task specifications than the strongest baselines, although language hurts all baselines on MetaWorld.

D.2. Zero-Shot Long-Horizon Task Generalization

The RealRobot evaluation tests whether a policy trained on short-horizon atomic demonstrations can generalize to unseen long-horizon sequences. LIV achieves partial success on composite tasks requiring both visual and semantic generalization.

  • Zero-Shot Long-Horizon Task Generalization: RealRobot composite-task evaluation sequences atomic tasks, allocating the policy 100 control steps for each task before resetting and transitioning.
  • Zero-Shot Long-Horizon Task Generalization: The demonstrations contain only short-horizon atomic tasks and omit configurations in which fruits have already been placed.
  • Evaluation Criterion: Partial success counts trials that solve at least two of three tasks in the specified sequence, with results comparing LIV and R3M.
  • Results: LIV generalizes to composite tasks requiring both visual and semantic generalization, and sometimes solves all three tasks.
  • Camera Handling: The real-world policy uses both third-person and wrist-camera observations during policy learning, with pre-training and fine-tuning handled differently across views.

E.2. Additional Results

Additional experiments show that LIV fine-tuning remains effective across base-model quality and dataset size, while a TCN+CLIP baseline is sensitive to objective weighting. Few-shot LIV fine-tuning can substantially improve performance with limited demonstrations.

  • Base-Model Quality: LIV fine-tuning is effective for base models ranging from randomly initialized CLIP architectures to pre-trained LIV and CLIP models.
  • Few-Shot Fine-Tuning: With 10 demonstrations per task, LIV fine-tuning matches CLIP without fine-tuning using 50 demonstrations per task and improves base-model performance by more than 200%.
  • Few-Shot Fine-Tuning: TCN+CLIP produces smaller gains across dataset sizes than LIV fine-tuning.
  • Objective Weighting: TCN+CLIP is sensitive to objective weighting, whereas LIV avoids tuning the balance between its self-supervised and CLIP components.
  • Model-Based Planning: The experiments use CEM with SV2P on MetaWorld and MPPI with ground-truth dynamics on FrankaKitchen for model-based planning.

F.2. Additional Results & Analysis

Additional analyses show that LIV’s reward models are more robust to increased planning budgets and language-specific goals than several baselines. They also expose failure modes in competing reward predictors and a trade-off between reward learning and representation quality.

  • Reward–representation trade-off: CLIP fine-tuning improves reward learning but produces a much worse representation backbone for policy learning by collapsing intermediate observations.The contrast reflects different requirements for terminal reward scoring and temporally well-behaved imitation-learning representations.
  • Baseline analysis: LOREL is prone to overfitting on small in-domain FrankaKitchen data and performs best when initialized with a pre-trained representation.The R3M language-reward predictor is reported as inferior even to purely in-domain-trained LOREL on MetaWorld.
  • Planning budget: LIV-EPIC fine-tuned with three CEM iterations achieves the best performance, while most pre-trained baselines degrade as planning budget increases.LIV with one CEM iteration already matches LOREL with three, suggesting fewer false negatives for good trajectories.
  • Baseline analysis: R3M’s approximately 9% zero-shot FrankaKitchen success rate persists under random language goals, indicating limited dependence on task-specific language grounding.The analysis attributes this behavior to a reward predictor biased toward visually changing frames.

G. Representation Qualitative Results

Qualitative results show that LIV can track language- and image-conditioned progress across seen, unseen, simulated, and real videos, while fine-tuning improves domain-specific language grounding. The analyses also reveal failure modes from distribution shift and representation-smoothing choices.

  • Unseen robot videos: LIV captures language-conditioned progress in unseen robot videos, whereas CLIP’s zero-shot language rewards are much noisier.This comparison is presented on the same set of unseen robot videos.
  • Untrimmed videos: When opposite actions occur in one video, LIV’s image- and language-goal reward curves invert because the goals refer to different completed actions.The result demonstrates sensitivity to fine-grained, action-induced object-state changes.
  • Failure cases: LIV language rewards fail in some unseen robot videos, with likely causes including network capacity and shifts in viewpoint, embodiment, and language commands.The qualitative appendix also includes these failure examples alongside successful opposite-action cases.
  • RealRobot tasks: Pre-trained LIV produces reasonable visual progress on RealRobot tasks but suffers from a domain gap that weakens language-progress tracking; LIV fine-tuning remedies this.Fine-tuning also smooths the representation in both modalities.
  • FrankaKitchen tasks: On FrankaKitchen, pre-trained LIV-EPIC captures visual progress but lacks language grounding, while LIV fine-tuning captures fine-grained language-conditioned progression and improves temporal alignment.The comparison covers averaged and task-specific reward curves across five tasks.
  • FrankaKitchen tasks: CLIP fine-tuning over-aligns last-frame representations with text goals and collapses intermediate representations, while TCN+CLIP lacks the temporal smoothness needed for control.These effects are shown in the corresponding FrankaKitchen reward-curve comparisons.
Loading 2306.00958v1…