PaperlayerView exact revision on arXiv

Source-linked AI summary

Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?

Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, Franziska Meier

arXiv:2303.18240v2cs.CVcs.AIcs.LGcs.RO

TL;DR

Embodied AI lacks evidence for a single visual representation that works best across its diverse tasks. The paper builds CortexBench and systematically evaluates PVRs while scaling pre-training data and adapting its largest model. VC-1 improves average performance, but universal dominance requires task- or domain-specific adaptation.

  • Problem

    It remains unknown whether an artificial visual cortex, operationalized as a universal PVR for embodied AI, already exists.

  • Method

    The paper curates CortexBench’s 17 diverse tasks, evaluates existing PVRs, and trains ViTs with MAE on four datasets formed from egocentric video and ImageNet.

  • Results

    No single PVR is universally dominant; VC-1 improves average performance by 1.2% over the best existing PVR, while adapted VC-1 matches or exceeds prior results on all CortexBench benchmarks.

  • Takeaways & Limitations

    A broadly useful visual backbone is possible, but the strongest reported performance currently comes from adapting VC-1 to particular tasks or domains.

  • Takeaways & Limitations

    VC-1 is effective broadly but is not always best for a specific task, and task-specific pre-training produces multiple specialized models rather than one unified foundation model.

Abstract

from arXiv · show

We present the largest and most comprehensive empirical study of pre-trained visual representations (PVRs) or visual 'foundation models' for Embodied AI. First, we curate CortexBench, consisting of 17 different tasks spanning locomotion, navigation, dexterous, and mobile manipulation. Next, we systematically evaluate existing PVRs and find that none are universally dominant. To study the effect of pre-training data size and diversity, we combine over 4,000 hours of egocentric videos from 7 different sources (over 4.3M images) and ImageNet to train different-sized vision transformers using Masked Auto-Encoding (MAE) on slices of this data. Contrary to inferences from prior work, we find that scaling dataset size and diversity does not improve performance universally (but does so on average). Our largest model, named VC-1, outperforms all prior PVRs on average but does not universally dominate either. Next, we show that task- or domain-specific adaptation of VC-1 leads to substantial gains, with VC-1 (adapted) achieving competitive or superior performance than the best known results on all of the benchmarks in CortexBench. Finally, we present real-world hardware experiments, in which VC-1 and VC-1 (adapted) outperform the strongest pre-existing PVR. Overall, this paper presents no new techniques but a rigorous systematic evaluation, a broad set of findings about PVRs (that in some cases, refute those made in narrow domains in prior work), and open-sourced code and models (that required over 10,000 GPU-hours to train) for the benefit of the research community.

1 Introduction

The paper asks whether a universal artificial visual cortex exists for embodied AI and evaluates that question across diverse tasks, representations, datasets, and learning settings. It finds no universally dominant PVR, while VC-1 adapted is competitive with or better than prior results across CortexBench.

  • Research question: CortexBench contains 17 tasks spanning locomotion, manipulation, navigation, and mobile manipulation across varied environments, embodiments, and learning conditions.The benchmark includes settings from flat planes and tabletop scenes to photorealistic indoor scans, with learning ranging from few-shot imitation to large-scale reinforcement learning.
  • Findings: Existing PVRs generally outperform learning-from-scratch baselines, but no single representation is universally dominant across CortexBench.PVRs tend to work best in domains they were originally designed for, so the paper concludes that an artificial visual cortex does not already exist.
  • Scaling study: Over 4,000 hours of egocentric video from 7 sources and ImageNet were combined into four datasets for ViT models trained with MAE.The largest dataset contains over 5.6M images, and the released models required over 10,000 GPU-hours to train.
  • Scaling study: 1.2% average improvement over the best existing PVR was achieved by VC-1, although smaller-data PVRs still outperformed it on specific tasks.More data diversity helped on average but not universally; Mobile-Pick performed best with manipulation-focused pre-training.
  • Adaptation: Task- or domain-specific adaptation makes VC-1 competitive with or superior to the best prior results on every CortexBench benchmark.The comparison is unusually demanding because the benchmark-specific prior results need not share any design choices.
  • Hardware experiments: Real-world experiments on TriFinger and Franka Emika Panda found that VC-1 and VC-1 adapted substantially outperform pre-existing PVRs such as MVP.The experiments used few-shot imitation learning.

2 Related Work

Related work develops self-supervised visual representations and applies them to embodied AI, but prior evidence is fragmented across methods, datasets, and task domains. This paper extends scaling and adaptation questions to a broad, systematic EAI evaluation.

  • Pre-trained visual representations: Self-supervised visual representation learning uses contrastive, distillation-based, or reconstructive objectives, increasingly with vision transformers and masked image modeling.Masked image modeling can reduce pre-training computation time.
  • PVRs for embodied AI: Prior EAI studies use PVRs to improve navigation and manipulation, including egocentric-video pre-training methods targeting robot manipulation.These studies report promising results under protocols such as few-shot imitation learning.
  • Language-guided foundation models: Language-guided EAI work learns representations from image-text pairs, language-aligned videos, or open-vocabulary visual detectors and vision-language models.Applications include goal-conditioned value functions and tabletop object detection.
  • Scaling model and dataset size: Earlier EAI work reported consistent benefits from scaling model and data sizes for robot manipulation, motivating broader testing across embodied tasks.This paper identifies a broader evaluation as necessary for assessing whether that trend generalizes.
  • Adapting PVRs: PVR adaptation remains open: fine-tuning can reduce simulation performance, while end-to-end navigation fine-tuning and simple k-nearest-neighbor adaptation have shown gains in other settings.The reported outcomes vary by task and adaptation strategy.

3 Benchmarking Progress Towards an Artificial Visual Cortex

CortexBench fixes diverse EAI tasks and downstream learning protocols so evaluations can focus on the contribution of visual representations. Its task suite spans control, navigation, dexterous manipulation, and mobile manipulation.

  • Benchmark design: CortexBench selects 17 tasks from 7 existing EAI benchmarks and pairs each with a community-standard policy-learning paradigm and evaluation protocol.Fixing tasks and downstream methods isolates the contribution of the visual representation.
  • Benchmark design: Mean Success averages success rates across benchmarks, while Mean Rank averages each PVR’s benchmark-specific success-rate rankings.These metrics summarize performance across the heterogeneous suite.
  • Dexterous manipulation: Adroit evaluates Relocate and Reorient-Pen, where a 28-DoF anthropomorphic hand manipulates objects to target positions and orientations.The goal must be inferred from the scene.
  • Manipulation and locomotion: MetaWorld contributes five Sawyer-arm tabletop manipulation tasks, while DeepMind Control contributes five image-based locomotion and manipulation tasks.The included MetaWorld tasks are Assembly, Bin-Picking, Button-Press, Drawer-Open, and Hammer; DeepMind Control includes Finger-Spin, Reacher-Hard, Cheetah-Run, Walker-Stand, and Walker-Walk.
  • Multifinger manipulation: TriFinger contributes Push-Cube and Reach-Cube tasks requiring coordinated three-finger control to move a cube toward a goal.The robot has three fingers with 3 DoF per finger.
  • Navigation and mobile manipulation: Habitat evaluates image-goal and object-goal navigation in unseen photorealistic 3D environments, while Habitat 2.0 adds mobile manipulation with a Fetch robot.The Mobile-Pick task requires navigating to and grasping an object outside the robot’s initial reach.
  • Benchmark scope: The suite combines low-level imitation learning, large-scale reinforcement learning, semantic scene understanding, and fine-grained pose control.This diversity makes CortexBench a broad test of visual representations across embodied settings.

4 Do we already have a visual foundation model for EAI?

The evaluation compares representative frozen visual representations across CORTEXBENCH and finds that no single model performs consistently best across benchmarks. Several pre-trained models nevertheless outperform random-training baselines, with strengths varying by task.

  • Evaluation setup: The study evaluates CLIP, R3M, MVP, and VIP across CORTEXBENCH, spanning architectures, pre-training objectives, and datasets.Randomly initialized frozen and finetuned ViTs provide additional comparisons with training from scratch.
  • Overall finding: No single PVR consistently performs best across all CORTEXBENCH benchmarks.The table aggregates performance by benchmark and compares frozen pre-trained representations with prior results and random-training baselines.
  • Interpretation: Several pre-trained models often outperform a random training-from-scratch baseline, despite the absence of a universally dominant representation.The comparison is intended to separate representation quality from downstream task learning by using frozen visual features before Section 6.
  • Benchmark-specific strengths: R3M performs best on Adroit, MetaWorld, and DMControl, while MVP (ViT-L) leads TriFinger, ImageNav, and Mobile Pick.These results show that different representations achieve the strongest performance on different benchmarks.
  • Benchmark-specific strengths: CLIP achieves the best results on ObjectNav, adding another task-specific leader to the comparison.The variance across existing PVRs is also illustrated in Figure 5 in Appendix A.4.

5 Analyzing the Scaling Hypothesis for EAI

Scaling model size and pre-training data generally improve average CORTEXBENCH performance, but gains are not universal across tasks. VC-1 ranks best on average and surpasses prior PVRs overall, while task-specific models and adaptation remain important.

  • Model Size: Increasing model size generally improves average CORTEXBENCH performance, although ViT-B outperforms ViT-L on MetaWorld and TriFinger with Ego4D+MNI.The broad trend favors larger backbones, but the reported exceptions show that scaling model size is not uniformly beneficial.
  • Dataset Size and Diversity: Adding navigation data improves average performance more than adding similarly sized manipulation data: Ego4D+N gains 1.6 points for ViT-B and 3.6 points for ViT-L over Ego4D.The reported comparisons are 62.2 →63.8 for ViT-B and 63.5 →67.1 for ViT-L.
  • Dataset Size and Diversity: Adding about 800K frames to form Ego4D+MN yields only 0.3 and 0.1 additional points over Ego4D+N for ViT-B and ViT-L, respectively.These results support diversity as more influential than simply increasing dataset size in this comparison.
  • Dataset Size and Diversity: Adding ImageNet improves average performance by 1.9 points for ViT-B and 1.5 points for ViT-L when comparing Ego4D+MNI with Ego4D+MN.The corresponding results are 64.1 →66.2 and 67.2 →68.7.
  • Scaling Hypothesis Findings: VC-1, the ViT-L trained on Ego4D+MNI, achieves the best average rank at 2.4, ahead of Ego4D+MN ViT-L at 3.1.VC-1 is the largest model trained on all assembled datasets.
  • Comparison with Existing PVRs: VC-1 is not best on every benchmark: Mobile Pick favors Ego4D+M ViT-L and ImageNav favors Ego4D+N ViT-L.The findings indicate that task-specific pre-training datasets can improve individual-task performance but require multiple specialized models.

6 Adapting VC-1

The study evaluates end-to-end fine-tuning and MAE adaptation of VC-1, finding that their benefits depend on downstream data regime: MAE adaptation helps few-shot domains, while end-to-end fine-tuning is stronger with large-scale data.

  • End-to-end fine-tuning: End-to-end fine-tuning does not improve few-shot IL domains and can reduce performance, consistent with overfitting a 307M-parameter model on at most 50K frames.The reported hypothesis attributes this pattern to the small size of the downstream dataset.
  • Adaptation methods: VC-1 adaptation uses end-to-end fine-tuning or continued MAE training on task-specific data, followed by frozen-representation policy learning for MAE adaptation.MAE adaptation targets task specialization and domain-gap mitigation, while prompting-based adaptation is not studied.
  • MAE adaptation: +12.7 on Adroit (59.3 →72.0), +7.2 on MetaWorld (88.8 →96.0), +14.0 on DMC (66.9 →80.9), and +8.9 on TriFinger (71.7 →80.6) follow MAE adaptation.The gains occur in few-shot learning domains; in-domain MAE baselines also perform strongly on DMC and TriFinger.
  • Adaptation by data regime: In large-scale IL or RL domains, MAE adaptation produces small reductions, whereas end-to-end fine-tuning is the superior adaptation approach.This pattern is reported for ObjectNav, ImageNav, and Mobile Pick.
  • Benchmark outcomes: VC-1 with MAE adaptation is comparable to the best known results on MetaWorld, DMControl, and TriFinger, while end-to-end fine-tuning matches or exceeds them on ImageNav and Mobile Pick.Across the benchmark domains, adapting VC-1 improves performance overall.

7 Proof-of-Concept Hardware Experiments

The authors test VC-1 and its adaptations on TriFinger and Franka-Emika Panda hardware using few-shot imitation learning, observing trends similar to simulation.

  • Experimental setup: Hardware evaluation covers one TriFinger task and four Franka-Emika Panda tasks under a few-shot imitation-learning protocol.Demonstrations are collected directly in the real-world setup.
  • Results: Frozen VC-1 substantially outperforms both MVP and in-domain MAE across both robot setups.The comparison uses MVP as the strongest pre-existing PVR baseline and evaluates frozen, MAE-adapted, and end-to-end-adapted VC-1.

8 Discussion

The paper argues that CORTEXBENCH exposes the absence of a universal visual backbone for Embodied AI and supports broader benchmarks and more general representation learning.

  • Discussion: CORTEXBENCH contains 17 tasks spanning locomotion, indoor navigation, and dexterous and mobile manipulation, enabling a broad study of PVRs for Embodied AI.The study aims to connect research communities through perception as a shared cornerstone.
  • Discussion: The authors contend that embodied agents should ultimately use one visual cortex across diverse sensorimotor skills, environments, and embodiments.They speculate that temporal signals, 3D spatial priors, or objectness could help advance this goal.

A.1 Limitations

The study’s conclusions are bounded by benchmark coverage, the chosen MAE and ViT design space, adaptation costs, and the proxy nature of its evaluations for real robotic tasks.

  • Benchmark scope: CORTEXBENCH balances task diversity against evaluation cost, so newer challenging Embodied AI benchmarks may warrant future inclusion.The authors identify benchmark expansion as an ongoing limitation.
  • Design scope: The study evaluates masked auto-encoders and vision transformers, leaving open whether other self-supervised algorithms or architectures scale differently or perform better.This limits conclusions about the broader design space of visual foundation models.
  • Adaptation cost: Each adapted PVR requires separate in-domain training and hyperparameter tuning, increasing effort with the number of benchmarks.The limitation applies to adaptation procedures involving training epochs and dataset sampling ratios.
  • Real-world validity: Benchmark performance may not preserve model rankings on real robots because the evaluations are proxies for robotic tasks.The authors call for further real-world evaluation.

A.2 Overview of Downstream Policy Learning in CORTEXBENCH

CORTEXBENCH pairs frozen visual representations with task-specific policy learning, using imitation learning or reinforcement learning and architecture choices matched to each task domain.

  • Policies are trained for each task because agents must learn a task-specific policy given a frozen PVR.The benchmark follows community-standard policy-learning algorithms for state-of-the-art performance in each domain.
  • MuJoCo Tasks: MuJoCo tasks use behavior cloning from 25 or 100 expert demonstrations, with frame stacking and a 3-layer MLP policy.ViT-based PVRs provide the [CLS] token, while ResNet-based PVRs provide final-layer features.
  • TriFinger Tasks: TriFinger Reach-Cube and Push-Cube policies use behavior cloning on 100 demonstrations per task and a 3-layer MLP.Training lasts 100 epochs for Reach-Cube and 1,000 epochs for Push-Cube, using [CLS] or average-pooled convolutional features.
  • Habitat Tasks: Habitat ObjectNav policies use behavior cloning, while ImageNav and Mobile-Pick use reinforcement learning for 500M environment steps.ViT patch representations or ResNet grid-features are compressed before entering 2-layer LSTM navigation or 2-layer GRU manipulation policies.

A.3 Additional Details of Tasks and Downstream Learning in CORTEXBENCH

The appendix specifies benchmark environments, training procedures, scaling datasets, and analyses showing that representation performance varies substantially across tasks while size and diversity generally improve average performance.

  • ImageNav: ImageNav uses Gibson environments with 72 training scenes and 14 validation scenes, totaling 4,200 validation episodes.Agents use 128×128 RGB observations, can take up to 1000 steps, and succeed within 1m of the goal after STOPACTION.
  • ImageNav: ImageNav training uses 500M timesteps with 320 parallel environments, PPO updates, AdamW, and a learning rate of 2.5 × 10^-4.The training setup uses 25k updates and weight decay of 10^-6.
  • ObjectNav: ObjectNav evaluates HM3D-SEM scenes and deems episodes successful when the agent stops within 0.1m of a valid visible viewpoint near the target object.Training uses 77k human demonstrations and approximately 400 million environment steps.
  • Mobile Pick: Mobile Pick requires a Fetch robot to navigate a ReplicaCAD apartment and grasp a target object using egocentric RGB, proprioceptive, and grasping sensors.The task combines mobile navigation with manipulation in a cluttered receptacle.
  • Representation scaling: Existing PVRs show high cross-benchmark variability, with no single representation consistently ranking best.Figure 5 visualizes each model’s rank across seven benchmarks, while Figure 6 provides rank distributions for all evaluated models.
  • Scaling hypothesis results: Greater dataset diversity reduces rank variance, and Ego4D+MNI consistently performs well across benchmarks among the top models.OH24 contributes 139 hours of walkthrough video from over 1600 homes to the scaling datasets.
  • Adaptation analysis: End-to-end adaptation makes VC-1 attention focus on task-relevant regions while dropping attention from irrelevant image areas.The cited analysis gives object manipulation as an example of task-relevant focus.
Loading 2303.18240v2…