PaperlayerView exact revision on arXiv

Source-linked AI summary

Compact Robot Policies Need Fine-Grained Visual Representations

Nanhe Chen, Runqiu Yang, Jiawei Tang, Sichao Liu, Yuquan Wang

arXiv:2610.08183v1cs.ROcs.AIcs.LG

TL;DR

Published multi-task manipulation comparisons vary architecture, scale, and pretrained priors together, leaving the source of their performance difficult to isolate. CoRP holds the action generator fixed while testing representation properties, and achieves competitive results across LIBERO and RoboTwin 2.0 with a compact policy.

  • Problem

    Multi-task manipulation policies vary architecture, scale, and pretrained priors together, preventing comparisons from attributing performance to a single component.

    Source: §1 · pp. 1–2

  • Method

    CoRP is a compact policy that factorizes representation extraction from flow-matching action generation, with controlled tests varying one extractor property at a time.

    Source: §1 · pp. 1–2

  • Results

    97.0% on LIBERO and 75.8%/73.4% on RoboTwin 2.0 Clean/Randomized; CoRP is competitive with systems up to 163× larger.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Effective compact policies use pretrained, task-adapted, compressed representations; language matters when observations leave the goal ambiguous.

    Source: §1 · p. 2, §5 · p. 9

    The evaluation is limited to closed-set benchmarks and does not assess the open-world semantic and physical generalization targeted by VLAs and WAMs.

    Source: §5 · p. 9

Abstract

from arXiv · show

Multi-task manipulation policies differ in architecture, scale, and pretrained priors all at once, so published comparisons cannot attribute performance to any single component. We argue that most of it comes from the visual representation, and that parameter scale and generative priors are largely incidental. To test this, we build CoRP (Compressed Representation Policy), a deliberately compact policy (48.9M parameters, no vision-language model and no video-generative prior) that factorizes into a representation extractor and a flow-matching action generator. It reaches 97.0% on LIBERO and 75.78%/73.36% on RoboTwin 2.0 Clean/Randomized, matching systems 40.9-163.6x larger. Holding the action generator fixed, we then vary one extractor property at a time. Pretrained initialization is decisive: a random ViT-S/14 drops to 78.1% and an ImageNet ResNet-34 to 74.5% on LIBERO. Pretraining alone is not enough, as freezing the encoder costs 19.8 points. Compression matters as much: resampling each view to 48 tokens beats passing all patch tokens (97.0% vs 83.2%), and a variational information bottleneck over those tokens is worse than a hard token budget, cutting LIBERO-Goal from 95.8% to 33.0% by suppressing the instruction-dependent token selection the policy relies on. Language conditioning contributes only where the observation leaves the goal ambiguous (LIBERO-Goal: 9.2% to 95.8%), while on RoboTwin 2.0, where observations are unambiguous, removing it slightly improves success. Therefore, we argue that a compact policy works when its representation is pretrained, task-adapted, and compressed. Project page: https://corp-policy.github.io/

1 INTRODUCTION

Published multi-task policy comparisons vary architecture, scale, pretraining, and generative priors together, so they cannot isolate what drives performance. CoRP provides a compact testbed for varying representation properties while holding the action generator fixed.

1 INTRODUCTION

  • Holding the action generator fixed, the study varies visual-encoder initialization, adaptation, token count, and instruction availability to isolate representation properties.The central question is how much policy performance depends on the representation and which of its properties matter.Source: §1 · pp. 1–2
  • 48.9M-parameter CoRP reaches 97.0% average success on LIBERO and 75.8%/73.4% on RoboTwin 2.0 Clean/Randomized, competing with systems up to 163× larger without a VLM or video-generative prior.The paper presents this benchmark result as a starting point for its controlled interventions.Source: §1 · p. 2
  • The interventions identify the extractor as the main performance source and find that effective compact-policy representations are pretrained, task-adapted, and compressed.The study is restricted to closed-set benchmarks and does not evaluate the open-world generalization targeted by VLAs and WAMs.Source: §1 · p. 2
  • Real-robot validation covers three bimanual tasks and provides qualitative evidence that CoRP transfers to hardware after large-scale real-world pretraining.The tasks include adaptation to object displacement, coordinated two-arm placement, and precision plug insertion.Source: §1 · p. 2

2 RELATED WORK

Prior work uses generative action heads, multi-task policy scaling, pretrained vision-language models, and video priors, but these components often change together. Related visual-representation studies leave open how initialization, fine-tuning, and compression jointly affect multi-task, language-conditioned generative policies.

2 RELATED WORK

  • Diffusion Policy and ACT use generative action heads for single-task control, while Octo scales across tasks and embodiments but learns its visual tokenizer from robot data.VLAs add pretrained vision-language representations, and WAMs add video-generative priors; across these lines, multiple policy factors change together.Source: §2 · p. 2
  • Supervised pretraining has established convolutional and patch-based Transformer backbones as transferable visual features, while DINO methods yield fine-grained patch features [Caron et al., 2021; Oquab et al., 2024; Siméoni et al., 2026].Source: §2 · p. 2
  • Prior representation studies often use frozen features with single-task or lightweight heads, leaving initialization, fine-tuning, and token compression jointly untested in multi-task language-conditioned generative policies.The paper addresses this gap by comparing initialized, random, and supervised encoders, frozen and fine-tuned, under one action generator.Source: §2 · p. 3

3 METHOD

CoRP separates representation extraction from conditional action generation, combining multi-view visual features, language, and state into a conditioning sequence. Its training uses flow matching for action chunks and an auxiliary task-classification objective.

3.1 POLICY ARCHITECTURE

  • CoRP maps multi-view observations, proprioceptive state, and instruction to a conditioning sequence that a flow-matching DiT-style generator uses to model an action chunk.The architecture comprises a representation extractor and conditional action generator.Source: §3.1 · p. 3, Figure 1 · p. 3
  • Each camera view is independently encoded by a shared DINOv2-initialized ViT-S/14, producing 256 patch tokens at 224 × 224 resolution that are fine-tuned with the policy.The class token is discarded before the patch tokens enter the resampler.Source: §3.1 · p. 3
  • A fine-tuned T5-Efficient-Tiny encoder supplies instruction tokens and a pooled embedding that conditions both the resampler and the action generator.Source: §3.1 · p. 3
  • A learned-query cross-attention resampler compresses each view’s patch tokens to 48 visual tokens, with language-conditioned queries and a state token included in the resulting representation.The fixed visual-token capacity is used to test the effect of compression.Source: §3.1 · pp. 3–4

3.2 TRAINING OBJECTIVE

  • Flow matching forms a straight interpolation from sampled noise to the demonstrated action chunk and trains the generator to predict its constant target velocity.At inference, ten Euler steps produce the predicted chunk, whose actions are executed before replanning.Source: §3.2 · p. 4
  • An auxiliary task-classification loss encourages task-relevant information in the language-conditioned visual prefix, while the task head is used only during training.The flow-matching and task-classification terms are weighted in the overall objective, and modules are trained end-to-end.Source: §3.2 · pp. 4–5

4 EXPERIMENTS

The experiments compare CoRP on LIBERO and RoboTwin 2.0, then hold the action generator fixed to test representation choices. They also evaluate the policy on three bimanual tasks on real robots.

4.1 SETUP

  • LIBERO supports controlled representation comparisons on its demonstration distribution, while RoboTwin 2.0 tests whether findings extend to bimanual control.RoboTwin 2.0 has Clean and Randomized settings, with training demonstrations provided for both.Source: §4.1 · p. 5

4.2 CAN A COMPACT POLICY MATCH MUCH LARGER SYSTEMS?

  • CoRP is competitive on LIBERO and, on RoboTwin 2.0 Clean/Randomized, scores 75.8%/73.4%, beating X-VLA and π0.5 while approaching Motus without video pretraining.It uses about 160× fewer parameters than that Motus variant, but trails pretrained Motus and GigaWorld-Policy.Source: Table 2 · p. 6, §4.2 · p. 6

4.3 DOES A PRETRAINED VISUAL ENCODER HELP?

  • 97.0% average LIBERO success: the DINOv2-initialized ViT-S/14 leads every suite, while ImageNet ResNet-34 reaches 37.0% on Goal despite 100% task identification.The pretrained ViT also yields the most action-decodable visual prefix; the authors note that a supervised-pretrained ViT control remains future work.Source: Table 3 · p. 6, §4.3 · pp. 6–7

4.4 IS PRETRAINING ENOUGH WITHOUT ADAPTATION?

  • 19.8 points: freezing the pretrained encoder lowers average LIBERO success from 97.0% to 77.2%, matching a randomly initialized ViT trained end-to-end at 78.1%.Frozen features have lower action-probe R2 (0.24 vs. 0.81); fine-tuning also yields more stage-ordered features in the t-SNE analysis.Source: §4.4 · p. 7, Figure 2 · p. 7

4.5 DO WE NEED LANGUAGE CONDITIONING?

  • Language disambiguates ambiguous goals: without it, LIBERO-Goal task-head accuracy and success are 8.6% and 9.2%; removing it raises RoboTwin success by about 2.7 points in both settings.With language, task-head accuracy reaches 100.0% on every suite; on RoboTwin, task accuracy stays near 100% either way.Source: §4.5 · pp. 7–8

4.6 DOES THE INFORMATION BOTTLENECK HELP?

  • 95.8% vs. 33.0% LIBERO-Goal success: adding VIB to the resampled prefix suppresses instruction-driven change from 49.7% to 13.6%.The hard, instruction-aware bottleneck raises average LIBERO success from 83.2% to 97.0%, whereas VIB lowers it with or without resampling.Source: Table 7 · p. 8, §4.6 · p. 8

4.7 CAN CORP WORK ON REAL ROBOTS?

  • 12/12 pick-cup trials succeeded, including 8/8 after manual object displacement; the policy retargeted within the next action chunk.It also succeeded in 9/10 spoon placements and completed charger insertion in 2 trials.Source: Figure 3 · p. 8, §4.7 · p. 9, Figure 4 · p. 9

5 CONCLUSION AND LIMITATIONS

CoRP uses a compact policy to study where multi-task manipulation performance comes from, concluding that its visual representation must be pretrained, task-adapted, and compressed. The study is limited to closed-set evaluation, and its hardware experiments do not isolate the representation’s role.

5 CONCLUSION AND LIMITATIONS

  • CoRP is a 48.9M-parameter policy competitive with systems up to 163× larger on LIBERO and RoboTwin 2.0, and it transfers unchanged to real bimanual robots.The authors attribute compact-policy performance primarily to the representation extractor, while language matters when scenes leave the goal ambiguous.Source: §5 · p. 9
  • Closed-set benchmarks leave open whether pretrained, task-adapted, compressed representations suffice for the open-world semantic and physical generalization targeted by VLAs and WAMs.The study does not evaluate that regime.Source: §5 · p. 9
  • Without a supervised-pretrained ViT control, the encoder ablation cannot distinguish DINOv2’s patch-level pretraining benefit from the benefit of pretraining more generally.The authors identify adding this control as the next direct step.Source: §5 · p. 9
  • The real-robot experiments cover three tasks, have no real-robot baselines, and follow large-scale pretraining, so they show transfer but do not isolate the representation’s role.Repeating controlled interventions on hardware would address this gap.Source: §5 · pp. 9–10

AI USE STATEMENT

The authors disclose using LLMs to polish the writing and assist with software development, while stating that they reviewed the final text and developed the core algorithmic implementation.

AI USE STATEMENT

  • LLMs assisted with writing and software tasks, including debugging and scripts for preprocessing and visualization; the authors reviewed and edited the final text and developed the core algorithm.The statement describes the LLMs’ role as primarily improving productivity and quality.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The statement describes the paper’s experimental documentation and distinguishes reproducibility support for simulation from limits on reproducing real-world results.

REPRODUCIBILITY STATEMENT

  • The simulation experiments use publicly available LIBERO and RoboTwin 2.0 datasets, and the authors plan to release simulation code and configurations with the camera-ready version.Sections 3 and 4 describe the architecture, training objectives, benchmarks, and metrics.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Proprietary real-world training datasets cannot be released because of commercial restrictions, limiting independent reproduction of those results.Source: REPRODUCIBILITY STATEMENT · p. 10

A ADDITIONAL EXPERIMENTAL DETAILS

The appendix section provides additional experimental details.

A ADDITIONAL EXPERIMENTAL DETAILS

A.1 POLICY ARCHITECTURE

CoRP extracts visual, language, and state representations for a flow-matching action generator. Its visual resampler compresses each camera view, while additional probes and an instruction-swap measure assess representation properties.

A.1 POLICY ARCHITECTURE

  • T5-Efficient-Tiny encodes instructions, with a LayerNorm and linear adapter mapping its output to the policy width.The encoder uses fixed-length tokenization, masked mean pooling, and zeroed padded rows.Source: §A.1 · p. 15
  • Each camera view is compressed into 48 learned query tokens, and the default visual encoder is fine-tuned DINOv2 ViT-S/14.The encoder processes 224 × 224 images as 256 patch tokens per camera; its weights are shared across views.Source: §A.1 · p. 15
  • ∆prefix measures the relative change in resampled visual prefixes when an instruction is replaced with one from another dataset example.The instruction-swap experiment uses N = 8000 and the same swap pairs for all variants.Source: §A.1 · p. 15
  • The action generator conditions on visual, language, and state tokens; action queries attend bidirectionally to the action chunk and valid prefix keys.Visual-prefix queries cannot attend to noisy action keys, and language padding is masked.Source: §A.1 · p. 15
  • The archived variational information bottleneck is applied to resampled visual tokens before language tokens are concatenated.Its token-level Gaussian uses a standard-normal prior and βIB = 4 × 10−8.Source: §A.1 · p. 15
  • The action probe freezes each representation extractor and predicts action chunks on held-out LIBERO episodes to assess readable action information.Episodes are split 80% for training and 20% for testing; NMAE and R2 are reported.Source: §A.1 · pp. 15–16

A.2 INFERENCE COST

CoRP has low measured inference latency and a compact GPU-memory footprint in the reported configuration.

A.2 INFERENCE COST

  • 7.01 ms is the default unoptimized inference time, reduced to 5.79 ms with CUDA Graph capture and replay.The reported results use unaccelerated inference because it is already fast enough.Source: §A.2 · p. 16
  • 3.43 ms of the unoptimized inference time is attributed to representation extraction and about 3.08 ms to action generation.Both components contribute to the reported 7.01 ms full inference time.Source: §A.2 · p. 16
  • 95.4 MB is the total GPU memory occupied by model weights stored in BF16.DINOv2-S, T5, and DiT account for 42.1 MB, 23.7 MB, and 23.7 MB, respectively.Source: §A.2 · p. 16

A.3 SIMULATION

The appendix describes LIBERO and RoboTwin 2.0 simulation protocols, training-loss behavior, and a precision peg-insertion evaluation.

A.3 SIMULATION

  • The policy uses a fine-tuned DINOv2 ViT-S/14 encoder, a 48-token-per-camera resampler, T5-Efficient-Tiny, and a one-block DiT action generator.The complete policy has about 48.9M parameters; training uses 80k steps on LIBERO and 120k on RoboTwin 2.0.Source: §A.3 · p. 16
  • Lower training loss does not necessarily correspond to better policy performance; the best-performing configurations often have the highest training losses.Freezing the encoder is an exception, raising training loss while lowering success; validation losses are not reported because the full dataset is used for training.Source: §A.3 · p. 16, Figure 6 · p. 17
  • LIBERO evaluation covers 40 tasks across four suites, with 50 randomized-initial-condition trials per task and language-conditioned representation visualizations.Figure 7 shows task clusters with language; without it, Goal tasks collapse into one cloud while Object tasks remain separable.Source: §A.3 · pp. 16–17, Figure 7 · p. 18
  • RoboTwin 2.0 evaluation covers 50 tasks in Clean and Randomized settings, with 100 evaluation repetitions per task.The settings differ in evaluation conditions while sharing the same training setup, for 10,000 trials total.Source: §A.3 · p. 17
  • On Forge peg insertion, using visual information only and training from scratch for 30k steps, the method achieved an 84% success rate over 50 trials, similar to the best result reported in.Source: §A.3 · p. 18

A.4 BASELINE POLICIES

The appendix surveys baseline policy families and includes real-world experiments using CoRP on dual-arm robots. The Forge figure reports 84% peg-insertion success, while the real-world setup uses extensive pretraining before task fine-tuning.

A.4.1 POLICIES BASED ON PRETRAINED VISION–LANGUAGE MODELS

  • The baseline comparison groups policies by pretrained vision-language or video-generation backbones, rather than by language input or pretraining in general.The listed VLM-based policies include OpenVLA, SmolVLA, π0, π0.5, OpenVLA-OFT, and X-VLA, which vary in action decoding, inputs, and training designs.Source: §A.4 · p. 18, §A.4.1 · pp. 18, 20, 31
  • X-VLA uses domain-specific soft prompts, while π0.5 can condition action generation on intermediate language subtasks.The rollout figures show CoRP examples across LIBERO tasks and RoboTwin Clean and Randomized task groups.Source: Figure 8 · p. 19, Figure 9 · p. 20, Figure 10 · p. 21

A.4.2 POLICIES BASED ON PRETRAINED VIDEO-GENERATION MODELS

  • The video-generation baseline group includes models that couple video and action generation or use future-video prediction as an auxiliary training objective.Motus transfers multimodal priors through staged training; Fast-WAM-Joint retains future-video generation at inference, whereas GigaWorld-Policy omits it.Source: §A.4.2 · p. 32

A.4.3 POLICIES WITHOUT EITHER PRETRAINED BACKBONE

  • Baselines without either pretrained backbone include ACT, Diffusion Policy, Octo, and ProgVLA, spanning single-task generative heads, multi-robot pretraining, and compressed representations.ProgVLA is included only on LIBERO because it does not currently release code.Source: §A.4.3 · p. 32

A.5 REAL-WORLD

  • Real-world experiments use two dual-arm robots and three cameras, with more than 10,000 hours of diverse pretraining followed by fine-tuning on 10 tasks.The policy uses synchronous inference at about 20 ms and runs place-spoon, pick-cup, and plug-insertion tasks.Source: §A.5 · p. 33
  • The 20D bimanual action encodes left and right end-effector positions, rotations, and gripper states as chunk-relative delta states.Relative rotations are computed on the SO(3) manifold; the rollout figure shows spoon placement, cup picking, and plug insertion.Source: §A.5 · p. 33, Figure 21 · p. 35
  • 84% success over 50 trials is reported for vision-only peg insertion on Forge, using an 8.0 mm peg and an 8.1 mm hole.The figure shows rollouts from different initial conditions.Source: Figure 20 · p. 34

Related papers

Loading 2610.08183v1…