PaperlayerView exact revision on arXiv

Source-linked AI summary

MINERVA: How Small Can a Manipulation Policy Be and Still Solve LIBERO?

Kohei Sendai, Tatsuya Matsushima, Yusuke Iwasawa

arXiv:2609.03715v1cs.RO

TL;DR

Large VLAs dominate LIBERO, but the benchmark’s required model capacity is unclear. MINERVA measures that task-specific floor with compact, language-free visuomotor policies and finds that 0.54M parameters achieve 95.1% average success, while robustness falls sharply under perturbations.

  • Problem

    LIBERO is dominated by large VLA models, but the model capacity actually required by this benchmark remains unclear.

  • Method

    MINERVA scales a from-scratch CNN policy with a learned task-ID embedding and compact action head across architectures, objectives, inference strategies, and robustness probes.

  • Results

    0.54M parameters achieve 95.1% average success on the four standard LIBERO suites, while the recipe reaches 94.6% on LIBERO-90 and 46–56% under LIBERO-Plus perturbations.

  • Takeaways & Limitations

    Standard LIBERO can be solved by roughly 0.5M parameters of task-indexed visuomotor memorization, supporting capacity-aware design and distillation for deployment-efficient policies.

  • Takeaways & Limitations

    MINERVA is explicitly limited to standard LIBERO and lacks language understanding, open-vocabulary perception, and transfer to unseen tasks.

Abstract

from arXiv · show

Vision-language-action (VLA) models with billions of parameters now dominate the LIBERO manipulation benchmark, but the model capacity actually required by the benchmark remains unclear. We introduce MINERVA (MINimal Efficient Robotic Vision-Action policy), a family of deliberately compact visuomotor policies designed to measure this task-specific capacity floor. A 0.54M-parameter policy achieves 95.1% average success over 2,000 rollouts on the four standard LIBERO suites, only 2.4 points below the reported LeRobot $\pi_{0.5}$ result despite using 7,700$\times$ fewer parameters. Performance saturates near 1M parameters and collapses below 0.25M. Across broad architectural, training, and inference sweeps, only action-chunk length and vision capacity consistently exceed a $\pm$1-point training-seed band. Flow matching provides no detectable advantage over direct L1 regression across three seeds, while regression is up to 3.8$\times$ faster on GPU. A task-ID permutation probe shows that standard LIBERO instruction conditioning primarily selects among memorized tasks: changing only the task-ID mapping reduces success to near chance. The same recipe achieves 94.6% success across 89 LIBERO-90 tasks, while LIBERO-Plus perturbations reduce performance to 46--56%, with near-zero robustness to photometric shifts. The 0.54M policy replans every control step in 5--9 ms per chunk on a laptop CPU, 113$\times$ faster than SmolVLA and 1,400$\times$ faster than $\pi_{0.5}$, without a GPU. These results establish a first empirical estimate of LIBERO's task-specific capacity floor and motivate capacity-aware design and distillation for deployment-efficient robot policies.

I. INTRODUCTION

MINERVA asks how little capacity standard LIBERO requires and tests whether compact, task-indexed visuomotor policies can match much larger VLAs. Its results identify a sub-million-parameter floor, robust design effects, and benchmark limitations.

  • Method: MINERVA reaches this result with a from-scratch CNN, a 40-entry task-ID embedding, and no language encoder or pretrained perception.The design follows the benchmark’s same-task, same-scene evaluation setting, where language primarily identifies memorized tasks.
  • Results: 0.54M parameters achieve 95.1% average success across four standard LIBERO suites, only 2.4 points below LeRobot π0.5 despite 7,700× fewer parameters.At 0.99M parameters, performance is within 0.75 points of the reported baseline.
  • Results: Success saturates near 1M parameters and collapses below 0.25M, with long-horizon tasks failing before short-horizon tasks.The pattern localizes the benchmark’s tested capacity floor and suggests that chaining subgoals is especially capacity-sensitive.
  • Ablations: Across three seeds, only action-chunk length and vision allocation exceed the ±1-point seed band; flow matching offers no detectable advantage over direct L1 regression.Regression uses one forward pass rather than ten Euler steps, while chunk-length effects differ across goal and long-horizon suites.
  • Scope and robustness: Permutation, LIBERO-90, and LIBERO-Plus studies show task-ID dependence, 94.6% success on 89 tasks, and only 46–56% success under perturbations.Photometric robustness remains near zero across tested scales.
  • Inference: Inference choices also matter: temporal ensembling outperforms the compared chunking strategies, and the 0.54M policy runs in 8.9 ms per chunk on laptop CPU.This is reported against 1.0 s for SmolVLA and 12.8 s for π0.5 under the stated protocol.

II. RELATED WORK

Prior work makes LIBERO accessible through large pretrained VLA systems or smaller efficient variants, while MINERVA instead measures the benchmark’s minimum task-specific capacity. It combines compact visuomotor design choices with established chunking and generative-head ideas.

  • VLA policies and LIBERO: Standard LIBERO leaders use billion-scale pretrained VLA backbones, while MINERVA tests whether the benchmark can be solved with far less capacity.The related work frames standard LIBERO scores as largely measuring memorization and motivates replacing language with a task index.
  • Efficient VLAs: Efficient VLA methods reduce inference cost but retain language pathways or pretrained components and remain at least 0.2B parameters.TurboVLA reaches 97.7% on LIBERO with 0.2B parameters, still more than two orders of magnitude above MINERVA’s 0.54M model.
  • Novelty: MINERVA differs by removing open-vocabulary capability and asking how much capacity the benchmark itself demands rather than how to make a general VLA cheaper.Its headline model trades language and pretrained components for a much smaller task-specific policy.
  • Small visuomotor policies: The design inherits action chunking and temporal ensembling from ACT, generative action modeling from Diffusion Policy, and keypoint extraction from earlier CNN policies.The paper combines these precedents while pushing total capacity below prior sub-100M approaches.
  • Inference-time chunking strategies: The study evaluates temporal ensembling, BID, RTC, and temperature control as alternative ways to execute chunked predictions.These methods represent distinct inference-time strategies for handling overlapping or newly sampled action chunks.

III. METHOD

MINERVA is deliberately limited to the capabilities exercised by standard LIBERO: visual state estimation, closed-loop action generation, and selecting among 40 known tasks. The language pathway is replaced by a learned task-ID representation.

  • Method: MINERVA replaces language conditioning with a learned 40-entry task-ID embedding because standard LIBERO uses instructions mainly to identify known tasks.The full budget is allocated to visual state estimation and closed-loop action generation.

B. Architecture

The architecture combines compact visual and proprioceptive encoders with a chunked action head, using conditional flow matching or direct regression. Width, attention, and distillation determine the model-scale variants.

  • Perception: Both camera views share a from-scratch depthwise-separable CNN with task-conditioned FiLM, spatial-softmax keypoints, and a separate MLP for proprioception.The resulting features, state representation, and task embedding are combined across two observation steps.
  • Action head: Actions are generated in 16-step chunks using conditional flow matching, integrating the learned velocity field from noise with 10 Euler steps.The interpolant combines actions and Gaussian noise at a sampled noise level.
  • Action head: A token-mixing MLP replaces self-attention over action tokens, and shared AdaLN-Zero modulation reduces the head’s parameter cost.The same head can instead use direct L1 regression, which requires one inference pass and shows no detectable benchmark advantage for flow matching.
  • Model family: The model family scales from 0.54M to 9.66M parameters by changing widths and progressively restoring full-width CNNs, head attention, and unshared AdaLN.The teacher used for velocity distillation is excluded from parameter counts and absent at inference.
  • Distillation: Velocity distillation strongly improves the 5M comparison by 1.85 average points, while its 1M effect remains within the reported seed band.The distilled 4.89M model also exceeds its 7.12M teacher in the cited single-run comparison.

C. Inference-time strategy

MINERVA evaluates inference-time execution choices through stepwise replanning, temporal ensembling, and mode-seeking sampling. The supplied passages define the deployment protocol but do not report its comparative outcomes.

  • C. Inference-time strategy: At deployment, the policy replans every control step and averages overlapping chunk predictions with exponentially decaying weights.Each executed action averages predictions from all chunks covering that timestep, with decay 0.01 per step of age.
  • C. Inference-time strategy: For sub-1M flow models, scaling initial noise by σ = 0.85 makes sampling mode-seeking.The passage states that this setting is validated in the inference-time evaluation.
  • C. Inference-time strategy: The standard evaluation uses 2,000 rollouts across four ten-task suites with fixed seeds and hard environment resets.This protocol comprises 50 episodes per task across Spatial, Object, Goal, and Long/“10”.

V. RESULTS

MINERVA reaches near-baseline LIBERO performance with a compact policy, while performance saturates near 1M parameters and degrades sharply below 0.25M. The results section evaluates this scaling behavior using full-suite rollouts and three-seed ablations.

  • V. RESULTS: 95.05% average success at 0.54M parameters is 2.4 points below π0.5, while 0.99M reaches within 0.75 points.The π0.5 comparison uses a reported reference measured with a different rollout protocol.
  • V. RESULTS: Performance saturates near 1M parameters; below 0.25M it collapses, with the long-horizon suite falling first.Figure 3 reports success versus total parameters over 2,000 rollouts per point.
  • V. RESULTS: Table II’s top ablations are full 2,000-rollout evaluations retrained with three seeds.The table caption identifies the scale, evaluation size, and seed-retraining scope.

BOTTOM: SINGLE-RUN DELTAS (SEED 1000) — SUGGESTIVE ONLY

Seed-replicated results show that most single-run ablation differences fall within a ±1-point variability band. Chunk length, vision allocation, and the regression-versus-flow choice receive the strongest supported comparisons in the supplied passages.

  • BOTTOM: SINGLE-RUN DELTAS (SEED 1000) — SUGGESTIVE ONLY: A single training seed moves the 4-suite average by ±1 point, making many one-point single-run ablation deltas difficult to distinguish.The baseline spans 94.60–96.75, while its long-suite score spans 87.6–92.8.
  • BOTTOM: SINGLE-RUN DELTAS (SEED 1000) — SUGGESTIVE ONLY: Chunk 8 reduces average success by 3.16 points and chunk 32 by 2.20 points, whereas chunk 16 sacrifices neither suite.Chunk 8 breaks Goal while chunk 32 breaks Long, reflecting different temporal demands.
  • BOTTOM: SINGLE-RUN DELTAS (SEED 1000) — SUGGESTIVE ONLY: 94.22 ± 0.20 results from starving vision to 0.13M parameters, a replicated −1.41-point mean difference concentrated in Long by −4.6 points.The starved configuration is more seed-stable, with standard deviation 0.20 versus 1.08.
  • BOTTOM: SINGLE-RUN DELTAS (SEED 1000) — SUGGESTIVE ONLY: Flow matching is indistinguishable from direct L1 regression across three seeds, while regression reduces inference time by up to 3.8×.At 0.54M, regression scores 95.75% versus 95.05% for flow matching, within the seed band.
  • BOTTOM: SINGLE-RUN DELTAS (SEED 1000) — SUGGESTIVE ONLY: Conditioning style, 1M distillation, and restored self-attention remain within the seed band, while the mixer head matches the score with 26% fewer parameters.The restored self-attention change is reported as +0.05 points at +26% parameters.

D. Where to spend a fixed budget: the eyes, not the head

Fixed-budget experiments identify vision allocation as the load-bearing capacity choice, while inference execution benefits from short horizons, temporal ensembling, and mode-seeking sampling. These choices jointly target accuracy and deployment cost.

  • D. Where to spend a fixed budget: the eyes, not the head: Allocating vision 50–80% of a roughly 1M budget stays within 96.3–96.8%, but starving vision to 0.13M costs 1.41 points overall and 4.6 points on Long.The passage also reports a 10-point Long-suite improvement when parameters move from the attention head into the CNN at constant total budget.
  • D. Where to spend a fixed budget: the eyes, not the head: For every execution method, horizons of 1–2 actions per replan outperform 4, which outperforms 8.The passage identifies execution horizon as the dominant regularity in the strategy comparison.
  • D. Where to spend a fixed budget: the eyes, not the head: Table III reports inference cost per action chunk on an RTX 5080 laptop GPU and eight CPU threads with batch size 1.The supplied passage identifies the hardware and measurement unit but not the table’s individual cells.
  • D. Where to spend a fixed budget: the eyes, not the head: At horizon 1, temporal ensembling reaches 96.3%, ahead of BID at 95.0% and plain chunking at 94.1%, while RTC falls to 91.8%.The comparison uses the same 0.54M checkpoint; RTC soft inpainting is described as over-constraining a noisy velocity field.
  • D. Where to spend a fixed budget: the eyes, not the head: Temperature 0.85 adds 0.2–2.25 points on sub-1M models but adds nothing at 1M and above.The gain is largest where the model is weakest.

F. Efficiency: closing the loop on a CPU

MINERVA makes per-step replanning practical on a laptop CPU, while task-ID conditioning supplies the dominant task-selection signal on standard LIBERO.

  • CPU efficiency: 113× and 1,400×: the 0.54M model is faster than SmolVLA and π0.5 on identical hardware, enabling CPU-only replanning.It runs at 8.9 ms per chunk on eight laptop CPU threads, or approximately 5 ms with the regression head.
  • CPU efficiency: The 0.54M policy can replan every control step without a GPU, unlike the much slower VLA baselines.The reported protocol supports approximately 100 Hz control-loop replanning for MINERVA, versus approximately 1 Hz for SmolVLA and 0.08 Hz for π0.5.
  • Task conditioning: 96.75% falls to 6.5% when task IDs are rotated, showing that task-ID conditioning selects the executed task.All weights remain fixed while only the task-ID mapping changes.

H. Beyond 40 tasks: LIBERO-90

The task-indexed recipe extends beyond the 40-task benchmark: it retains standard-LIBERO-level success on LIBERO-90, but perturbations expose substantial robustness limits.

  • LIBERO-90: 94.6% success is achieved over 89 LIBERO-90 tasks with the 0.995M-parameter architecture, despite shared instruction IDs across some tasks.The evaluation uses 3,959 demonstrations across 90 tasks, with one task lacking demonstrations.
  • LIBERO-Plus: 46.7%, 46.0%, and 55.7%: the 0.54M, 0.99M, and 4.89M models fall sharply on LIBERO-Plus perturbations.Language variants lose approximately 0 points, indicating that the drop is driven by visual or environmental perturbations rather than instruction rephrasing.

VI. DISCUSSION AND LIMITATIONS

The discussion interprets MINERVA as a measurement of LIBERO’s task-specific capacity floor rather than a generalist VLA, while documenting robustness, reproducibility, and real-robot scope limits.

  • What this says about the benchmark: Standard LIBERO is satisfiable by approximately 0.5M parameters of task-indexed visuomotor memorization, while perturbation protocols expose what that memorization omits.The permutation probe identifies task ID as the task-selection signal, and LIBERO-Plus measures robustness outside the standard setting.
  • A capacity floor for deployment: Approximately 0.5M parameters suffice for standard LIBERO, with approximately 1M providing long-horizon headroom, but the floor is task-set-specific.LIBERO-Plus suggests that robustness may require greater capacity, with the 4.89M model improving semantic robustness over sub-1M models.
  • Limitations: The headline models, LIBERO-90 run, remaining sweep points, and distillation comparison are single runs within a ±1-point seed band, using one simulator and one evaluation seed.Only five key ablation configurations are evaluated across three training seeds.
  • Limitations: The models retain roughly half their standard score under LIBERO-Plus perturbations, fail hardest on photometric shifts, and lack real-robot validation.The authors identify photometric augmentation as a potentially addressable gap and leave physical validation for future work.
  • Conclusion: The study concludes that measuring a task set’s empirical floor can guide direct construction or distillation of deployment-efficient policies.The proposed deployment pipeline is to train or reuse a larger model, distill to the floor, and execute with one forward pass.
Loading 2609.03715v1…