PaperlayerView exact revision on arXiv

Source-linked AI summary

CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification

Wei Li, Renshan Zhang, Rui Shao, Jie He, Liqiang Nie

arXiv:2508.21046v3cs.CVcs.RO

TL;DR

VLA models face expensive post-training and limited scalability when aligning multimodal representations with continuous actions. CogVLA addresses this with instruction-driven routing, sparsification, and coupled attention in a three-stage architecture, achieving state-of-the-art performance while reducing computational cost and inference latency. Its current routing schedules and evaluated task settings leave adaptation to varying complexity and broader generalization as open boundaries.

  • Problem

    Aligning high-dimensional VLM features with continuous actions remains computationally expensive, while lightweight methods can cause cross-modal semantic degradation through modular disconnection.

  • Method

    CogVLA uses a cognition-aligned three-stage architecture combining EFA-Routing, LFP-Routing, and CAtten for instruction-driven visual sparsification and coherent cross-modal reasoning.

  • Results

    CogVLA achieves state-of-the-art performance while significantly reducing computational cost and inference latency on LIBERO and real-world robotic tasks.

  • Takeaways & Limitations

    Instruction-driven multimodal sparsification is presented as a route toward scalable and efficient embodied AI systems.

  • Takeaways & Limitations

    CogVLA relies on predefined sparsity ratios and fixed pruning schedules, and its performance under out-of-distribution instructions or unseen manipulation categories remains insufficiently evaluated.

Abstract

from arXiv · show

Recent Vision-Language-Action (VLA) models built on pre-trained Vision-Language Models (VLMs) require extensive post-training, resulting in high computational overhead that limits scalability and deployment.We propose CogVLA, a Cognition-Aligned Vision-Language-Action framework that leverages instruction-driven routing and sparsification to improve both efficiency and performance. CogVLA draws inspiration from human multimodal coordination and introduces a 3-stage progressive architecture. 1) Encoder-FiLM based Aggregation Routing (EFA-Routing) injects instruction information into the vision encoder to selectively aggregate and compress dual-stream visual tokens, forming a instruction-aware latent representation. 2) Building upon this compact visual encoding, LLM-FiLM based Pruning Routing (LFP-Routing) introduces action intent into the language model by pruning instruction-irrelevant visually grounded tokens, thereby achieving token-level sparsity. 3) To ensure that compressed perception inputs can still support accurate and coherent action generation, we introduce V-L-A Coupled Attention (CAtten), which combines causal vision-language attention with bidirectional action parallel decoding. Extensive experiments on the LIBERO benchmark and real-world robotic tasks demonstrate that CogVLA achieves state-of-the-art performance with success rates of 97.4% and 70.0%, respectively, while reducing training costs by 2.5-fold and decreasing inference latency by 2.8-fold compared to OpenVLA. CogVLA is open-sourced and publicly available at https://github.com/JiuTian-VL/CogVLA.

1 Introduction

CogVLA addresses the computational cost and cross-modal coordination challenges of VLA models through instruction-driven sparsification and a cognition-inspired three-stage design. Its routing modules and coupled attention target efficient perception, semantic filtering, and coherent action generation.

  • Motivation: VLA models align rich multimodal representations with continuous actions, but standard training requires substantial memory, FLOPs, and time, limiting deployment.Fine-tuning a 7B VLA model on a single LIBERO task can consume over 600 GPU hours on 80G A100 GPUs.
  • Evaluation: CogVLA achieves state-of-the-art performance and improved efficiency on LIBERO and real-world robotic manipulation tasks.Ablation studies validate the complementarity and synergistic effect of the routing modules and coupled attention mechanism.
  • Framework: CogVLA introduces a biomimetic three-stage architecture corresponding to visual focusing, semantic intent filtering, and action sequence planning.The design is inspired by human multimodal coordination during manipulation.
  • Routing: EFA-Routing and LFP-Routing provide instruction-driven vision sparsification across perception-reasoning pipelines.EFA-Routing aggregates and compresses instruction-relevant visual information, while the paired routing design targets irrelevant visual inputs.
  • Coupled attention: CAtten preserves cross-modal logical consistency and temporal action coherence under doubly compressed multimodal representations.It supports coherent processing across vision, language, and action streams.

2 Methods

CogVLA uses instruction-driven routing and sparsification across a three-stage vision-language-action pipeline. It compresses and filters visual information before coupled attention supports parallel, coherent action decoding.

  • Parallel Decoding: Parallel decoding predicts all K future actions in one pass, avoiding the K × D forward passes required by token-level autoregressive decoding.The input combines observation embeddings with K learnable zero-action placeholders under bidirectional attention.
  • Progressive Design: Across the three stages, CogVLA routes and sparsifies information from perception to control while reducing computational overhead and preserving task-relevant semantics.The design aligns focused visual aggregation, semantic pruning, and coupled multimodal decoding.
  • EFA-Routing: EFA-Routing uses instruction-conditioned Encoder-FiLM modules to aggregate visual tokens and dynamically fuse heterogeneous encoder outputs.The resulting dual-aggregated representation retains instruction-relevant information while discarding redundant image tokens.
  • EFA-Routing: EFA-Routing reduces visual tokens to 25% of the original scale by retaining aggregation tokens instead of image tokens.This compression targets visual redundancy while preserving task-relevant information for later stages.
  • LFP-Routing: LFP-Routing applies instruction-conditioned modulation and pruning inside the language model to remove visually grounded tokens with low task relevance.The filtered representation maintains critical visual semantics while reducing redundant attention computation.
  • CAtten: CAtten hierarchically combines causal vision-language attention with bidirectional action attention to preserve semantic consistency and temporal coherence under sparsification.Compressed visual and instruction tokens are processed before action chunks are decoded in parallel.

3 Experiments

CogVLA is evaluated on LIBERO and real-world manipulation tasks, with experiments examining task performance, efficiency, module contributions, and visual comparisons. It achieves strong success rates and substantial computational savings through instruction-driven sparsification and coupled attention.

  • Efficiency: CogVLA outperformed OpenVLA-OFT in simulation and real-world success rates while reducing inference time by 31%.It also required 3.1× fewer FLOPs and 2.7× shorter training time than OpenVLA-OFT.
  • Performance: 97.4% success rate was the highest achieved by CogVLA across the LIBERO task suites.The evaluation used 500 trials for each task suite.
  • Performance: CogVLA achieved the highest subtask and overall success rates on real-world long-horizon manipulation tasks.The tasks included Object Placement, Drawer Manipulation, and T-shirt Folding on the ALOHA platform.
  • Efficiency: 2.79× faster inference, 22.54× higher throughput, 3.12× lower FLOPs, and 2.49× lower training cost were reported versus OpenVLA.The gains were attributed to instruction-driven vision sparsification and parallel action decoding through CAtten.
  • Qualitative Analysis: Visual comparisons showed that CogVLA more accurately executed manipulation tasks and avoided failures such as drawer collisions, with efficiency benefits increasing for longer tasks.The comparison covered LIBERO simulation and ALOHA real-world settings against OpenVLA-OFT.
  • Ablation Studies: Ablations found that both routing stages contribute to performance, with larger Stage 1 sparsity ratios producing greater improvements under fixed 8× visual sparsification.The combined Stage 1+2 design also outperformed existing visual compression methods.

4 Related Work

Related VLA research uses multimodal representations to connect perception, language, and robotic action. Efficiency efforts have primarily targeted either language-model computation or visual-token processing.

  • Vision-Language Action Models: VLA models build on VLM representations to bridge visual perception, language understanding, and action generation for robotic control.Earlier approaches aligned visual features with language-conditioned policies, while later systems introduced action tokenization.
  • Efficient Design in VLA Models: Efficient VLA methods generally follow LLM-centric or vision-centric strategies to reduce computation.LLM-centric methods reduce decoding overhead, whereas vision-centric methods reduce the number of visual tokens passed onward.

5 Conclusion

CogVLA integrates instruction-driven routing and coupled attention into a three-stage framework for efficient vision sparsification and coherent cross-modal reasoning. Evaluations on LIBERO and real-world robotic tasks report state-of-the-art performance with reduced computational cost and inference latency.

  • CogVLA combines EFA-Routing, LFP-Routing, and CAtten in a unified 3-stage progressive design.The framework targets vision sparsification and coherent cross-modal reasoning.
  • The supplementary material covers implementation details, experimental settings, extended quantitative analyses, qualitative analyses, and broader limitations and risks.
  • CogVLA demonstrations use remote communication during action-chunk prediction, which introduces slight delays from network latency.The authors plan local deployment on hardware with more than 20 GB of GPU memory to eliminate this latency.

A Implementation Details

CogVLA implements instruction-conditioned routing through visual aggregation and language-model token pruning. Its training setups fine-tune an OpenVLA backbone with LoRA under separate simulation and real-world configurations.

  • EFA-Routing: EFA-Routing uses 64 aggregation tokens per vision encoder to reduce visual tokens to 25% of the original input.FiLM scale and shift vectors derive from text embeddings, while a two-layer MLP produces routing weights for the two vision encoders.
  • LFP-Routing: LFP-Routing uses a shifted cosine schedule to control the proportion of visual tokens retained across LLM layers.For CogVLA, L = 32, η = 0.5, β_l is clamped to [0.05, 0.85], and the resulting pruning rate is approximately 50%.
  • LIBERO Training Setup: CogVLA is trained on LIBERO for 60K steps with batch size 64, learning rate 5e-4, and LoRA rank 32 with α = 64.Checkpoints are evaluated every 10K steps and the best-performing checkpoint is reported.
  • Real-World Training Setup: Real-world training uses chunk size K = 25, batch size 32, and 80K steps with LoRA rank 32 and alpha 64.The learning rate decreases from 5e-4 to 5e-5 after 50K steps, with checkpoint evaluation beginning at step 60K.

B.1 Simulation Benchmark

The evaluation uses LIBERO’s four language-conditioned manipulation suites and real-world ALOHA tasks to assess language grounding, generalization, and practical applicability. CogVLA is compared with OpenVLA under the same simulation setting and evaluated on varied single- and dual-arm tasks.

  • Simulation Benchmark: LIBERO contains Spatial, Object, Goal, and Long suites with complex instructions averaging 10.48 words per command.The benchmark is used to assess language grounding and multimodal reasoning across diverse manipulation tasks.
  • Simulation Benchmark: CogVLA and OpenVLA are trained and evaluated under the same setting, with results reported across all four LIBERO suites.This evaluates generalization, efficiency, and semantic grounding capabilities.
  • Real-World Setup: Real-world evaluation on the Cobot Agilex ALOHA platform uses varied single-arm and coordinated dual-arm manipulation tasks with moderate augmentation.The reported tasks include object placement, drawer manipulation, and T-shirt folding, with spatial and semantic variations introduced during data collection.
  • Real-World Tasks: The real-world tasks test temporal sequencing, articulated-object interaction, deformable-object manipulation, multi-attribute grounding, and spatial reasoning.Task success may require completing all sequential subtasks, and subtask and overall success rates are reported.
  • Simulation Results: Table 7 compares task success rates across the four LIBERO categories and reports strong, consistent CogVLA performance.

C.1 Multi-Seed Evaluation

Multi-seed LIBERO evaluations assess CogVLA’s statistical robustness across the Spatial, Object, Goal, and Long suites. The reported variability is low across independent runs.

  • Multi-Seed Evaluation: Three independent random seeds are used for each LIBERO suite, with mean success rate and standard deviation reported.
  • Multi-Seed Results: Standard deviations range from 0.2% to 0.6% across seeds, indicating consistent performance across diverse task types.

C.2 Extended Real-World Task Results

CogVLA extends real-world evaluation to Tasks 4–5, testing compositional language grounding, sequential actions, and egocentric spatial reasoning. The model maintains strong performance across these fine-grained manipulation settings.

  • Task 4: CogVLA achieves the highest success rates across both subtasks and the overall Task 4 evaluation.Task 4 requires grounding color and size attributes while executing sequential actions.
  • Task 5: CogVLA maintains a high success rate on Task 5, which requires interpreting left-right spatial references from the agent’s visual perspective.The task evaluates egocentric spatial reasoning.
  • Table 8 compares performance on Tasks 4–5 for the Cobot Agilex ALOHA tasks, with † marking reproduced results.
  • The extended results further validate CogVLA’s generalization to real-world tasks requiring fine-grained language grounding and spatial understanding.

C.3 Extended Ablation Studies

The ablations show that how sparsity is distributed across CogVLA’s two routing stages affects the performance–efficiency trade-off. A more aggressive reduction in Stage 1 combined with finer Stage 2 pruning provides the strongest reported spatial result.

  • The 2×2 setting provides a favorable trade-off between performance and computational efficiency.
  • The 4×4 setting slightly degrades performance, suggesting that excessive sparsification in both stages can hinder preservation of task-relevant information.
  • Spf.Ratio denotes the adjustable sparsification ratio used to tune the performance–efficiency trade-off.
  • 98.6 is the highest spatial success rate, achieved by the asymmetric 4×–2× allocation across Stages 1 and 2.This configuration outperforms symmetric alternatives.
  • The 4×–2× allocation assigns stronger compression to Stage 1 and finer-grained context-aware pruning to Stage 2.Instruction-guided aggregation compresses redundant visual tokens early, while language-model pruning preserves task-relevant information later.
  • The ablation supports progressive sparsification with stage-aware, asymmetric allocation to balance early compression, late selectivity, computational efficiency, and task accuracy.

D.1 Additional Visualizations of Simulation and Real-World Results

Additional visualizations illustrate CogVLA’s execution across simulation and real-world manipulation tasks and show how instruction-aware routing focuses perception on task-relevant regions. The examples also connect the model’s routing stages to biologically inspired cognitive functions.

  • Simulation and real-world execution: CogVLA consistently completes multi-step simulation tasks across diverse environments, object configurations, and instruction variants.
  • Simulation and real-world execution: In real-world tasks, CogVLA interprets long-horizon commands and produces coherent action sequences across simulation-to-reality settings.Task 1 uses Front, Left Wrist, and Right Wrist camera views; Tasks 2–5 show Front Camera observations.
  • Instruction-guided attention: Attention maps highlight task-relevant image regions, indicating that instruction-aware routing guides visual grounding in cluttered or ambiguous scenes.The visualizations use DINOv2 and SigLIP aggregation-to-patch attention maps for two camera views.
  • Cognitive grounding: The architecture is motivated by improving computational efficiency and cross-modal semantic alignment in instruction-conditioned robotic systems.
  • Cognitive grounding: CogVLA’s routing modules are mapped to the Visual Attention System, Supplementary Motor Area, and Premotor Cortex.The proposed correspondences are Encoder-FiLM, LLM-FiLM, and V-L-A Coupled Attention, respectively.
  • Cognitive grounding: The paper relates biologically aligned modular design to potential gains in generalization and interpretability for embodied multimodal agents.

E.2 Limitation and Future Work

CogVLA’s current sparsification schedules may not adapt optimally to different instructions or scenes, and its out-of-distribution generalization remains insufficiently evaluated. Future work targets adaptive sparsification, online adaptation, and richer sensory feedback, while deployment risks require safeguards.

  • Limitations: CogVLA relies on predefined sparsity ratios and fixed token-pruning schedules that may not adapt optimally to instruction complexity or scene difficulty.
  • Limitations: Performance under out-of-distribution instructions and unseen manipulation categories has not been thoroughly evaluated.The reported generalization scope is primarily the LIBERO and ALOHA settings.
  • Future work: Future work proposes task- and uncertainty-conditioned sparsification, lifelong learning or online adaptation, and multimodal feedback such as haptic or force sensing.
  • Broader impact and risk: Safety-critical deployment risks include ambiguous-instruction misinterpretation, failures in unpredictable environments, and training-data bias amplification.The paper recommends robust evaluation, transparency, and human-in-the-loop designs.
  • Broader impact and risk: Potential application areas include assistive robotics, household automation, and industrial assembly.
Loading 2508.21046v3…