DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution

• MLLM-based control is promising for complex manipulation, including novel-command understanding, unseen-object generalization, and reasoning [7] [8].RT-2 demonstrated end-to-end MLLM control that produces performant robotic policies and emergent capabilities.Source: §1 · pp. 1–2
• Robots require real-time, low-latency interaction despite limited computational capability, memory, and battery capacity, whereas MLLM inference activates billions of parameters.These demands create substantial GPU-memory requirements, power consumption, and control delays.Source: §1 · p. 2
• Easier situations dominate robot interaction, allowing smaller models to produce correct actions while only harder cases require full MLLM capacity.This motivates unevenly allocating computation across situations rather than using the full model at every timestep.Source: §1 · p. 2
• DeeR uses multiple intermediate exits to stop inference once sufficient model capacity produces a robotic action, avoiding redundant computation.Termination criteria can target average computational cost, peak computational cost, latency, or GPU memory overhead, with online adjustment through the criterion.Source: §1 · p. 2
• 5.2-6.5x lower LLM computational cost and 2GB activated-LLM memory preserve competitive performance across 3 CALVIN LH-MTLC challenges.DeeR is evaluated with RoboFlamingo on CALVIN robot manipulation tasks and remains competitive under GPU-memory-aware termination.Source: §1 · p. 2

• Language-conditioned robotics ranges from LLM-based high-level planning to end-to-end MLLM control, but planner pipelines often rely on domain-specific low-level models.SayCan and PaLM-E translate commands into primitives, whereas RT-2 pursues end-to-end MLLM control [7] [9] [17] [18] [19].Source: §2 · p. 3
• Efficient LLM research spans structural design, model compression, and dynamic networks, with DeeR focusing on input-adaptive computation through dynamic networks.Dynamic networks reduce unnecessary computation by adjusting resource use to the input.Source: §2 · p. 3
• Unlike conventional early-exit methods using confidence or entropy, DeeR predicts sequential robot actions with temporal information and exits using action consistency.The method adapts early exiting from classification-style intermediate predictions to multimodal sequential decision-making.Source: §2 · p. 3

Motivation

• DeeR addresses robotic MLLM deployment bottlenecks by adapting activated model size to each situation instead of always inferring through every layer.Robotic platforms have limited computation and memory, while MLLM action inference can activate billions of parameters and incur substantial latency and power use.Source: §3 · p. 3

Basic architecture

• The system processes language and camera observations with a pretrained vision-language model, producing fused multimodal features for action prediction.A vision encoder converts images into visual tokens, while interleaved language and cross-attention blocks integrate visual and linguistic information.Source: §3.1 · p. 4

Multi-exit architecture

• DeeR divides the language model into consecutive groups with intermediate exits, allowing a situation-appropriate representation to be selected without activating later parameters.Each exit aggregates token-level features into a compact representation that a lightweight action head converts into low-level robot actions.Source: §3.1 · pp. 4–5

Action prediction

• The action head predicts 7-DoF end-effector actions, combining six continuous pose dimensions with a discrete gripper state and temporal information.Because the environment is partially observable, the prediction uses historical information in addition to the current observation.Source: §3.1 · p. 5

Adaptive inference

• At inference, DeeR selects an exit using a criterion conditioned on the current timestep, then computes the action from that intermediate feature and prior hidden state.The selected exit index ranges from 1 to N, and the resulting action is produced through the action head using temporal history.Source: §3.1 · p. 5

Termination and training

• The termination criterion compares adjacent intermediate action predictions, while training randomly samples exits to reduce the discrepancy between dynamic inference and training.Auxiliary action heads further optimize intermediate exits so that each activated model size can support action prediction.Source: Figure 1 · p. 4, §3.2 · p. 5, §3.3 · p. 6

Comparison with SOTA baselines

• DeeR slightly outperforms RoboFlamingo while requiring less computation and remains competitive with GR-1, while outperforming traditional imitation-learning methods particularly in ABC→D generalization.Source: §4.1 · p. 8

Scalability of DeeR

• 1.8-5.7x lower computation and 2.7x-4.0x lower peak FLOPs and memory are reported for DeeR on OpenFlamingo 9B at the same performance.Source: §4.1 · p. 9, Figure 4 · p. 9

Auxiliary losses

• Removing auxiliary losses substantially lowers performance on ABCD→D, whereas the effect is small on D→D, where the smaller dataset may be easier to fit.Source: §4.2 · p. 9, Table 3 · p. 9

Early-termination criterion

• Action consistency outperforms feature-similarity and time-based exit criteria across several average computational budgets.Source: §4.2 · p. 9, Table 4 · p. 9

Real inference efficiency

• 68.1% lower LLM inference time is achieved than RoboFlamingo++ at the same performance, alongside a theoretical 80.7% FLOPs reduction without early-exit code optimizations.Source: §4.2 · p. 9, Table 5 · p. 9

DeeR with Quantization

• DeeR can combine early exiting with quantization, which reduces memory through lower parameter precision while early exiting reduces computation by skipping unnecessary layers.Source: §4.2 · p. 10, Table 6 · p. 10

• DeeR dynamically configures MLLM size using multiple exits, action-consistency termination, threshold solving, and temporal-information training.Source: §5 · p. 10
• Experiments demonstrate reduced LLM computational costs and GPU memory usage, supporting DeeR’s potential on resource-constrained robotic platforms.Source: §5 · p. 10
• The study focuses on LLM efficiency, while the visual encoder’s substantial computational cost remains outside the proposed improvements.Source: §5 · p. 10

• DeeR preserves RoboFlamingo++ hyperparameters for fair comparison, changing the number of LLM layers and adding dynamic early exiting.Source: A Implementation Details · p. 16

Network Architecture

• OpenFlamingo uses a frozen LLM and vision encoder, trained perceiver-sampler and cross-attention fusion modules, and an action head combining a 4-layer LSTM with a 3-layer MLP.Source: §A.1 · p. 16
• Six exits are created by placing exits every two attention layers over the first 12 layers of OpenFlamingo 3B or 9B.Source: §A.1 · p. 16, Table 7 · p. 16

Dynamic Inference

• After training, exit thresholds dynamically adjust DeeR’s computation for fixed models under GPU-memory and FLOPs constraints.Source: §A.2 · p. 16
• Thresholds computed from only 1% of the training set achieve similar performance to validation-set thresholds, indicating robust generalization.Source: §A.2 · p. 16

A.3 Training Details

• Two-phase training jointly optimizes trainable MLLM components and the action head, then freezes the MLLM for action-head post-training.The additional action-head finetuning produces slightly better performance.Source: §A.3 · p. 16
• Post-training for ABCD→D uses dropout rates of 0.4 for the LSTM and 0.5 for the MLP.Results use the final-epoch checkpoint rather than the best-performing checkpoint.Source: §A.3 · p. 17
• Training hyperparameters for the D→D, ABC→D, and ABCD→D settings are reported in Table 8.Source: Table 8 · p. 17

A.4 Training Cost

• DeeR training uses PyTorch Automatic Mixed Precision on eight NVIDIA GPUs.The 3B model uses V100 32G GPUs, while the 9B model uses A100 80G GPUs.Source: §A.4 · p. 17
• The OpenFlamingo 3B model requires approximately 14–25 hours for the reported training schedules across D, ABC, and ABCD datasets.Source: §A.4 · p. 17
• Training the OpenFlamingo 9B model for 4+4 epochs on Dataset D takes around 24 hours.Source: §A.4 · p. 17

B.1 Quantative Results

• Success rates are reported separately for the 1st through 5th tasks in a task chain.The results appear in Tables 9, 10, and 11.Source: §B.1 · p. 17
• Tables 9, 10, and 11 organize the task-chain success-rate results across the reported experimental settings.Source: §B.1 · p. 17
• The supplied passage identifies the success-rate analysis but does not report numerical table entries or a comparative winner.Source: §B.1 · p. 17

B.2 Rollout Visualization

• DeeR successfully completes all five consecutive subtasks in the visualized task chain.Source: §B.2 · p. 17, Figure 6 · p. 19
• The visualization shows DeeR rollouts across five subtasks within one task chain.Source: §B.2 · p. 17, Figure 6 · p. 19