PaperlayerView exact revision on arXiv

Source-linked AI summary

sensVLA: Spatially-Grounded Vision-Language-Action Model for Autonomous Wheel Loader

Gopi Krishna Erabati, Bjarne Johannsen, Angus Stewart, Vardeep Singh Sandhu

arXiv:2609.17021v1cs.CVcs.RO

TL;DR

Wheel-loader control must combine task semantics, egocentric vision, proprioception, and 3D geometry, while RGB tokens alone capture metric spatial cues poorly. sensVLA routes lidar BEV features and VLM context to a flow-matching action expert; it matches a camera-only baseline in aggregate error and improves loading and camera-loss results.

  • Problem

    Wheel-loader control requires joint reasoning over task semantics, vision, proprioception, and 3D geometry, while RGB tokens alone capture metric spatial cues poorly.

    Source: §I · p. 1

  • Method

    sensVLA routes lidar BEV features to a trainable flow-matching action expert while a VLM provides task-conditioned semantic context from RGB views.

    Source: §I · p. 1, C. Action Expert and Flow-Matching Head · p. 3

  • Results

    sensVLA matches a strong camera-only baseline in aggregate per-step error and reduces loading-group vx RMSE by 28% and dx RMSE by 9.4%, while degrading 29% less when camera input is ablated.

    Source: §VI · p. 4

  • Takeaways & Limitations

    BEV grounding provides an asymmetric fallback for metric-consistent motion when RGB is lost, while the no-BEV result indicates it is not the sole load-bearing modality.

    Source: B. Results and Discussion · p. 4

Abstract

from arXiv · show

Autonomous wheel-loader control requires joint reasoning over task semantics, egocentric vision, proprioception, and 3D scene geometry. We present sensVLA, a Vision-Language-Action (VLA) architecture that combines a Qwen3-2B Vision-Language Model (VLM) with a fully trainable transformer action expert trained by flow-matching velocity regression. sensVLA routes Bird's-Eye-View (BEV) features, extracted from fused front and rear lidar, directly to the action expert through a dedicated cross-attention pathway, while the VLM consumes front and rear RGB views to provide task-conditioned semantic context. This design decouples spatial grounding from linguistic reasoning while preserving interaction between both streams at decision time. The expert predicts six action dimensions: longitudinal velocity, steering, body-frame displacement, arm rate, and bucket rate. On a real-world dataset from a wheel loader, sensVLA reaches aggregate per-step parity with a strong camera-only baseline and reduces longitudinal velocity RMSE by 28% and displacement error by 9% on loading centric scenarios. It also degrades 29% less when the camera stream is corrupted or removed, evidencing that explicit spatial grounding improves accuracy and fault-tolerance for heavy equipment autonomy.

I. INTRODUCTION

Wheel-loader control must combine task semantics with metric spatial reasoning, but RGB tokens alone capture geometry such as pile distance and bucket clearance poorly. sensVLA routes lidar BEV features to a trainable action expert alongside VLM context, and the paper reports improved loading accuracy and robustness to camera loss.

I. INTRODUCTION

  • Wheel loaders must navigate unstructured terrain, coordinate drivetrain and hydraulic actuators, and respect scene semantics such as rock piles and tunnel walls.VLA backbones may provide semantic priors, but the action head must remain expressive for continuous, high-frequency control.Source: §I · p. 1
  • RGB tokens capture metric spatial cues poorly, while limited real-world machine data makes full VLM fine-tuning statistically inefficient.The paper identifies lidar-derived BEV geometry and parameter-efficient fine-tuning as responses to these challenges.Source: §I · p. 1
  • sensVLA routes BEV lidar features to a fully trainable action expert through heterogeneous cross-attention while using a LoRA-adapted VLM for task-level context.This decouples geometric grounding from language reasoning.Source: §I · p. 1
  • Flow-matching velocity regression predicts temporally chunked 6-DoF actions for continuous multi-actuator control without discretisation artifacts.The paper also proposes staged parameter-efficient training with adapter unfreezing, state-history dropout, and EMA validation to mitigate overfitting under data-scarce conditions.Source: §I · p. 1
  • BEV grounding improves loading-centric accuracy over a strong camera-only baseline and degrades less under simulated camera loss.The paper presents spatial grounding as an accuracy and fault-tolerance lever for heavy-equipment VLAs.Source: §I · p. 1

II. RELATED WORK

Related work spans BEV-based metric grounding and VLA policies that combine language, vision, and action. sensVLA combines these directions by routing BEV features to action generation alongside VLM semantic reasoning.

II. RELATED WORK

  • BEV methods encode scene geometry in an ego-centric metric frame, while sensVLA routes BEV features directly to action generation rather than keeping them in the perception stack.The cited examples include PointPillars, Lift-Splat-Shoot, BEVFormer, BEVFusion, and TransFuser –.Source: §II · p. 2
  • VLA and generalist robot-policy work combines language, vision, and action, with LoRA adaptation useful for limited in-domain data and action heads moving toward continuous prediction.The passage cites VLA work,, [9]–[13], LoRA, and continuous or chunked action prediction [3], [15],.Source: §II · p. 2
  • sensVLA pairs VLM-based semantic reasoning with explicit BEV grounding for articulated heavy-machine control rather than relying on modular loading pipelines –.Source: §II · p. 2

III. SYSTEM OVERVIEW AND DATA PIPELINE

The data pipeline synchronizes asynchronous onboard sensors to control timestamps and builds samples that combine RGB, task prompts, lidar, proprioception, and action targets. The real-world corpus covers loading and tunnel-driving scenarios and is split by recording sequence to avoid near-duplicate temporal leakage.

III. SYSTEM OVERVIEW AND DATA PIPELINE

  • Onboard lidar, RGB cameras, and IMU sensors are stored asynchronously and synchronized to control timestamps during training.Source: §III · p. 2
  • Each sample combines RGB images, a short task prompt, 0.5 seconds of accumulated lidar, proprioception, and action targets.Supervision uses ten-step action chunks and conditioning includes the current observation and a ten-step state-history window; samples with missing packets, invalid synchronization, or incomplete calibration are discarded.Source: §III · p. 2
  • The corpus covers pile loading and underground-tunnel driving, with splits made by recording sequence rather than frame to prevent near-duplicate temporal leakage.Source: §III · p. 2
  • The retained examples form a compact real-world dataset for articulated loading-and-hauling with synchronized semantic, geometric, and proprioceptive context.Source: §III · p. 2

IV. SENSVLA ARCHITECTURE

sensVLA processes RGB and text through a VLM and lidar through a BEV encoder, then fuses their features with proprioceptive conditioning in an action expert. The expert produces temporally chunked actions using flow-matching velocity regression.

IV. SENSVLA ARCHITECTURE

  • The architecture takes RGB, text, proprioceptive state with a ten-step history, and accumulated lidar, and produces ten future 6-DoF actions, as depicted in Fig. 1.Source: §IV · p. 2, Figure 1 · p. 2
  • A LoRA-adapted Qwen3-2B VLM produces task-conditioned context from RGB and text, while a frozen PointPillars backbone encodes lidar into a BEV feature grid.Source: §IV · p. 2
  • The fully trainable transformer action expert fuses VLM context, BEV features, and proprioceptive conditioning through heterogeneous cross-attention, then decodes actions via flow-matching velocity regression.Source: §IV · p. 2

A. Image, Text and State Encoding

sensVLA encodes front and rear RGB views, text, and proprioceptive state in a shared VLM sequence. The truncated decoder’s hidden states provide context for the action expert.

A. Image, Text and State Encoding

  • Front and rear images share the frozen ViT encoder, preserving pretrained image–text alignment while allowing the decoder to select which view to attend to.The decoder is truncated to its first 14 transformer blocks; its final hidden states form the action expert’s VLM context.Source: A. Image, Text and State Encoding · p. 2
  • A 20-dimensional proprioceptive state is projected into the VLM hidden dimension and appended as an input token.It includes longitudinal velocity, yaw rate, and six-channel readings from three IMUs.Source: A. Image, Text and State Encoding · p. 2
  • The shared decoder processes the fused image, text, and state tokens, with its last-layer hidden states serving as action-expert context.Source: A. Image, Text and State Encoding · p. 2

B. State-History and BEV Encoding

The model conditions its action expert on proprioceptive history, fused-lidar BEV features, and VLM context. A flow-matching expert converts noised action chunks into continuous multi-actuator commands.

B. State-History and BEV Encoding

  • A ten-step proprioceptive history is compressed into a mean-pooled token, supplying low-frequency inertial context without lengthening the VLM sequence.Source: B. State-History and BEV Encoding · pp. 2–3
  • Fused front and rear lidar sweeps over 0.5 s pass through PointPillars to produce spatial BEV tokens routed directly to the action expert.Multi-scale features are upsampled and concatenated into a dense map; direct routing decouples spatial-to-action mapping from the LoRA adapters’ limited capacity.Source: B. State-History and BEV Encoding · p. 3

C. Action Expert and Flow-Matching Head

  • A 4-layer, 8-head transformer processes noised 10-step action chunks, with successive cross-attention layers using VLM context, BEV tokens, and both together.The BEV pathway uses a gated residual initialized near identity, and the heterogeneous layout adds spatial grounding at essentially no additional parameter cost relative to a VLM-only expert.Source: C. Action Expert and Flow-Matching Head · p. 3
  • Flow-matching velocity regression trains the expert on noised actions, and a ten-step midpoint ODE solver integrates its learned velocity field at inference.A per-component weighted MSE balances the six action dimensions’ physical units; continuous commands avoid discretization artifacts and VQ-style losses.Source: C. Action Expert and Flow-Matching Head · p. 3

A. Implementation Details

The implementation uses onboard wheel-loader data, a parameter-efficient VLM adaptation, a frozen PointPillars encoder, and a camera-only comparison baseline.

A. Implementation Details

  • The note reports collecting data onboard a wheel loader and curating it as a dataset; the dataset comprises 200K training chunks and 40K validation.Source: A. Implementation Details · p. 3
  • The VLM decoder is truncated to 14 blocks, with LoRA adapters in its final eight layers, while the pretrained PointPillars encoder remains frozen.LoRA uses rank 16, α 16, and 0.05 dropout.Source: A. Implementation Details · p. 3
  • The action expert predicts 10-step chunks at a five-step stride, corresponding to a 2 s horizon at 25 Hz; the baseline removes BEV and uses RGB, text, and proprioception.Training runs for 30 epochs with bf16 mixed precision and an effective batch size of 32.Source: A. Implementation Details · p. 3

B. Results and Discussion

On the 40K validation chunks, sensVLA improves longitudinal-velocity and displacement-dx RMSE while leaving steering and lateral-dy errors statistically unchanged. Loading gains persist over the prediction horizon, and the model degrades less than the camera-only baseline under camera loss.

B. Results and Discussion

  • Across 40K validation chunks, longitudinal-velocity RMSE fell from 1.134 to 0.886 m/s (−21.9%) and displacement-dx RMSE from 9.52 to 8.98 cm (−5.7%), while steering and lateral-dy errors remained statistically identical.Source: B. Results and Discussion · p. 3
  • On loading, vx RMSE fell from 0.996 to 0.717 m/s (−28.0%) and dx RMSE from 9.27 to 8.40 cm (−9.4%); driving vx also improved while dx remained effectively flat.The passage attributes the strongest loading gains to metric cues such as pile-face geometry and the bucket-approach corridor.Source: B. Results and Discussion · p. 3
  • On the loading group, final displacement error over the 2-second horizon fell from 0.538 m to 0.521 m (−3.1%), preserving the per-step gains under integration.Source: B. Results and Discussion · pp. 3–4
  • Under camera loss, aggregate degradation was 2.34× for the baseline and 1.67× for sensVLA; the no-BEV check was 1.18×.The table caption defines Agg as mean per-component RMSE in z-normalised action space and ∆× as the ratio to each model’s full-input anchor.Source: Table II · p. 4, B. Results and Discussion · p. 4
  • In two validation examples—a forward pile approach and a reverse manoeuvre after bucket interaction—the predicted path closely follows ground truth despite partial camera occlusion and low rear-camera light.Source: B. Results and Discussion · p. 4

VI. CONCLUSION AND FUTURE WORK

sensVLA routes front and rear lidar BEV features to a flow-matching action expert while using a LoRA-adapted Qwen3-VL backbone for language and RGB understanding; it matches a camera-only baseline in aggregate per-step error and improves loading-group errors. The authors describe decision-stage spatial grounding as a data-efficient, fault-tolerant recipe and propose encoder co-training, longer temporal BEV fusion, and multitasking as future work.

VI. CONCLUSION AND FUTURE WORK

  • On loading tasks, sensVLA reduces vx RMSE by 28% and dx RMSE by 9.4%, matches the camera-only baseline in aggregate per-step error, and degrades 29% less when camera input is ablated.The conclusion reports loading-group improvements and aggregate parity with a strong camera-only baseline.Source: §VI · p. 4
  • sensVLA sends front and rear lidar BEV features to a flow-matching action expert while a LoRA-adapted Qwen3-VL backbone specializes in language and RGB understanding.The conclusion frames this routing as placing spatial grounding at the decision stage rather than inside the language decoder.Source: §VI · p. 4
  • Figure 2 shows two loading-approach validation frames, with the ground-truth 2 s ego-path in green and sensVLA’s prediction in red on BEV.The caption identifies the visualized paths and their color encoding.Source: Figure 2 · p. 4
  • The authors identify PointPillars co-training, longer temporal BEV histories, and multitasking as future work.They describe explicit decision-stage spatial grounding as a data-efficient and fault-tolerant recipe for VLA-based heavy-equipment autonomy.Source: §VI · p. 4

Related papers

Loading 2609.17021v1…