Source-linked AI summary
Data-Efficient Adaptation of a Driving VLA to Class 8 Trucks
Satyajeet Das, Aaron Buxbaum, Niels Joubert, Gaurav S. Sukhatme
TL;DR
Passenger-car VLAs do not readily transfer to Class 8 trucks in long-tail highway scenarios. The paper adapts Alpamayo 1.5 with targeted truck demonstrations, then applies Flow Velocity Steering (FVS) without updating the adapted model. In open-loop evaluation, targeted fine-tuning reduces trajectory errors, and FVS further improves them.
Problem
Data-efficient adaptation of pretrained driving VLAs to trucks in long-tail highway scenarios has not been systematically studied, despite truck-specific embodiment differences from passenger cars.
Source: §I · p. 1Method
The adapt-then-steer strategy fine-tunes Alpamayo 1.5’s action-generation stack on targeted truck scenarios, then applies FVS corrections while keeping the adapted VLA fixed.
Source: §I · p. 2Results
Approximately 56% lower full-horizon ADE and FDE versus the off-the-shelf VLA; FVS further reduces ADE and FDE by 13.9% and 16.5%, respectively.
Source: §I · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
Class 8 trucks differ from passenger cars in geometry, dynamics, and maneuvering requirements. As a result, vision-language-action (VLA) models trained for passenger vehicles do not readily transfer to Class 8 trucks, particularly in unstructured scenarios such as accident scenes and construction zones. Rather than training a truck-driving VLA from scratch, we propose an adapt-then-steer strategy that adapts an off-the-shelf VLA to generate trajectories for Class-8 trucks in these challenging scenarios. In the adapt stage, we use NVIDIA's Alpamayo 1.5 as the base model, fine-tuning only its action-generation stack on a few hundred real-world construction and accident-related highway scenarios. In the steer stage, we introduce Flow Velocity Steering (FVS) to further refine the model's predictions while holding the adapted VLA fixed. FVS is a compact, flow-time-conditioned residual module that adds learned corrections to the action-space flow velocity used to update the action sequence at each generation step. In open-loop evaluation on a scenario-disjoint held-out set, targeted fine-tuning more than halves single-candidate average displacement error (ADE) and final displacement error (FDE) over the entire 6.4 s horizon compared to the base model. Using the same targeted demonstrations, FVS further reduces the fine-tuned model's full-horizon ADE and FDE by 13.9% and 16.5%, respectively. At matched data budgets, targeted supervision yields 19-26% lower full-horizon ADE than general truck-driving supervision, while the targeted model remains competitive with a model fine-tuned on approximately 65 times as many general truck-driving scenarios. These results support adapt-then-steer for data-efficient vehicle-domain transfer to Class 8 trucks. Our project website is available at https://truckvla.github.io.
I. INTRODUCTION
Truck-specific geometry and dynamics make passenger-vehicle trajectories unreliable for Class 8 trucks in construction and accident scenarios. The paper adapts a pretrained VLA’s action-generation stack, then steers its generation process, reporting lower trajectory errors and data-efficient gains.
Motivation
- Truck-specific embodiment matters: under the same reference path, trailer off-tracking can put a truck’s swept envelope inside a work-zone closure that a passenger car clears.Construction zones and accident-related highway disruptions are the target long-tail scenario families.Source: §I · p. 1, Figure 1 · p. 1
Approach
- The adapt-then-steer strategy fine-tunes only the action-generation stack with the vision-language backbone frozen, then trains FVS to correct flow velocity while keeping the adapted VLA fixed.FVS is a flow-time-conditioned residual module trained on the same targeted demonstrations; its corrections can influence subsequent action generation.Source: §I · p. 2
Results
- Approximately 56% lower full-horizon ADE and FDE versus the off-the-shelf VLA after Targeted Truck-SFT; FVS further reduces them by 13.9% and 16.5%, respectively.These are single-candidate results over the full 6.4 s horizon on a scenario-disjoint test set.Source: §I · p. 2
Data efficiency
- At matched data budgets, targeted supervision yields 19–26% lower full-horizon ADE than general truck-driving supervision, and remains competitive with training on approximately 65× as many general scenarios.The general scenarios were not specifically selected from the two target long-tail families.Source: §I · p. 2
II. RELATED WORK
Related work spans autonomous-driving VLAs, truck-domain transfer, and residual adaptation of generative action policies. This paper distinguishes its approach by adapting the action-generation stack using real truck trajectories while preserving the pretrained vision-language backbone.
Autonomous-driving VLAs and long-tail adaptation
- Autonomous-driving VLAs combine scene understanding and trajectory generation; prior approaches include reasoning-based planning, unified multimodal outputs, and long-tail trajectory prediction [1] [8] [2].Other adaptation interfaces include language-guided policies [9], soft prompts [10], takeover data with reinforcement post-training [11], and parameter-efficient adapters [13].Source: §II · p. 2
Autonomous trucking and vehicle-domain transfer
- Truck-domain work documents sensing and operating differences, while MVAdapt transfers policies across embodiments in simulation; this paper uses real truck trajectories in construction and accident scenarios [4] [5].Truck-specific challenges include elevated sensor perspectives, trailer occlusions, and long sensing and anticipation ranges.Source: §II · p. 2
Residual adaptation of generative action policies
- Residual methods modify actions or generative processes while policy weights remain fixed; FVS corrects intermediate action-space flow velocities and trains on truck demonstrations [16] [18] [19].Unlike the described prior methods, FVS exposes its residual network to action states induced by its own earlier corrections.Source: §II · pp. 2–3
III. METHOD
The method represents logged truck motion as a future pose trajectory referenced to the vehicle control point, and uses Alpamayo’s native action representation and decoder. Adaptation proceeds in two stages: fine-tune action generation, then learn FVS with the adapted policy frozen.
A. Problem Formulation and Overview
- The target is the logged future pose trajectory of the truck’s vehicle control point, expressed in the coordinate frame of its current pose.The VCP is the common reference point for logged and predicted truck motion.Source: A. Problem Formulation and Overview · p. 3
- The method retains Alpamayo’s normalized acceleration-curvature actions; a fixed mapping constructs action targets, while the decoder denormalizes and integrates generated actions using the native unicycle model.The decoder is differentiable with respect to generated actions during FVS training.Source: A. Problem Formulation and Overview · p. 3
- Stage I fine-tunes the action-generation stack with the vision-language backbone frozen; Stage II freezes the adapted policy and trains FVS to correct action-space flow velocity throughout generation.FVS acts on the generative rollout rather than updating the adapted policy weights.Source: A. Problem Formulation and Overview · p. 3
B. Truck-Specific Action-Expert Fine-Tuning
The method reuses the pretrained vision-language backbone while fine-tuning the flow-matching action stack on truck demonstrations. At inference, action sequences evolve through Euler updates in generative flow time.
B. Truck-Specific Action-Expert Fine-Tuning
- The vision-language backbone stays frozen while the Action Expert and its action input/output projections are fine-tuned on truck demonstrations.The multimodal context z is computed from observations by the frozen backbone.Source: B. Truck-Specific Action-Expert… · p. 3
- SFT retains conditional flow matching, constructing each training action state by interpolating Gaussian noise with an expert truck action sequence.The model is optimized to predict the action-space flow velocity.Source: B. Truck-Specific Action-Expert… · p. 3
- At inference, sampling starts from Gaussian noise and uses native Euler updates while the multimodal context remains fixed.Generative flow velocity describes change in normalized actions over flow time, not the truck’s physical velocity.Source: B. Truck-Specific Action-Expert… · p. 3
C. Flow Velocity Steering
Flow Velocity Steering (FVS) refines generation by adding learned residuals to the adapted policy’s action-space flow velocity at each Euler step. The adapted policy remains fixed as the corrected action state is fed through the rollout.
C. Flow Velocity Steering
- FVS adds a learned residual to the frozen Action Expert’s nominal action-space flow velocity at each Euler step.The residual is conditioned on flow time and the current action state and uses the Action Expert’s hidden action representation.Source: C. Flow Velocity Steering · p. 3
- The steered sampler applies the correction to update the action state before the next Euler step.Because the updated state returns to the frozen Action Expert, FVS steers the rollout rather than only the completed sequence.Source: C. Flow Velocity Steering · p. 3
- The steering-strength parameter α controls the correction, and α = 0 recovers the frozen SFT sampler.Source: C. Flow Velocity Steering · p. 3
D. Trajectory-Aware Steering Objective
FVS is trained on the same expert demonstrations as SFT while the adapted policy remains fixed. Its objective combines intermediate bridge supervision with decoded-trajectory imitation and rollout-aware optimization.
D. Trajectory-Aware Steering Objective
- FVS training uses the same logged demonstrations as SFT and matches initial Gaussian noise between frozen and steered rollouts.A training-only difficulty weight derived from frozen-rollout trajectory error gives greater weight to examples with larger errors.Source: D. Trajectory-Aware Steering Objective · pp. 3–4
- The bridge target uses the remaining flow time to define a constant action-space velocity that would move the current state to the expert action.The bridge loss penalizes deviation of the predicted residual correction from this target.Source: D. Trajectory-Aware Steering Objective · p. 4
- After a complete steered rollout, the decoded trajectory receives imitation supervision, with a separate penalty for regression relative to the matched frozen-policy prediction.The objective also includes correction-magnitude and architecture-specific regularization terms.Source: D. Trajectory-Aware Steering Objective · p. 4
- Rollout-aware optimization trains FVS on action states induced by its own earlier corrections while keeping the adapted VLA fixed.Training requires neither environment interaction nor additional expert queries.Source: D. Trajectory-Aware Steering Objective · p. 4
IV. EXPERIMENTAL RESULTS
The evaluation uses real truck encounters from construction and accident-related scenarios, with a scenario-disjoint held-out set. It compares targeted and general truck fine-tuning, post-SFT residual methods, and a deterministic motion reference under open-loop trajectory evaluation.
A. Dataset and Evaluation Protocol
- The held-out evaluation contains 26 real-world encounters disjoint from the targeted and general training corpora, which contain 229 and 14,891 scenarios, respectively.The targeted training set comprises 138 accident-related and 91 construction scenarios.Source: A. Dataset and Evaluation Protocol · p. 4
- Targeted Truck-SFT and General Truck-SFT fine-tune the action-generation stack on their respective datasets, while General→Targeted SFT adds targeted fine-tuning after general initialization.Post-SFT FVS and final-action residual models share the same frozen Targeted Truck-SFT checkpoint and matched training budgets within each architecture.Source: A. Dataset and Evaluation Protocol · pp. 4–5
- The data-efficiency protocol compares three independently sampled subset runs per supervision source and budget using a common held-out test set.Subset runs and full-data runs use the same training-update budget.Source: A. Dataset and Evaluation Protocol · p. 5
- Open-loop predictions use synchronized truck-camera inputs and are scored against logged expert trajectories over a 6.4-second horizon.The protocol reports single-candidate ADE and FDE alongside oracle best-of-16 minADE and minFDE; each metric may select a different candidate.Source: A. Dataset and Evaluation Protocol · p. 5
- Reported aggregates weight scenarios equally through equal numbers of windows and summarize inference-sampling variability across seeds.Each window is compared with one logged expert trajectory, so alternative valid maneuvers can have nonzero displacement error.Source: A. Dataset and Evaluation Protocol · p. 5
B. Targeted Fine-Tuning for Truck Adaptation
With the vision-language backbone frozen, fine-tuning the action-generation stack on 229 targeted scenarios reduces full-horizon trajectory error. The adapted model improves on both the Base model and the CTRV baseline across reported single-candidate horizons.
B. Targeted Fine-Tuning for Truck Adaptation
- Approximately 56% lower full-horizon single-candidate ADE and FDE, with ADE falling from 2.636 to 1.165m after fine-tuning on 229 targeted scenarios.The vision-language backbone remains frozen during action-side fine-tuning.Source: B. Targeted Fine-Tuning for Truck… · p. 5
- Targeted Truck-SFT achieves lower error than Base and CTRV at every reported single-candidate horizon, although Base has lower ADE than CTRV at 1 and 3s.CTRV extrapolates recent truck motion without visual or language input.Source: B. Targeted Fine-Tuning for Truck… · p. 6
- Targeted Truck-SFT lowers scenario-mean ADE@6.4s relative to Base in all 26 scenarios, with mean reductions of approximately 63% for accident-related scenarios and 52% for construction scenarios.Lateral and longitudinal ADE are roughly halved, while final heading error decreases from 3.293° to 0.885° (Table II).Source: B. Targeted Fine-Tuning for Truck… · p. 6
- None of the tested truck-specific prompts reduced single-candidate ADE@6.4s relative to the default prompt in a one-seed held-out diagnostic.The tested prompts incorporated expert feedback on turning, trailer off-tracking, braking, clearance, and control smoothness.Source: B. Targeted Fine-Tuning for Truck… · p. 6
C. Data Efficiency of Targeted Supervision
At matched training-set sizes, targeted long-tail supervision yields lower held-out ADE@6.4s than general truck-driving supervision. Targeted Truck-SFT trained on 229 scenarios remains competitive with General Truck-SFT trained on approximately 65 times as many scenarios.
C. Data Efficiency of Targeted Supervision
- Approximately 19–26% lower mean ADE@6.4s with targeted than general truck-driving supervision at matched budgets of 2, 10, 100, and 200 scenarios.Figure 3 plots mean performance across three independently trained subset runs at each size, with shaded bands showing ±1 standard deviation.Source: Figure 3 · p. 6, C. Data Efficiency of Targeted… · p. 6
- For targeted supervision, error decreases as training-set size grows, with diminishing gains at larger evaluated budgets.Figure 3 compares targeted and general supervision across training-set sizes.Source: C. Data Efficiency of Targeted… · p. 6, Figure 3 · p. 6
- Targeted Truck-SFT achieves lower full-horizon ADE@6.4s and FDE@6.4s than General Truck-SFT: 1.165 versus 1.287m ADE and 3.159 versus 3.740m FDE.General Truck-SFT is slightly more accurate at 1s; General→Targeted SFT does not consistently improve over targeted-only adaptation.Source: C. Data Efficiency of Targeted… · p. 6
- Targeted Truck-SFT trained on 229 scenarios remains competitive with General Truck-SFT trained on approximately 65× as many general truck-driving scenarios.Figure 3 marks the full-data checkpoints at 229 targeted scenarios and 14,891 general scenarios.Source: C. Data Efficiency of Targeted… · p. 6, Figure 3 · p. 6
D. Post-SFT Refinement with Flow Velocity Steering
Flow Velocity Steering (FVS) refines Targeted Truck-SFT predictions while keeping the adapted model fixed. It improves full-horizon accuracy and yields consistent gains across the held-out scenarios.
D. Post-SFT Refinement with Flow Velocity Steering
- Transformer FVS reduces Targeted Truck-SFT’s full-horizon ADE and FDE by 13.9% and 16.5%, respectively, versus 3.2% and 5.0% for a Transformer final-action residual.The comparison uses the same 229 targeted training scenarios; FVS also outperforms matched-parameter, matched-update final-action residuals for MLP and Transformer architectures.Source: D. Post-SFT Refinement with Flow… · p. 6
- Transformer FVS lowers scenario-mean ADE@6.4s relative to Targeted Truck-SFT in all 26 scenarios, with mean reductions of approximately 16% for accident-related and 11% for construction scenarios.It also lowers all reported oracle minADE and minFDE values relative to Targeted Truck-SFT.Source: D. Post-SFT Refinement with Flow… · pp. 6, 8
- FVS adds learned residual corrections to the action-space flow velocity at each generation step while keeping the adapted VLA fixed.At inference, it requires one lightweight residual-network evaluation per existing Euler step, without adding flow-integration steps or vision-language backbone passes.Source: D. Post-SFT Refinement with Flow… · pp. 6, 8
E. Qualitative Analysis
In selected held-out scenarios, targeted fine-tuning brings predictions closer to logged truck trajectories, and FVS further refines those predictions. The study reports open-loop agreement with logged trajectories and identifies closed-loop evaluation and simulator-based post-training as future work.
E. Qualitative Analysis
- In selected accident and construction examples, Targeted Truck-SFT brings predicted maneuvers closer to logged expert trajectories, while FVS stays close to or makes a smaller correction to the SFT prediction.The examples include yielding or increasing clearance around emergency vehicles and pedestrians, cone intrusions, and lane closures.Source: E. Qualitative Analysis · p. 8
V. CONCLUSION
- With the vision-language backbone frozen, fine-tuning the action-generation stack on 229 scenario-aligned events reduces full-horizon ADE and FDE by approximately 56%.The evaluation measures open-loop agreement with logged truck trajectories rather than closed-loop driving performance.Source: §V · p. 8
- At matched data budgets, scenario-aligned supervision yields lower full-horizon ADE than general truck-driving supervision, while 229 targeted scenarios remain competitive with approximately 65× as many general scenarios.FVS further reduces full-horizon ADE and FDE by 13.9% and 16.5%, respectively, while keeping the adapted VLA fixed.Source: §V · p. 8
- The conclusion supports adapt-then-steer for Class 8 vehicle-domain transfer: specialize action generation with scenario-aligned demonstrations, then steer the adapted policy’s action-space flow velocity.The authors identify closed-loop evaluation and simulator-based post-training as future work.Source: §V · p. 8