PaperlayerView exact revision on arXiv

Source-linked AI summary

Unifying Language-Action Understanding and Generation for Autonomous Driving

Xinyang Wang, Qian Liu, Wenjie Ding, Zhao Yang, Wei Li, Chang Liu, Bailin Li, Kun Zhan, Xianpeng Lang, Wei Chen

arXiv:2603.01441v1cs.CVcs.RO

TL;DR

VLA models face persistent misalignment between language instructions and physical actions, as well as inefficient autoregressive action generation. LinkVLA unifies language and action tokens, trains bidirectional action understanding, and uses coarse-to-fine decoding. On closed-loop benchmarks, it improves instruction following and driving performance while reducing inference latency.

  • Problem

    VLA models can misalign language instructions and physical actions, while typical autoregressive action generation is inefficient.

    Source: §1 · p. 2

  • Method

    LinkVLA uses a shared discrete language-action codebook, an action-understanding objective mapping trajectories back to text, and coarse-to-fine action generation.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 3

  • Results

    LinkVLA improves instruction-following and driving performance on closed-loop benchmarks, while its coarse-to-fine decoder cuts inference latency by 86%.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results provide a practical path toward reliable, responsive language-guided agents for real-world deployment.

    Source: §5 · p. 9

Abstract

from arXiv · show

Vision-Language-Action (VLA) models are emerging as a promising paradigm for end-to-end autonomous driving, valued for their potential to leverage world knowledge and reason about complex driving scenes. However, existing methods suffer from two critical limitations: a persistent misalignment between language instructions and action outputs, and the inherent inefficiency of typical auto-regressive action generation. In this paper, we introduce LinkVLA, a novel architecture that directly addresses these challenges to enhance both alignment and efficiency. First, we establish a structural link by unifying language and action tokens into a shared discrete codebook, processed within a single multi-modal model. This structurally enforces cross-modal consistency from the ground up. Second, to create a deep semantic link, we introduce an auxiliary action understanding objective that trains the model to generate descriptive captions from trajectories, fostering a bidirectional language-action mapping. Finally, we replace the slow, step-by-step generation with a two-step coarse-to-fine generation method C2F that efficiently decodes the action sequence, saving 86% inference time. Experiments on closed-loop driving benchmarks show consistent gains in instruction following accuracy and driving performance, alongside reduced inference latency.

1. Introduction

VLA systems promise explicit reasoning and flexible instruction following, but language-action misalignment remains a barrier to safety and reliability. LinkVLA combines shared language-action tokens, bidirectional action understanding, and coarse-to-fine generation; it reports stronger closed-loop driving and instruction following with reduced latency.

1. Introduction

  • LinkVLA reports state-of-the-art results on challenging closed-loop benchmarks, with significant gains in instruction following and driving alongside reduced inference latency.Figure 1 summarizes the reported higher performance and lower latency in closed-loop evaluation.Source: Figure 1 · p. 1, §1 · p. 2
  • Language-action misalignment can make a model correctly decide to change lanes yet output a lane-keeping trajectory, undermining safety and reliability.The paper identifies this as a significant barrier to VLA instruction following.Source: §1 · p. 2
  • Prior alignment efforts modify data collection [10] [37], apply post-hoc reinforcement learning, or implicitly match latent distributions [8], motivating explicit bidirectional supervision.Source: §1 · p. 2
  • LinkVLA shares a discrete codebook for language and trajectories, adds action-to-text understanding for bidirectional consistency, and uses coarse-to-fine generation to reduce latency.The contribution list characterizes the framework as a structural bridge, an explicit semantic alignment objective, and a lower-latency generation scheme.Source: §1 · p. 2

2. Related Work

Prior autonomous-driving work advances end-to-end control and vision-language-action systems, but limitations in reasoning, modality alignment, and decoding efficiency remain. LinkVLA's shared language-action codebook and action-understanding objective address those gaps.

2.1. End-to-end Autonomous Driving

  • Despite varied end-to-end designs, these systems remain limited in complex reasoning, long-tail events, and human-interactive tasks.The surveyed approaches include integrated planning, vectorized representations, sparse and parallel architectures, and generative trajectory methods.Source: §2.1 · p. 2

2.2. VLM and VLA for Autonomous Driving

  • Earlier VLMs explain actions or textualize driving inputs, while VLAs directly output actions; a modality gap still leaves language-action misalignment.LinkVLA's overview instead places language and action tokens in one shared codebook.Source: §2.2 · pp. 2–3, Figure 2 · p. 3

2.3. Language and Action Alignment

  • AutoVLA's stepwise action generation is inefficient, and misalignment persists in ORION [8] and SimLingo [37]; LinkVLA adds action understanding without extra data curation.The paper draws on findings that multimodal understanding and generation can benefit one another [47] [54] [62].Source: §2.3 · p. 3

3. Method

LinkVLA unifies language and trajectory tokens in one VLM, adds reciprocal trajectory-to-language supervision, and decodes actions through endpoint-based coarse initialization and parallel refinement. Its method combines shared tokenization, bidirectional learning, and two-stage action generation.

3. Method

  • LinkVLA combines a unified autoregressive language–action framework, reciprocal action understanding, and coarse-to-fine decoding to target alignment and latency.These are the three methodological components outlined for the model.Source: §3 · p. 3

3.1. Unified Tokenization Framework

  • Continuous trajectories become BEV-cell tokens in a codebook merged with text vocabulary; signed-log coordinates prioritize near-field precision, and spatial soft labels preserve grid adjacency.The Gaussian target assigns probability to neighboring action tokens, smoothing minor ground-truth errors.Source: §3.1 · p. 4

3.2. Unified Language-Action Understanding and Generation

  • The same decoder learns action generation and its reciprocal, reconstructing the instruction from vision and executed actions under a weighted two-loss objective.The two losses are combined as Ltotal = Lgeneration + λLunderstanding.Source: §3.2 · pp. 4–5

3.3. Coarse-to-Fine Action Generation

  • C2F replaces T sequential waypoint passes with endpoint prediction and coarse initialization followed by parallel trajectory refinement.This reduces trajectory generation from sequential decoding to two stages.Source: §3.3 · p. 5
  • Training reorders targets to teach endpoint prediction, then uses endpoint-interpolated coarse waypoint tokens as inputs for fine-trajectory refinement.The coarse trajectory is linearly interpolated from the ground-truth endpoint during training.Source: §3.3 · p. 5

4. Experiments

The experiments evaluate LinkVLA on closed-loop driving, instruction-following, and language benchmarks, with implementation details and component comparisons. LinkVLA improves driving and instruction-following results while reducing trajectory-generation latency.

4.1. Settings

  • The experiments use 220 CARLA v2 closed-loop routes across 44 interactive scenarios, plus Action Dreaming instruction-following and DriveLM-hard VQA and commentary evaluations.Bench2Drive reports Driving Score, Success Rate, Efficiency, Comfortness, and Multi-Ability; instruction following is measured by success rate.Source: §4.1 · pp. 5–6

4.2. Implementation details

  • Action coordinates are log-transformed and discretized into a 56 × 101 grid of 5,656 tokens; training uses InternVL2-1B with AdamW and LoRA adaptation.Inference first generates a textual rationale, then predicts the action sequence; unified training alternates action-understanding and action-generation supervision.Source: §4.2 · pp. 6–7

4.3. Results

  • Driving Score 91.01 and Success Rate 74.55 surpass SimLingo’s 85.07 and 67.27, while multi-ability performance reaches 73.40 versus 67.28.LinkVLA also leads on Efficiency (255.84) and Comfortness (34.62).Source: Table 1 · p. 6, §4.3 · p. 7
  • Latency falls from 361 ms for the auto-regressive variant to 48 ms with C2F, while Driving Score rises from 90.66 to 91.01.Compared with SimLingo, C2F gains 5.94 Driving Score points with a 14 ms latency increase; Table 2 reports average per-step inference time on H20 GPU.Source: Table 2 · p. 6, §4.3 · p. 7

4.4. Comparison Analysis

  • The final instruction-following configuration reaches 87.16% mean success, with peak scores of 96.48% for Faster, 74.73% for Target Speed, and 97.42% for Lane Change.The evaluation uses the Action Dreaming dataset; Table 3 identifies alignment as unified training.Source: §4.4 · p. 7, Table 3 · p. 8
  • Across DriveLM-VQA and commentary, adding alignment after tokenization and C2F yields the best language-understanding results, improving all reported metrics.With C2F, SPICE reaches 71.3 on DriveLM-VQA and 53.6 on commentary; Table 4 defines S/B/R as SPICE/BLEU/ROUGE-L.Source: §4.4 · p. 7, Table 4 · p. 8
  • Tokenization raises Driving Score from 85.07% to 89.57% and Success Rate from 67.27% to 73.18%; adding C2F alone gives only a marginal Driving Score gain and slightly lowers Success Rate.Adding alignment to tokenization and C2F yields 91.01% Driving Score and 74.55% Success Rate; soft labels raise Success Rate from 72.73% to 74.55%, and the two navigation forms have comparable closed-loop performance.Source: §4.4 · pp. 7–8, Table 5 · p. 8, Table 6 · p. 8

5. Conclusion

LinkVLA combines shared language-action tokens, bidirectional captioning, and coarse-to-fine decoding to improve alignment and efficiency. It reports an 86% latency reduction and gains in instruction following and driving performance on closed-loop benchmarks.

5. Conclusion

  • LinkVLA is presented as a model designed to improve language-action alignment and inference efficiency in autonomous driving.Source: §5 · p. 8
  • LinkVLA’s coarse-to-fine decoder cuts inference latency by 86%, while the model gains in instruction following and driving performance on closed-loop benchmarks.The conclusion describes this combination as a practical path toward reliable and responsive language-guided agents for real-world deployment.Source: §5 · p. 9
  • A shared codebook for language and action tokens, paired with bidirectional captioning, is the core strategy for deep cross-modal consistency.Source: §5 · p. 9

Supplementary Material

The supplementary material describes non-uniform waypoint quantization using log-transformed coordinates and compares its grid and waypoint distributions with uniform grids.

A.1. Log-Coordinate Transformation

  • A non-linear transformation is applied independently to waypoint coordinates x and y before quantization.Source: §A.1 · p. 12
  • The log-coordinate scheme prioritizes waypoint precision near the ego-vehicle.Source: §A.1 · p. 12
  • Figure S1 compares uniform and log token grids alongside the waypoint distributions under each grid.Source: Figure S1 · p. 12

A.2. Number of Action Tokens

This supplement evaluates how the number of action tokens affects Bench2Drive closed-loop driving performance, varying the symlog scaling factor that determines coordinate compression and bin widths.

A.2. Number of Action Tokens

  • The token-count effect is evaluated on the Bench2Drive benchmark in closed-loop evaluation.Source: §A.2 · p. 12, Table S1 · p. 12
  • Varying symlog scaling factor k from 5.0 to 10.0 increases the number of action tokens from 5,656 to 7,245.Source: §A.2 · p. 12
  • The factor k maps physical coordinates in meters to transformed space, setting compression before binning and effective bin widths in the original coordinate space.Source: §A.2 · p. 12

A.3. Spatial Soft-Labeling

The supplement varies the spatial soft-labeling spread parameter σ in Bench2Drive closed-loop evaluation; larger σ broadens the Gaussian target over neighboring action tokens.

A.3. Spatial Soft-Labeling

  • Larger σ broadens Gaussian smoothing and distributes probability over a wider neighborhood of action tokens.Source: §A.3 · p. 12
  • The spread parameter σ is varied from 1.2 to 3.0 to evaluate its effect on Bench2Drive driving performance.Source: §A.3 · p. 12, Table S2 · p. 12
  • The target distribution is a normalized 2D Gaussian centered on the ground-truth token, with Z ensuring its probabilities sum to 1.Source: §A.3 · p. 12

B.1. Action Dreaming

The appendix defines Action Dreaming’s simulator-based evaluation and category-specific success criteria, then describes the VQA-DriveLM data and heuristic commentary labels used by SimLingo.

B.1. Action Dreaming

  • Action Dreaming evaluates ego-trajectory predictions in an offline, nonreactive simulator using Success Rate, with simulator states and object forecasts supporting feasibility checks.The simulator offers modes including collision, speed changes, target speed, lane changes, and stop.Source: §B.1 · p. 13
  • Collision and speed-task success use trajectory or speed criteria, including ADE comparisons, a slope threshold of 0.05 times current speed, and a target-speed range of 20%.For collision cases with similar expert and dreamer paths, predicted speed must also be within 30% of dreamer speed; lane-change and stop criteria use final location and minimum speed, respectively.Source: §B.1 · p. 13

B.2. VQA-DriveLM

  • VQA-DriveLM uses 28M heuristically generated question–answer pairs over 1M Town 12 frames, with keyframe-based evaluation and up to 20 paraphrases per question.The pairs come from the DriveLM-Carla dataset through the SimLingo data-creation pipeline, rather than the original DriveLM release.Source: §B.2 · p. 13

B.3. Commentary

  • SimLingo commentary labels combine a route action with justification, a speed action and reason, and—near junctions—a notice about other vehicles.The labels are generated from saved simulator state using heuristic, template-based rules; speed reasons draw on IDM features.Source: §B.3 · p. 13

C. Qualitative Results

The qualitative results show representative closed-loop driving scenarios, including lane changes around construction and waiting for traffic gaps. The examples pair current-speed context with route and speed instructions.

C. Qualitative Results

  • Figure S2 presents representative closed-loop CARLA scenarios, including intersections and obstacle avoidance.The accompanying examples include waiting for a traffic gap before changing lanes and going around a construction site.Source: C. Qualitative Results · p. 13, Figure S2 · p. 14
  • One example instructs a stationary vehicle to wait for a traffic gap before changing into the lane with oncoming traffic.The displayed current speed is 0.0 m/s.Source: C. Qualitative Results · p. 14
  • Construction-site examples pair instructions to go around the site with lane and speed guidance, including accelerating to enter the oncoming lane.The examples show current speeds of 2.5, 7.9, and 8.4 m/s across the displayed prompts.Source: C. Qualitative Results · p. 14

Related papers

Loading 2603.01441v1…