Source-linked AI summary
RSKD: Grounding Semantic Consistency in Driving VLMs via Distillation
Chuchu Xie, Yang Zhong, Kangan Qian, Hao Ye, Kun Jiang, Diange Yang
TL;DR
Uniform token-level matching can dilute planning-relevant semantics in driving answers, and distillation must also account for whether teacher predictions are semantically consistent. RSKD localizes task-critical phrases and routes teacher and ground-truth supervision by phrase-level consistency; it reports gains on DrivingVQA and LingoQA and competitive performance on MapLMv2.
Problem
Uniformly matching answer tokens can dilute supervision on planning-relevant driving semantics.
Source: §1 · p. 1Method
RSKD localizes task-critical phrases in ground-truth answers and uses phrase-level semantic consistency to balance JSD distillation with ground-truth supervision.
Source: §1 · p. 1Results
RSKD gains on the two QA benchmarks and remains competitive on MapLMv2 under the reported setting.
Source: §1 · p. 2Takeaways & Limitations
The reported comparisons highlight the need to identify task-critical semantics and assess whether teacher predictions preserve their ground-truth meaning.
Source: §5 · p. 4Evaluation is currently limited to a single teacher–student pair.
Source: §4.4 · p. 4
Abstract
from arXiv · showhide
Knowledge distillation can transfer large multimodal models to compact models for autonomous driving, but uniform token-level matching can dilute planning-relevant semantics and amplify teacher errors. We propose RSKD, a semantic-consistency-guided distillation framework that routes supervision according to both the task importance of an answer phrase and its semantic consistency with the teacher prediction. Task-critical semantic phrases are localized in the ground-truth answer, and a phrase-level consistency score controls the balance between Jensen--Shannon divergence (JSD) distillation and ground-truth supervision. Teacher information is emphasized only where it is semantically consistent, while semantically inconsistent phrases receive stronger ground-truth supervision. We evaluate RSKD on DrivingVQA, LingoQA, and MapLMv2. RSKD improves over the compared student-based baselines on DrivingVQA and LingoQA and remains competitive on MapLMv2.
1. INTRODUCTION
Driving VLM distillation must preserve planning-relevant semantics while accounting for possible teacher errors. RSKD localizes task-critical phrases and routes supervision by semantic consistency, with reported gains on DrivingVQA and LingoQA and competitive MapLMv2 performance.
1. INTRODUCTION
- Large MLLMs support driving scene understanding and reasoning, but their computational and memory costs challenge practical deployment, motivating distillation to compact students.Source: §1 · p. 1
- Uniform token matching can dilute planning-relevant driving semantics, while token-level learning signals do not establish semantic criticality or teacher correctness.Source: §1 · p. 1
- RSKD localizes task-critical phrases in ground-truth answers and uses phrase-level consistency to route JSD distillation versus ground-truth supervision.Source: §1 · pp. 1–2
- RSKD gains on DrivingVQA and LingoQA and remains competitive on MapLMv2 under the reported setting.Source: §1 · p. 2
2. RELATED WORK
Prior distillation work aligns teacher and student distributions or representations, while fine-grained methods adapt supervision using token-level signals. RSKD differs by identifying task-critical driving phrases and evaluating teacher consistency on them.
2. RELATED WORK
- Distribution- and representation-alignment approaches do not explicitly identify task-critical semantic phrases in driving answers.Source: §2 · p. 2
- Fine-grained methods adapt supervision using divergence, learning difficulty, or uncertainty signals, whereas RSKD localizes task-critical phrases and evaluates teacher consistency on them.Source: §2 · p. 2
3. METHOD
RSKD replaces uniform token-level distillation with phrase-aware supervision: it identifies driving-critical phrases, estimates teacher consistency with the ground truth, and routes JSD and ground-truth supervision accordingly.
3.1. Problem Formulation and Conventional Distillation
- RSKD combines token-level JSD distillation with ground-truth supervision, while noting that uniform weighting ignores semantic importance and teacher consistency.The conventional objective uses α to control ground-truth supervision strength.Source: §3.1 · p. 2
3.2. Framework Overview
- The framework localizes task-critical phrases, estimates teacher semantic consistency on them, then broadcasts phrase scores to answer tokens to route supervision.The teacher remains frozen during semantic extraction and distillation.Source: §3.2 · p. 2
3.3. Teacher-guided Task-Critical Semantic Localization
- Task-critical phrases are extracted by annotating the known ground-truth answer, then aligned to answer tokens to form a mask for consistency-aware supervision.Phrase alignment uses exact token-subsequence matching.Source: §3.3 · p. 2
3.4. Semantic Consistency Estimation
- For each critical phrase, the teacher prediction is reconstructed under teacher forcing and compared semantically with the phrase to produce a consistency score.Higher agreement increases reliance on teacher distributions; lower agreement increases ground-truth supervision.Source: §3.4 · p. 3
3.5. Semantic-consistency-aware Knowledge Routing
- Critical tokens with high semantic consistency receive stronger JSD supervision, while low-consistency tokens receive stronger ground-truth supervision.Each supervision branch is normalized by its own weight sum, so routing does not impose a fixed global JSD-to-SFT ratio.Source: §3.5 · p. 3
4. EXPERIMENTS
Across three benchmarks, RSKD leads student baselines on DrivingVQA and LingoQA and is competitive on MapLMv2; ablations examine phrase localization and consistency-aware routing.
4.1. Experimental Setup
- Official-metric comparisons cover three benchmarks; trained students share one training mixture, and Qwen3-VL-4B-Instruct is adapted with LoRA while its visual encoder and aligner remain frozen.Hyperparameters were selected on validation data, with no test data used.Source: §4.1 · pp. 3–4
4.2. Main Results
- RSKD leads student results on DrivingVQA (16.60) and LingoQA (0.672), and reaches 81.82 on MapLMv2, nearly matching SFT (81.85).The teacher scores 79.00 on MapLMv2.Source: Table 1 · p. 4, §4.2 · p. 4
4.3. Ablation Study
- On the reduced 50% LingoQA subset, random spans score 0.474 versus 0.480, removing consistency scores 0.475, and removing SFT routing scores 0.468.Removing SFT routing produces the largest drop.Source: Table 2 · p. 4, §4.3 · p. 4
4.4. Limitations
- RSKD depends on teacher phrase extraction and frozen MiniLM embeddings, uses thresholds and rule-based contradiction detection that may need recalibration, adds training computation, and is evaluated with one teacher–student pair.Source: §4.4 · p. 4
5. CONCLUSION
RSKD routes teacher and ground-truth supervision according to task-critical phrase localization and semantic similarity. Experiments show gains over distillation baselines on two QA benchmarks and competitive performance on MapLMv2.
- RSKD localizes task-critical semantic phrases and routes teacher and ground-truth supervision according to semantic similarity.Source: §5 · p. 4
- Experiments show gains over distillation baselines on DrivingVQA and LingoQA, with competitive performance on MapLMv2 when the teacher advantage is limited.Source: §5 · p. 4
- The conclusion emphasizes identifying task-critical semantics and checking whether teacher predictions preserve their ground-truth meaning.Source: §5 · p. 4