GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Models

1 Introduction

• CAVG addresses command grounding in changing traffic scenes, where ambiguous language and evolving visual conditions can lead to unsafe maneuvers.Source: §1 · p. 2
• Existing grounding approaches can overlook driving context or emotional dynamics, while integrated and Transformer-based models still face complex multi-object scenarios.Source: §1 · p. 2
• CAVG combines contextual cues and multimodal prediction of plausible outcomes with likelihoods to address complex, ambiguous driving situations.The model also categorizes subtle emotions in commanders’ language.Source: §1 · p. 2
• CAVG is described as adapting to commanders’ emotional context while maintaining contextual accuracy in grounding objects in dynamic traffic scenes.Source: §1 · pp. 2–3
• The framework uses pre-trained semantic and global image features for cross-modal attention, GPT-4 for emotion analysis, and a Region-Specific Dynamic layer to prioritize multimodal data.Source: §1 · p. 3
• The authors report that CAVG surpasses existing state-of-the-art methods, particularly on Talk2Car, including low-light, long-command, rainy-weather, and dense-urban scenarios.They also report less training time, superior performance, and enhanced interpretability.Source: §1 · p. 3

2.1 Visual Grounding

• Visual grounding methods localize command-relevant image regions; one-stage methods streamline inference but may struggle with dense or overlapping objects, while two-stage methods use proposals and linguistic alignment.CAVG adopts a two-stage approach extended to traffic context beyond bounding boxes.Source: §2.1 · pp. 3–4

2.2 Language-to-Image in Autonomous Driving

• Talk2Car provides a benchmark of commands tailored to self-driving cars, and subsequent AV grounding work explores cross-modal attention, command uncertainty, navigable regions, LiDAR grounding, and unified vision-language models.Source: §2.2 · p. 4

2.3 Emotion Classification

• Prior emotion-classification work includes BERT-based approaches, EmotionX-KU, and Sentiment GPT; this study uses GPT-4 to analyze commands’ emotional content and contextual nuances.Source: §2.3 · p. 4

2.4 Vision-and-Language Model

• Single-stream models integrate visual and linguistic modalities across encoder layers, whereas dual-stream models begin with within-modality interactions; the paper adopts a hybrid approach for AV tasks.Source: §2.4 · p. 4

3.1 Overview

• Five specialized encoders process the command, image, emotion, context, and cross-modal relationships before a multimodal decoder identifies destination regions.The decoder scores regions and selects the top-k candidates matching the command semantics.Source: §3.1 · p. 4, Figure 2 · p. 5

3.2 Text Encoder

• BERT converts WordPiece-tokenized commands into a contextual text vector, and is fine-tuned for autonomous-driving vocabulary, syntax, and command structures.The resulting embedding supports subsequent multimodal fusion and contextual analysis.Source: §3.2 · pp. 5–6

3.3 Emotion Encoder

• GPT-4 analyzes each command’s emotional tone, which the Emotion Encoder categorizes as urgent, commanding, or informative.The predicted emotion category is fused with the text vector.Source: §3.3 · p. 6

3.4 Vision Encoder

• CenterNet detects scene entities as RoIs, while ResNet-101 features capture fine details such as color and texture that can distinguish command-relevant objects.Figure 3 depicts ground-truth boxes in blue and CAVG outputs in green.Source: §3.4 · pp. 6–7, Figure 3 · p. 7

3.5 Context Encoder

• ViT represents the whole scene, capturing contextual cues beyond object boxes, while BLIP connects command words with visual semantics to form context-aware features.The depicted self-attention maps are described as supporting object segmentation without segmentation annotations or a complex self-supervised training recipe.Source: §3.5 · pp. 7–8, Figure 4 · p. 8

3.6 Cross-Modal Encoder

• The Cross-Modal Encoder uses multi-head attention to combine text, emotion, vision, and context representations into a fused cross-modal vector.Command and image position embeddings are generated with UNITER; the paper describes the fusion as a more efficient “hindsight fusion” strategy than the cited alternatives.Source: §3.6 · pp. 8–10

3.7 Multimodal Decoder

• The decoder ranks candidate RoIs using a Transformer Decoder and Region-Specific Dynamic attention, which weights region representations across encoder layers.Layer-weighted features yield hypothesis credibility scores; skip connections carry the high-level command vector into each decoder layer.Source: §3.7 · p. 10

4.1 Dataset

• Talk2Car contains 11,959 commands across 9,217 urban images from Singapore and Boston, with varied conditions and an average command length of 11 words.The dataset includes object-disambiguation and elaborate-query tasks; Figure 6 summarizes its word cloud and referred-object-number distribution.Source: §4.1 · p. 10, Figure 6 · p. 11

4.2 Evaluation Metric

• CAVG is evaluated using IoU 0.5, also called AP 50, which measures the percentage of predicted boxes whose IoU exceeds the 0.5 threshold against ground truth.The paper describes IoU as overlap area divided by union area.Source: §4.2 · p. 10

• CAVG trains for six epochs with AdamW and batch size 16, using BCE loss and cosine annealing warm restarts.Training uses an initial learning rate of 10^-4 and takes approximately an hour on an NVIDIA A100 40GB GPU.Source: §4.3 · p. 11
• BERT encodes commands into 768-dimensional text vectors, while GPT-4 classifies command emotion into aligned 768-dimensional embeddings.The text encoder allows up to 60 tokens; the emotion and text embeddings are concatenated and projected to dimension d.Source: §4.3 · p. 11
• Fast R-CNN with ResNet-101 extracts 36 region proposals, while a ViT-B context encoder combines the raw image with the text vector.The vision regions have 1024-dimensional features; the context embedding has dimension d.Source: §4.3 · p. 11
• A 16-head cross-modal encoder processes text, emotion, vision, and context vectors, followed by a 12-layer multimodal decoder with 12-head attention.The decoder layers also use linear projections, batch normalization, and dropout.Source: §4.3 · p. 11

4.4 Experiment Setup

• The full test set contains 2,447 samples, and the training, validation, and test splits have similar command word-count distributions.The split contains 8,349 training and 1,163 validation samples; the long-text subset uses commands longer than 23 words, while corner cases cover challenging visual or linguistic conditions.Source: §4.4 · p. 12, Figure 7 · p. 12, Figure 8 · p. 12

4.4.1 Performance Benchmarking Against SOTA Models on full Testset

• CAVG exceeds the four C4AV Challenge leaders on full-test IoU 0.5 by 4.8% to 11.0%.The reported margins are 4.8% over Stacked VL-BERT, 7.4% over CMRT, 7.9% over CMSVG, and 11.0% over ASSMR.Source: §4.4.1 · p. 13, Table 1 · p. 13

4.4.2 Performance Evaluation of Training on Reduced Datasets

• CAVG trained on 75% of the training data outperforms Stacked VL-BERT and CMRT by 5.07% and 7.96%, respectively.The paper also reports competitive results with 50% of training data, outperforming almost all of the top six SOTA baselines.Source: §4.4.2 · p. 13, Table 1 · p. 13

4.4.3 Performance Evaluation on Long-text and Corner-case Testsets

• On long-text commands, CAVG exceeds open-source SOTA baselines by 4.0%–10.1%; on corner cases, its reported accuracy gains are 1.9%–4.9%.The 75%-data variant rivals or surpasses Stacked VL-BERT on long text and outperforms other SOTA baselines on corner cases.Source: §4.4.3 · p. 14, Table 2 · p. 14, Table 3 · p. 15

4.4.4 Comparative Analysis of Model Inference Speed

• CAVG (Small) is 62.9% faster in inference than CAVG (baseline) and improves prediction performance by 2.1% compared with Stacked VL-BERT (1st).CAVG (baseline) averages about 0.037 seconds per sample.Source: §4.4.4 · p. 15

• Decoder layers 7–10 receive a larger fraction of attention weights, while the top layer is not dominant.RSD dynamically allocates weights across decoder layers based on hidden states, then integrates their outputs.Source: §4.5 · pp. 15–16, Figure 10 · p. 16

4.6.1 Ablation Study for Core Module

• Model H, with all components, consistently outperforms across the evaluation metric; replacing its Vision Encoder with R-CNN lowers IoU0.5 by 5.8%.The paper attributes the Vision Encoder's advantage to its single-stage, anchor-free design, which directly locates object centers and regresses object dimensions.Source: §4.6.1 · p. 16
• Removing context and emotion, text and cross-modal, visual text, decoder UNITER, or three components reduces performance relative to Model H.Model C drops 6.6% in metric; omitting UNITER or the Emotion Encoder lowers IoU0.5.Source: §4.6.1 · p. 16

4.6.2 Case Study for the Context Encoder and Cross-modal Attention

• In challenging examples, adding the Context Encoder and cross-modal attention markedly improves predictions by incorporating scene cues beyond object boxes.The Context Encoder highlights lane boundaries, signs, pedestrian paths, lighting, and weather cues; cross-modal attention fuses these with the command.Source: §4.6.2 · pp. 16–17, Figure 11 · p. 17

4.6.3 Subtle Value Analysis of the Emotion Encoder

• The Emotion Encoder's broad-test-set gains appear modest, but the authors argue its value is greater in less frequent, critical emotion-laden scenarios.They illustrate an anxious passenger's “drive fast” command as a case where emotional urgency could favor a route with fewer obstacles.Source: §4.6.3 · p. 17
• The authors present emotion-aware trajectory planning as a future direction beyond grounding commands in visual context.They envision planning that follows explicit commands while accounting for the commander's emotional state.Source: §4.6.3 · pp. 17–18

• CAVG uses context to model relationships between boxes, an Emotion Encoder for implicit command emotion, cross-modal attention to harmonize inputs, and RSD attention to tune multimodal fusion.The paper describes these design choices as supporting robust predictions and feature interpretability in complex scenarios.Source: §4.7 · p. 18
• For “Oh, right here! Park behind that white car,” the Emotion Encoder classifies the command as urgent and fuses that signal with the text vector.The paper contrasts this with a prediction in front of the approaching car when emotional context is absent.Source: §4.7 · pp. 18–19, Figure 13 · p. 19
• Cross-modal attention links command word vectors to image-region features, so “just” and global cues such as zebra crossings can guide vehicle selection.Regions more semantically aligned with command keywords receive higher attention scores.Source: §4.7 · p. 19

• Talk2Car examples show predictions in low light, complex urban interactions, ambiguous commands, and rainy conditions with compromised visibility.The paper presents these as evidence of robustness across a range of environmental conditions.Source: §4.8 · p. 19
• The Figure 14 illustration links the keyword “just” to vehicle identification through word vectors and regional features.Source: Figure 14 · p. 20

• The anonymous survey included 174 participants evaluating five models on response accuracy, inference time, and user experience.Participants also selected their most preferred model.Source: §5 · p. 20
• CAVG outperforms existing models by 5.2% to 10.6% in comprehensive user ratings and scores highest in response accuracy and user-experience alignment while maintaining faster response time.CAVG and CAVG (75%) both receive relatively high user ratings compared with other models.Source: §5 · p. 20
• Participant feedback points to further model-structure optimization to balance prediction accuracy with faster inference for real-time demands.Source: §5 · p. 20

6 Conclusion

• CAVG surpasses existing state-of-the-art methods on Talk2Car and maintains superior performance with 50%–75% of the full training data.The paper also reports robustness in low-light night scenes, long-text commands, rainy days, and dense urban streets.Source: §6 · p. 21
• Five encoders—Text, Emotion, Vision, Context, and Cross-Modal—feed a Multimodal Decoder using multi-head cross-modal attention and RSD layer attention.The encoders process language, emotional context, traffic scenes, and textual-visual information for multimodal predictions.Source: §6 · p. 21
• Future work will add bird’s-eye-view cues, lidar feedback, and HD maps, and investigate integration with trajectory planning.Source: §6 · p. 21
• Figure 16 contrasts CAVG’s red output boxes with blue ground-truth boxes across limited-visibility, ambiguous-command, and multi-agent scenes.Each scenario also displays its associated natural-language command.Source: Figure 16 · p. 22
• Figures 17–18 report participant age and driving-experience distributions, plus user-study scores for response accuracy, inference time, and user experience and favorite-model selections.Source: Figure 17 · p. 22, Figure 18 · p. 23