Comparative study of adapting pre-trained models for driving behavior video captioning

• Natural-language captions could help users understand machine behavior and support text-based scenario search, rule learning, and assessment of operational design-domain coverage.The introduction also presents language as a compact, high-level representation that can retain decision-relevant information.Source: §1 · p. 1
• The study aims to improve understanding of driving videos by applying multiple encoder-decoder pairs and large language models to feature understanding.Source: §1 · p. 1
• The paper compares different adaptation methods for driving-video captioning.Source: §1 · p. 1

• ADAPT [1] jointly trains driving-caption and vehicle-control prediction tasks using a shared video representation, and reports state-of-the-art automatic-metric and human-evaluation performance on BDD-X [2].Its work also includes ablation studies and deployment in simulator environments and the real world.Source: §2 · pp. 1–2
• The authors cite ADAPT [1] as a direct motivation but report that Docker setup limitations prevented them from directly using or updating its code.Source: §2 · p. 2
• GAIA-1 [3] uses video, text, and action inputs to generate driving scenarios, while LINGO-2 is described as a closed-loop vision-language-action driving model tested on public roads.The report notes that LINGO-2 had no publication available at the time of writing and also mentions LingoQA [4].Source: §2 · p. 2

• The setting uses visual inputs, optionally accompanied by text, and a set of textual labels for the driving-video captioning task.Source: §3 · p. 2
• A pretrained general-purpose video-captioning model, potentially assembled from components such as a video encoder and text decoder, is adapted to the target task.Source: §3 · p. 2
• The model’s conditional probability distribution factorizes over output tokens, and training aims to maximize the probability of the target captions.The stated objective can vary with the adaptation approach.Source: §3 · pp. 2–3
• Figure 1 summarizes the route from the dataset and pretrained model to a driving-video captioning solution.Source: Figure 1 · p. 2

• Transformers are presented as the experiments’ underlying mechanism, with their ability to capture long-range dependencies and contextual information cited as useful across NLP tasks.Source: §4 · p. 3
• ViViT [5] studies transformer models for video classification and proposes variants that factorize spatial and temporal dimensions to handle long video-token sequences.Source: §4 · p. 3
• The report divides its approach discussion into pretrained models and adaptation methods, and points to a supplementary table comparing text and video transformers.Source: §4 · p. 3

4.1.1 TimeSformer GPT-2

• TimeSformer GPT-2 combines a TimeSformer video encoder with a GPT-2 decoder to incorporate temporal information alongside text.It was pretrained on VaTeX, a video-captioning dataset.Source: §4.1.1 · p. 3, §4.1 · p. 3

4.1.2 TimeSformer BERT

• TimeSformer BERT combines TimeSformer with BERT and is described as slightly less powerful than the GPT-2 version.Source: §4.1.2 · p. 4

4.1.3 Video-LLaVA

• Video-LLaVA processes video in a text-like format, accepts prompts, and uses image/video understanding followed by instruction tuning.Its image pretraining uses LLaVA data and its video pretraining uses VALLEY data.Source: §4.1 · p. 3, §4.1.3 · p. 4

4.2.1 Full-fine tuning

• Full fine-tuning updates a pretrained model on driving-captioning data, potentially freezing some layers and replacing final layers for the target task.The passage notes that this approach can be slow and compute-intensive for large models, with overfitting risk when data are insufficient.Source: §4.2.1 · p. 4

4.2.2 Low-Rank Adaptation fine tuning

• LoRA adds low-rank matrices to selected layers while keeping most pretrained parameters fixed.The study encodes task-specific parameter changes with a smaller parameter vector than the original model.Source: §4.2.2 · pp. 4–5

4.2.3 Prompt Engineering and In-context learning

• In-context learning adapts a pretrained model by showing examples in the prompt, with zero-, one-, or many-shot variants.Its performance is bounded by the model’s existing knowledge and the quality of the examples.Source: §4.2.3 · p. 5
• Prompt engineering customizes input prompts without changing model parameters, while the study uses it with Video-LLaVA rather than fine-tuning that model.Full fine-tuning and LoRA for Video-LLaVA were not completed because of time and computational restraints.Source: §4.2.3 · p. 5

5.1 BDD100K

• BDD100K supplies an auxiliary task predicting weather, surroundings, and time of day to gauge TimeSformer’s understanding.The dataset contains 100,000 videos, with annotations at key frames every ten seconds.Source: §5.1 · p. 5

5.2 BDDX

• BDD-X provides video-label pairs for driving captioning, with more than 77 hours of video across 7,000 videos and a predefined train, validation, and test split.The processed dataset records video name, segment start and end times, caption, and reasoning.Source: §5.2 · p. 6
• The primary BDD-X task predicts both the vehicle’s action and why it takes that action, evaluated with BLEU-4, CIDEr, METEOR, and ROUGE-L.Source: §5.2 · p. 6
• The BDD-X comparison includes pretrained and fully fine-tuned TimeSformer GPT-2 and BERT models, plus TimeSformer GPT-2 LoRA configurations varying rank with constant alpha.The study also tests three prompt formulations with Video-LLaVA; Table 1 is identified as the comparison of methods.Source: §5.2 · p. 6, Table 1 · p. 6

6.1.1 Pretrained models

• Pretrained models fall well short because their training data differs from driving videos, which use an ego-vehicle viewpoint; GPT2-TimeSformer appears more expressive, with more parameters.Source: §6.1.1 · p. 7

6.1.2 Full fine tuning

• Full fine-tuning progresses from binary-like outputs after the first epoch toward more expressive, detailed outputs in the final epochs.Frequent outputs are learned first.Source: §6.1.2 · p. 7

6.1.3 LoRA fine tuning

• LoRA initially produced repetitive outputs and unexpected sentence structures with linear layers; adding embedding and 1D and 2D convolution layers improved outputs.Source: §6.1.3 · p. 7

6.1.4 Prompt engineering

• Three prompts ranging from simple to complex improved performance, but remained well below fine-tuning outputs.Source: §6.1.4 · p. 7

6.2 Discussion of results

• Full fine-tuning outperforms the other experiments on all metrics and some ADAPT metrics, while different LoRA configurations give similar results.The authors attribute the latter pattern to possible fitting issues.Source: §6.2 · p. 7

7 Conclusion

• TimeSformer-GPT2 is much smaller and less complex than ADAPT yet performs comparably on many metrics.Source: §7 · p. 7
• The report says more data could maximize these models’ capabilities, while adapting more capable models such as Video-LLaVA would require significant computational resources and time.Source: §7 · p. 7
• Full fine-tuning yields the best results but is computationally intensive, making adaptation choice dependent on available time and resources.Source: §7 · p. 7
• Simple prompt engineering can sometimes already lead to good results.Source: §7 · p. 7

A.1 Background on metrics

• BLEU-4 [16] compares candidate and reference n-grams, focusing on 4-grams, and uses a brevity penalty against shorter translations.Source: §A.1 · pp. 7–8
• CIDEr [17] evaluates generated captions against multiple references using word similarity and semantic relevance, with n-gram similarity weighted by TF-IDF.Source: §A.1 · p. 8
• ROUGE-L [18] measures recall through the longest common subsequence shared by candidate and reference summaries.Source: §A.1 · p. 8
• METEOR [19] scores word matches using precision, recall, alignment, stemming, synonyms, and word order.Source: §A.1 · p. 8