Source-linked AI summary
Mid-Training Language Models on Raw Video
Jaedong Hwang, Xiaoqian Shen, Ernie Chang, Changsheng Zhao, Chong Zhou, Saksham Suri, Qi Qian, Zechun Liu, Lemeng Wu, Qinsi Wang, Raghuraman Krishnamoorthi, Wei Wen
TL;DR
The paper asks whether raw web video can serve as mid-training data for an existing pretrained language model without captions or text loss. It trains the model to predict continuous visual tokens and reports gains on video and image benchmarks while preserving text performance.
Problem
Raw web video is rarely used to continue training existing language models, despite the expense and errors associated with captioning it at scale.
Source: §1 · p. 1Method
The paper mid-trains a pretrained language model on raw video by predicting next continuous visual tokens without captions or language loss.
Source: §1 · p. 2, Architecture · p. 4Results
Takeaways & Limitations
Next-frame caption prediction offers no advantage over raw visual-token prediction, so video mid-training can remain self-supervised without captioning pipelines.
Source: §5 · p. 8Gains plateau after the first 0.3 epochs, and whether they arise from temporal structure or added visual-token exposure remains unresolved.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Multimodal large language models learn mostly from paired image-text data or annotated video, and raw web video is rarely used to further train an existing language model. We study whether raw video, with no captions and no text loss, can serve as mid-training data for a pretrained language model. Frames are encoded into continuous visual tokens, and the language model learns to predict the next visual token. We mid-train Qwen3-1.7B on raw clips from YT-Temporal-1B and then apply the same image-text instruction tuning to it and to the model without mid-training, so that the two differ only in mid-training. The mid-trained model scores 2.9 points higher on average across four video benchmarks and 5.1 points higher across ten image benchmarks, spanning perception, document, and chart tasks. Text performance is preserved even though mid-training includes no text, with an average of 48.9 across 14 text benchmarks compared with 48.0 for the model without mid-training. Analyses across training show that the image and video gains emerge within 30% of training and plateau thereafter, varying by less than 0.5 points. Predicting captions fails to outperform next-visual-token prediction, demonstrating that video mid-training can remain purely self-supervised without the computational overhead or labeling noise of automated captioning.
1 Introduction
The paper asks whether pretrained language models can continue training on raw web video without captions or text supervision, addressing a practical gap in multimodal pretraining. It reports gains on video and image benchmarks, preserved text performance, and no advantage from caption prediction over visual-token prediction.
1 Introduction
- Raw web video offers a way to continue training an existing pretrained language model, a regime not directly addressed by prior multimodal approaches.The paper notes that prior efforts train unified models from scratch or add prediction heads over curated clips.Source: §1 · pp. 1–2
- +2.9 points on average across four video benchmarks and +5.1 across ten image benchmarks accompany a text-score increase from 48.0 to 48.9.The comparison is against a model given the same instruction tuning without mid-training.Source: §1 · p. 2
- The proposed mid-training predicts next visual tokens from raw video without captions or annotations.Frames are encoded into continuous visual tokens, and the model is trained without text loss.Source: §1 · p. 2
- Captioning current or next frames provides no advantage over self-supervised next-visual-token prediction.The paper presents avoiding captioning as a way to avoid its computational overhead and labeling noise.Source: §1 · p. 2
2 Related Work
Prior video-learning work includes visual-domain self-supervision and multimodal pretraining, but the paper focuses on continued training of an existing language model using raw web video. Its approach predicts continuous visual tokens without captions or text loss.
2.1 Learning from Video without Language
- Video models trained from scratch in the visual domain use predictive objectives, while language is typically introduced later through probing or fine-tuning.Examples include masked spatiotemporal patches, discrete visual-token generation, and latent-feature prediction.Source: §2.1 · p. 2
2.2 Video as Training Data for Language Models
- Unlike approaches that train multimodal transformers from scratch or add prediction heads to curated clips, this work continues a pretrained LLM on raw web video without captions or text loss.The model predicts the next continuous visual token, extending autoregressive pretraining rather than using video as an instruction-tuning signal.Source: §2.2 · pp. 2–3, Figure 1 · p. 3
2.3 Vision-Language Models and Mid-Training Data
- Vision-language models commonly use supervised multimodal instruction tuning, while mid-training approaches may rely on synthesized clip captions.Predicting next-frame captions gives no gain over continuous visual-feature prediction, allowing use of web collections without annotation.Source: §2.3 · p. 3
3 Next-Token Prediction on Visual Tokens
The method treats video as a serialized stream of continuous visual tokens and trains a causal language model to predict each next token. A feature-regression objective operates on that stream without captions, a discrete codebook, or language loss.
3 Next-Token Prediction on Visual Tokens
- Raw video frames are encoded into continuous features, serialized, and fed to the causal language model for self-supervised next-token prediction.This retains the decoder-only architecture and avoids captioning or discrete visual codebooks.Source: §3 · p. 3, Architecture · p. 3
Architecture.
- At each position, a two-layer prediction head maps the preceding-context hidden state to the next visual feature, which is trained against a stop-gradient target.Cosine distance supplies the regression objective; causal attention unifies within-frame patch prediction and prediction across frame boundaries.Source: Architecture · p. 4
4 Experiments
The experiments evaluate downstream performance across video, image, and text benchmarks, then examine training dynamics and objective ablations.
4 Experiments
- The experimental section begins by outlining the setup.The supplied overview does not specify the setup details.Source: §4 · p. 4
- The paper first outlines its experimental setup and then reports downstream evaluations across video, image, and text.Source: §4 · p. 4
- The experiments analyze cross-modal training dynamics and compare mid-training objectives through ablations.These analyses follow the benchmark evaluations.Source: §4 · p. 4
4.1 Experimental Details
The experiments connect a pretrained vision encoder to a causal language model, train on sampled video clips, and compare captioning targets under a shared mid-training and instruction-tuning setup. Evaluation covers video, image, and text tasks using benchmark-specific scoring protocols.
4.1.1 Architecture
- The architecture projects pretrained vision-encoder outputs into the language model through a two-layer GELU MLP, with a lightweight prediction head used during mid-training.The vision backbone is SigLIP SO400M/14 at 384 × 384 resolution, and the language model is Qwen3-1.7B.Source: §4.1.1 · p. 4
4.1.2 Dataset
- The video dataset contains 6.99M contiguous 16-frame clips, sampled from 17M accessible videos at 0.2 fps, totaling 64.4B visual tokens.The source collection is YT-Temporal-1B; subsequent supervised fine-tuning uses 3.9M images and 3.2B tokens from LLaVA-OneVision-Data.Source: §4.1.2 · p. 4
4.1.3 Captioning Mid-Training Data for Baselines
- Caption baselines use Qwen3-VL-30B-A3B to caption consecutive frames separately, targeting either the current frame's caption or the next frame's caption.The prompt asks for visible content only, and each caption describes its frame's action without referring to the other frame.Source: §4.1.3 · pp. 4–5, Figure 2 · p. 5
4.1.4 Training
- All model parameters are trained end-to-end for one epoch, then models are instruction-tuned for one epoch on LLaVA-OneVision data using the same training parameters.Mid-training runs for 27,300 steps across 128 H100 GPUs with an effective batch of 256 clips and a 16k-token context window.Source: §4.1.4 · p. 5
4.1.5 Benchmarks
- The benchmark suite groups video tasks by temporal and question-answering ability, image tasks by visual task type, and text tasks by language capability.Video includes long-form egocentric understanding, general video QA, and fine-grained temporal reasoning; image includes text-rich understanding, perception and diagrams, text recognition, and visual reasoning.Source: §4.1.5 · p. 5
4.1.6 Evaluation Protocol
- Image and video multiple-choice tasks use option-letter log-likelihood, while open-ended queries are decoded greedily; text tasks use zero-shot ranking or few-shot greedy decoding as appropriate.GPQA is evaluated with lighteval in non-reasoning mode with an 8,192-token generation budget.Source: §4.1.6 · p. 5, Table 1 · p. 6, Table 2 · p. 6
4.2 Video Mid-Training Improves Video and Image Understanding
Video mid-training improves video and image benchmark performance relative to instruction tuning without mid-training. The text results show lower reasoning scores after multimodal instruction tuning than in the original language model, with smaller drops for the mid-trained model.
4.2 Video Mid-Training Improves Video and Image Understanding
- Video mid-training improves every video benchmark by 2.9 points on average and all ten image benchmarks by 5.1 points on average.The largest video gain is 4.8 points on EgoSchema; image gains include 8.2 on ChartQA, 6.4 on DocVQA, and 6.1 on AI2D.Source: §4.2 · p. 6
- Relative to original Qwen3-1.7B, multimodal instruction tuning lowers GSM8K, MATH500, and BBH scores, with smaller drops for the video mid-trained model.The mid-trained model drops 18.6 points on GSM8K, 16.5 on MATH500, and 9.5 on BBH, versus drops of 23.5, 17.7, and 15.0 points without mid-training.Source: §4.2 · p. 6, Table 3 · p. 7
4.3 Image and Video Gains Appear Early in Mid-Training
Image and video gains appear early in mid-training and then remain nearly stable, while text performance stays close to the no-mid-training model. The benchmark trajectories show differences in gains across task types and in text-score stability.
4.3 Image and Video Gains Appear Early in Mid-Training
- Image and video average scores rise by 4.9 and 3.1 points within 0.3 epochs, then vary by less than 0.5 points over the remaining 0.7 epochs.The 0.3-epoch checkpoint corresponds to 19.5B visual tokens; average text performance stays within 0.9 points of the no-mid-training model across evaluated steps.Source: §4.3 · p. 7
- At full mid-training, task-type gains range from 5.8 points on text-rich images to 3.2 on text recognition, while reasoning benchmarks fluctuate by up to 9 points.Video gains are largest on EgoSchema and general video QA, while TempCompass improves by 1.5 points; commonsense, knowledge, coding, and GPQA stay within 3 points of baseline.Source: §4.3 · p. 7
4.4 Visual Next-Token Prediction Is Sufficient Without Annotations
The section finds that visual next-token prediction is sufficient for video mid-training, matching or outperforming caption-supervised alternatives. It also avoids the computational overhead and label noise of caption generation.
4.4 Visual Next-Token Prediction Is Sufficient Without Annotations
- At matched training budget, visual next-token prediction matches vision gains within 0.3 points while better preserving text capability than next-caption prediction.Next-caption prediction yields similar video and image scores but degrades language performance by 1.5 points.Source: §4.4 · p. 7, Table 4 · p. 8
- Current-caption prediction falls 5.0 points behind visual-token prediction on image understanding and shows inconsistent transfer across tasks.Source: §4.4 · p. 7
- Visual-token prediction keeps video mid-training self-supervised while avoiding caption generation’s computational overhead and label noise.Source: §4.4 · pp. 7–8
5 Discussion
The discussion concludes that raw-video mid-training improves video and image benchmark performance while retaining text performance despite no text loss. It also notes that gains appear early, while their source and generalization to larger models remain open questions.
5 Discussion
- The proposed objective mid-trains a pretrained language model on raw web video by predicting next visual tokens without text supervision.Source: §5 · p. 8
- 2.9 points higher on four video benchmarks and 5.1 points higher on ten image benchmarks, with the greatest image gains in text-rich understanding and general perception.Source: §5 · p. 8
- 48.9 average text score after mid-training versus 48.0 without it, but both models score below the original language model on GSM8K, MATH500, and BBH.The 5.5-point BBH margin is smaller than the 11.7-point discrepancy between instruction-tuning runs from the same initial state.Source: §5 · p. 8
- Image and video gains appear within 0.3 epochs, then scores vary by less than 0.5 points over the remaining 0.7 epochs.Whether denser temporal sampling or longer sequence horizons yield further gains remains open.Source: §5 · p. 8, Figure 4 · p. 9
- The discussion leaves open whether gains come from temporal structure or visual-token exposure, and whether the dynamics extend to larger language models.Source: §5 · p. 8