Source-linked AI summary
InternVideo-Next: Towards General Video Foundation Models without Video-Text Supervision
Chenting Wang, Yuhan Zhu, Yicheng Xu, Jiange Yang, Lang Lin, Ziang Yan, Yali Wang, Yi Wang, Limin Wang
TL;DR
Video-text methods can miss detailed motion and spatial knowledge, while self-supervised video pretraining has lagged on semantic tasks. InternVideo-Next combines semantic-guided pixel reconstruction with latent prediction in a coherent space, achieving state-of-the-art results across video-understanding tasks, including surpassing video-text-pretrained counterparts on Kinetics-400 and SSv2.
Problem
Video-text pretraining can miss detailed motion, causal structure, and 3D priors, while self-supervised video pretraining has lagged on semantically demanding benchmarks.
Source: §1 · pp. 1–2Method
InternVideo-Next uses an Encoder–Predictor–Decoder framework and two-stage pretraining to establish a semantic, detail-preserving latent space and predict within it.
Source: §1 · p. 2Results
Takeaways & Limitations
The method provides a unified, scalable pathway toward general-purpose world-understanding video models.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Large-scale video-text pretraining achieves strong performance but depends on noisy, synthetic captions with limited semantic coverage, often overlooking implicit world knowledge such as object motion, 3D geometry, and physical cues. In contrast, masked video modeling (MVM) directly exploits spatiotemporal structures but trails text-supervised methods on general tasks. We find this gap arises from overlooked architectural issues: pixel-level reconstruction struggles with convergence and its low-level requirement often conflicts with semantics, while latent prediction often encourages shortcut learning. To address these, we disentangle the traditional encoder-decoder design into an Encoder-Predictor-Decoder (EPD) framework, where the predictor acts as a latent world model, and propose InternVideo-Next, a two-stage pretraining scheme that builds a semantically consistent yet detail-preserving latent space for this world model. First, conventional linear decoder in pixel MVM enforces the predictor output latent to be linearly projected to, thus separable in pixel space, causing the conflict with semantic abstraction. Our Stage 1 proposes a conditional diffusion decoder and injects reliable image-level semantic priors to enhance semantics and convergence, thus bridging pixel-level fidelity with high-level semantic abstraction. Stage 2 further learns world knowledge by predicting frozen Stage 1 targets within this space, mitigating shortcut learning. Trained on public, unlabeled videos, InternVideo-Next achieves state-of-the-art results across benchmarks and provides a scalable path toward general video representation learning.
1. Introduction
InternVideo-Next addresses the gap between text-supervised video models’ semantic strengths and self-supervised methods’ access to video structure by aligning pixel reconstruction and latent prediction in a coherent semantic space. Its two stages build that space and then use it to learn temporal dynamics and world knowledge.
1. Introduction
- InternVideo-Next achieves state-of-the-art results across video-understanding tasks and surpasses video-text-pretrained counterparts on Kinetics-400 and SSv2.Figure 1 compares ViT-Large models using frozen-ViT probing across scene, motion, video-language, and implicit-world-knowledge tasks.Source: Figure 1 · p. 1, §1 · p. 3
- Video-text methods rely on noisy, limited-coverage annotations and often miss detailed motion, causal structure, and 3D priors, while self-supervised methods lag on semantic benchmarks.The paper frames these as complementary gaps in learning from captions versus raw video structure.Source: §1 · pp. 1–2
- The Encoder–Predictor–Decoder formulation exposes the predictor’s latent space, which should remain both semantically rich and detail-faithful to support world-model prediction.The authors argue this encourages completion through spatiotemporal relationships and implicit world knowledge rather than trivial correlations.Source: §1 · p. 2
- Stage 1 combines pixel reconstruction, a conditional diffusion decoder, and image-level semantic priors to build a structurally consistent, detail-preserving latent space.The diffusion decoder avoids requiring predictor latents to be linearly separable into pixels, while image-text priors provide cleaner semantics than video captions.Source: §1 · p. 2
- Stage 2 predicts frozen Stage 1 representations to learn spatiotemporal dynamics and causal relations while mitigating shortcut solutions.The student and teacher are initialized with Stage 1 weights.Source: §1 · pp. 2–3
2. Related Works
Prior video representation methods trade off semantic coverage against access to detailed visual and temporal structure. InternVideo-Next situates itself as a video-only approach that combines semantic priors with video reconstruction.
2. Related Works
- Video-text pretraining performs strongly on subject-semantic tasks but relies on noisy synthetic captions that often lack motion and spatial diversity.The passage contrasts this with shortcomings on depth, fine-grained motion, and causal-relation tasks.Source: §2 · p. 3
- Pixel-domain masked video methods capture low-level appearance, while latent prediction can suffer shortcut learning or semantic drift and miss detail or high-level information.The cited shortcomings span appearance-intensive, depth-estimation, and high-semantic tasks.Source: §2 · p. 3
- InternVideo and InternVideo2 fuse video and language knowledge at the weight or embedding level, whereas InternVideo-Next integrates CLIP-level semantic priors into video reconstruction.The authors characterize the earlier fusion approaches as not fully resolving the conflict between fine-grained detail and high-level semantics.Source: §2 · p. 3
3. Method
The method separates video encoding, masked-region prediction, and reconstruction or target mapping into an Encoder–Predictor–Decoder framework. It uses semantic-guided pixel reconstruction to establish a coherent latent space, then performs latent prediction within that space.
3.1. The EPD Disentanglement
- The EPD formulation separates a video encoder, a lightweight predictor for masked regions, and a decoder mapping predicted latents to pixels or target latents.This disentanglement makes the predictor’s output space explicit.Source: §3.1 · p. 3
3.2. S1: Semantic-Guided Pixel Reconstruction
- Stage 1 uses frozen SigLIP semantic guidance alongside pixel reconstruction, with a diffusion decoder and text-decoder augmentation supporting their combination.The semantic-alignment objective matches the masked-video student embedding to the corresponding visible-region embedding from the full-video teacher; the table describes an accuracy gain from combining diffusion and text-decoder augmentation.Source: §3.2 · p. 3, Table 1 · p. 4
3.3. S2: Semantically Coherent Latent Prediction
- Stage 2 builds on Stage 1 latents for latent prediction, retaining the Stage 1 predictor and using multi-block masking to strengthen temporal reasoning and reduce information leakage.The stage targets implicit world knowledge through prediction from large contiguous spatiotemporal blocks.Source: §3.3 · p. 4
4. Experiments
The experiments examine the two-stage framework through ablations and frozen-backbone probes spanning action, depth, tracking, prediction, retrieval, and chat-centric tasks. The reported results indicate strong performance across these settings, while the ablations identify design choices that support the representation.
4.1. Ablation Study
- Stage 1’s diffusion decoder turns the reconstruction-plus-alignment degradation into a +4.4% K400 gain, while pixel reconstruction alone reaches 47.2% on K400.The reported gain is attributed to complementarity between the supervision signals when the original linear decoder does not interfere.Source: §4.1 · p. 5
- Diffusion-based MLP decoders improve as capacity grows; the Depth 6, Width 1536 variant offers the best reported trade-off between representation richness and computational cost.The linear head underfits complex spatial relations.Source: §4.1 · p. 5
- Stage 2 performs strongest with the full configuration; changing predictor or target initialization, unfreezing the target encoder, or adding unmasked alignment degrades results.Pixel reconstruction in Stage 2 provides only marginal SSv2 benefit because Stage 1 targets already retain pixel-level detail.Source: §4.1 · p. 5, Table 3 · p. 5
- Semantic masking in Stage 1 and multi-block masking in Stage 2 support the respective goals of meaningful spatial focus and harder spatiotemporal prediction; more input frames consistently improve performance.The selected frame settings are F16 for Stage 1 and F32 for Stage 2, balancing accuracy and efficiency.Source: §4.1 · pp. 5–6, Table 4 · p. 5
4.2. Single-modality Tasks
- Frozen-backbone probing achieves the best reported video-classification results, while depth probing is reported as far better than image models and nearly state of the art against Video Depth Anything.The classification results cover K400, SSv2, and COIN; depth evaluation uses ScanNet and KITTI.Source: §4.2 · p. 6, Table 5 · p. 6, Table 6 · p. 7
- The model achieves the best reported object-tracking result and better future-action prediction on EK100.Tracking probes object-level motion on Waymo Open; action prediction reports mean-class recall-at-5 on EK100.Source: §4.2 · pp. 6–7, Table 7 · p. 7, Table 8 · p. 8
4.3. Multi-modality Tasks
- In zero-shot multimodal evaluation, the model outperforms InternVideo2s2 on action recognition, especially SSv2-Multiple Choice, and shows comparable text-to-video retrieval performance.The retrieval evaluation covers MSRVTT, DiDeMo, ActivityNet, LSMDC, and MSVD.Source: §4.3 · p. 8, Table 9 · p. 8, Table 10 · p. 8
- With a frozen encoder and language model connected through a trainable MLP, the model reports better results on MVBench, Perception Test, and Dream1K.These tasks cover video QA, general perception, and detailed captioning, respectively.Source: §4.3 · p. 8, Table 11 · p. 8
5. Conclusion and Future Work
InternVideo-Next unifies pixel reconstruction and latent prediction under a coherent semantic space, and the authors report that it surpasses prior video-only and video-text methods on general video tasks. Future work will investigate scalability and extend the framework toward multimodal and interactive world modeling.
5. Conclusion and Future Work
- InternVideo-Next unifies pixel-level reconstruction and latent prediction under a coherent semantic space, learning structured spatiotemporal priors from raw videos.The authors describe it as a two-stage framework without video-text supervision.Source: §5 · p. 8
- The authors report that InternVideo-Next surpasses previous video-only and video-text methods on general video tasks.They present it as a strong general video foundation model.Source: §5 · p. 8
- Future work will investigate scalability and extend the framework toward multimodal and interactive world modeling, efficient scaling, and open-world generalization.Source: §5 · p. 8
A. More implementation details
The experiments section introduces the model architectures, training hyperparameters, and test settings.
A. More implementation details
- The experiments section introduces model architectures, training hyperparameters, and test settings.Source: A. More implementation details · p. 9
A.1. Model architecture and training details
The models use patch size 14, with detailed architecture and pretraining settings reported in separate tables.
A.1. Model architecture and training details
- InternVideo-Next models use a patch size of 14.Source: §A.1 · p. 9
- The authors use patches of size 14 for InternVideo-Next models, report a Base-scaled model’s detailed architecture in Table 12, and provide pre-training details in Tables 13 and 14.Source: §A.1 · p. 9
A.2. Action Recognition
For action-recognition probing, the paper reports hyperparameters and uses attention pooling to form a classification token.
A.2. Action Recognition
- Action-recognition probing uses an attention pooling head that averages input tokens into a single token for classification as the common [CLS] token.The setup is shown in Figure 4(a).Source: §A.2 · p. 9
- The detailed hyperparameters for action-recognition probing are reported in Table 15.Source: §A.2 · p. 9
A.3. Depth Estimation
Depth estimation evaluates InternVideo-Next’s spatiotemporal and 3D-geometry capabilities on ScanNet and KITTI, with the backbone frozen. The evaluation compares probing heads, representation baselines, and standard geometric-accuracy metrics.
A.3. Depth Estimation
- The evaluation tests monocular video depth estimation on indoor ScanNet and outdoor KITTI while keeping the backbone frozen.ScanNet contains over 2.5 million views across more than 1,500 scans.Source: §A.3 · pp. 9–10
- The diffusion-loss procedure conditions noise prediction on the Predictor output and backpropagates its gradient to that output.The target tokens come from patching the video input with pixel shuffle and patch size 14.Source: §A.3 · p. 10
- Two prediction heads are used: a lightweight cross-attention pooling head and a temporally aware head built from four temporal modules.The temporally aware head is based on VideoDepthAnything and is also applied to video models.Source: §A.3 · p. 10
- Baselines include SigLip2 and DINOv3 for image representations, and VideoMAEv2, InternVideo2, and V-JEPA for video representations.All baselines are re-implemented under a unified training and evaluation protocol.Source: §A.3 · p. 10
- The evaluation reports AbsRel, average relative deviation from ground truth, and δ1, the proportion of predictions within a factor of 1.25.The two metrics jointly assess geometric accuracy.Source: §A.3 · pp. 10–11
A.4. Object Tracking
The object-tracking evaluation uses Waymo Open RGB videos and first-frame object boxes to assess prediction of later-frame boxes. It freezes the backbone and trains a probing head on reconstructed train and test sets.
A.4. Object Tracking
- On Waymo Open, models use RGB frames and first-frame 2D boxes to predict the target objects’ boxes in subsequent frames.The benchmark uses 2D and 3D box annotations, but only 2D boxes are used for losses and evaluation.Source: §A.4 · p. 11
- Because Scaling4D’s benchmark construction code is unavailable, the training and test sets are reconstructed from its paper description.Videos are downsampled from 10 fps to 5 fps after spatial downsampling and cropping.Source: §A.4 · p. 11
- Training windows contain 16 frames, with constraints on object count, visibility across frames, co-occurrence, and maximum missing gaps.Each first-frame object must appear in at least 70% of frames, and may have at most a continuous 5-frame gap.Source: §A.4 · p. 11
- The backbone is frozen while a probing head encodes initial boxes as queries and decodes pooled tokens into boxes for later frames.Training combines Smooth L1 loss weighted 1.0 and GIoU loss weighted 0.5; evaluation averages IoU across objects and frames.Source: §A.4 · p. 11