Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators

1 INTRODUCTION

• Visual demonstrations provide intricate guidance for learning actions and generalizing them across objects and contexts.The introduction frames visual imitation as a desirable capability for general AI agents interacting with the real world.Source: §1 · p. 1
• Prior image in-context methods rely on supervised source-target demonstrations, while image-level generation struggles to produce coherent contextual sequences.The paper proposes video imitation as a way for models to interact with real-world demonstrations through video generation.Source: §1 · pp. 1–2
• VidIT uses videos as both inputs and outputs, training an autoregressive Transformer with next-token prediction on discrete frame tokens from single-video sequences.Training uses self-supervised prediction of future frames from preceding frames, without demonstration-generation pairs.Source: §1 · pp. 1–2
• At inference, demonstrations guide generation of a subsequent video in a new scene that imitates their semantic actions.The proposed interface covers demonstrations such as object manipulation and camera movement in egocentric video.Source: §1 · p. 2
• Experiments report high-quality clips that follow demonstration semantics, and zero-shot imitation capacity that follows large-model scaling behavior.The evaluation measures visual quality, semantic accuracy, and consistency with prompted demonstrations using objective and subjective metrics.Source: §1 · pp. 2–3

• Imitation learning maps states to expert actions, while LLM in-context learning follows text templates without parameter updates; prompting designs support varied language tasks (Zare et al., 2024; Brown et al., 2020; Touvron et al., 2023; Wei et al., 2022; Guo et al., 2023; Li et al., 2023).Source: §2 · p. 3
• Vision in-context work uses image inpainting [Bar et al., 2022; Wang et al., 2023a;b] or autoregressive LVM [Bai et al., 2023], but relies on supervised training and lacks demonstrated video imitation.Existing models also struggle with video spatial-temporal complexity; LVM generates consecutive query frames, but differing outcomes from differing demonstrations were not demonstrated.Source: §2 · p. 3
• Conditional video generation has used text and, in embodied AI, actions and observations for planning and simulation; this paper chooses an autoregressive Transformer for long-range dependencies and contextual relationships.The cited approaches include text-to-video models (Ho et al., 2022b;a; Brooks et al., 2024; Hong et al., 2023; Villegas et al., 2023; Kondratyuk et al., 2023; Yu et al., 2023b), embodied AI (Yang et al., 2024), visual planning (Du et al., 2023), and simulation (Yang et al., 2023).Source: §2 · p. 3

3.1 PROBLEM DEFINITION

• VidIT aims to generate a video clip that is perceptually coherent with a query and semantically consistent with video demonstrations.Unlike image task imitators, it focuses on inheriting semantics from demonstrations rather than discerning a specific task.Source: §3.1 · p. 4

3.2 APPROACH

• VidIT tokenizes video frames and autoregressively predicts future tokens; at inference, demonstrations precede a query prefix whose continuation is decoded into frames.Training samples come from single videos, while inference conditions on selected demonstration clips.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
• Despite receiving no demonstrations during training, VidIT can imitate because training sequences implicitly resemble demonstration-query inputs and autoregressive prediction extends across videos.The experiments further report that explicit imitation examples do not significantly improve on the zero-shot approach.Source: §3.2 · p. 5
• The approach can also condition on text by adding pretrained language-model representations during training and inference; experiments report responses to both text and video demonstrations.Source: §3.2 · p. 5

3.3 DATA

• Training draws on Ego4D, Kinetics-600, and self-collected real-world videos; evaluation uses SSv2 and also includes RT-1 and MineRL for embodied and interactive tasks.Source: §3.3 · pp. 5–6

4.1 EVALUATION

• Visual quality is assessed with frame-comparison and distribution metrics, while semantic accuracy uses Video Accuracy and latent-representation Probing Accuracy.Most automatic metrics require ground-truth video; only Probing Accuracy applies in the setting without it.Source: §4.1 · pp. 6–7
• Evaluation uses no, random, in-class, and contrastive demonstrations, then assesses generations using query labels or demonstration labels to test semantic guidance.Source: §4.1 · p. 6
• Human evaluation uses 10 experienced users to score 20 contrastive sample pairs for visual quality, semantic alignment, and control on a 1–5 scale.Source: §4.1 · p. 7

4.2 IMPLEMENTATION DETAILS

• Implementation uses 16-frame training sequences and models with 300M, 700M, or 1.1B parameters; main inference uses one 8-frame demonstration with a 4-frame query and generation.Comparisons include LVM, DeLVM, LWM, and pretraining or fine-tuning variants.Source: §4.2 · p. 7

5.1 MAIN RESULTS

• In-class demonstrations improve P-Acc by 7.1%/8.4% and V-Acc by 1.8%/2.0% over no/random demonstrations, while random demonstrations trail no demonstration by 0.7%/0.2%.The first pair of gains is respectively over no demonstration and random demonstrations; the random-demonstration gaps are respectively in P-Acc and V-Acc.Source: §5.1 · p. 8, Table 1 · p. 8
• In-domain fine-tuning raises in-class P-Acc to 48.8%, an 8.1% improvement over imitation fine-tuning; explicit target-format fine-tuning yields no significant enhancement.The authors also report that both in-class and contrastive demonstrations produce above-average probing accuracy when evaluated using demonstration labels.Source: §5.1 · p. 8, Table 3 · p. 8

5.2 COMPARISON WITH BASELINE MODELS

• VidIT significantly outperforms baseline models in both objective and subjective evaluations.The authors attribute the gap to differences in training formats and whether models capture demonstration semantics across consecutive frames.Source: Table 2 · p. 8, §5.2 · p. 9

5.3 SCALING BEHAVIOR

• Larger models tend to produce higher-quality samples and more informative representations, with more precise control under different demonstrations.Visual quality is assessed with PSNR and FID; semantic scaling is shown through probing accuracy and gains over random demonstrations.Source: §5.3 · p. 9, Figure 3 · p. 9

5.4 GENERALIZATION ABILITY

• On RT-1, VidIT mimics robotic-arm actions, and as an interactive-agent video engine it generates frames converted to action signals through inverse dynamics.The paper also reports that video-segmentation exemplars let VidIT learn the task format from demonstrations.Source: §5.4 · pp. 9–10, Figure 4 · p. 10

5.5 ABLATION AND ANALYSIS

• Within a 16-frame context, providing demonstrations generally improves semantic accuracy, and more or longer demonstrations improve performance; adding text descriptions also enhances imitation capacity.The text-demonstration experiment fine-tunes the 300M Pretrain model on SSv2 with annotations encoded by pretrained T5-large.Source: §5.5 · p. 10, Table 4 · p. 10, Table 5 · p. 10

6 CONCLUSION

• VidIT uses discrete video tokens and next-token prediction to imitate demonstrations zero-shot, with experiments supporting semantic coherence and potential use in visual planning.The conclusion also reports that the model can integrate multimodal demonstrations.Source: §6 · p. 10

A ADDITIONAL RESULTS

• Additional samples cover Something-Something v2 and Robotics Transformer-1, and the paper describes VidIT as a simulator for reinforcement-learning tasks.The samples are presented in Tables 10 and 11, respectively.Source: A ADDITIONAL RESULTS · p. 15

A.1 VIDIT AS SIMULATOR

• On VP2, VidIT generates future frames for the demonstrated task, with more precise control than SVG on Push-red; MineRL tests show generated scenes producing action signals for chopping wood and attacking sheep.The MineRL results are illustrated for the two actions in Figure 6.Source: §A.1 · p. 15, Figure 5 · p. 15, Figure 6 · p. 15

A.2 VIDIT AS TASK IMITATOR

• After a few fine-tuning steps on VIPSeg, VidIT learns the segmentation task format from demonstrations and generates segmentation outputs for query videos.The example is shown in Figure 7.Source: §A.2 · p. 16, Figure 7 · p. 16

A.3 CASE STUDY: BASELINE METHODS

• Under the same conditioning, VidIT generates more semantically accurate, higher-quality videos than baseline models despite a lower parameter budget.Figure 8 presents generated cases for VidIT and the baselines.Source: Figure 8 · p. 16, §A.3 · p. 16

A.4 FAILURE CASES

• When demonstration and query videos depict unrelated actions, VidIT may ignore the demonstrated action and generate content incoherent with it.The paper also identifies unrelated or ambiguous demonstrations as a source of difficulty; Figure 9 shows failure cases.Source: §A.4 · p. 17, Figure 9 · p. 17

A.5 INTERPRETABILITY OF GENERALIZATION

• Across training and inference, attention concentrates around the moving object in demonstration frames, a pattern the authors interpret as leveraging prior-frame semantics for imitation from a different scene.The attention visualization compares a training clip with inference beginning from a different query frame.Source: Figure 10 · p. 17, §A.5 · p. 18
• Successful generations tend to allocate more attention to demonstration frames, whereas failures focus more on query frames and effectively ignore demonstration guidance.Figure 11 visualizes the attention distributions for successful and failed cases.Source: §A.5 · p. 18, Figure 11 · p. 18

A.6 FULL EVALUATION METRICS ON BASELINES

• VidIT consistently outperforms baseline models in video imitation tasks across semantic alignment and visual quality metrics.Table 6 contains the full automatic metrics; the supplied passage does not state individual metric values.Source: Table 6 · p. 18, §A.6 · p. 18

B.1 MODEL ARCHITECTURE

• The LLaMA-based VidIT models are trained at 300M, 700M, and 1.1B parameter scales, with hidden dimension, MLP dimension, and decoder-layer counts tuned across sizes.Table 7 gives the model configurations.Source: §B.1 · p. 19, Table 7 · p. 19

B.2 HYPERPARAMETERS

• Training uses an inverse square root learning-rate scheduler and 10,000 warmup steps.The hyperparameters are presented in Table 8.Source: §B.2 · p. 19, Table 8 · p. 19

B.3 DATASET STATISTICS

• Dataset statistics include video counts and token totals after tokenization and for VidIT training.The statistics are presented in Table 9.Source: §B.3 · p. 19

B.4 TRAINING DETAILS

• Training VidIT 1.1B took 720 GPU hours over 15 epochs on 2x8 H100 nodes.Source: §B.4 · p. 19
• Each training epoch took about 3 hours.Source: §B.4 · p. 19
• Table 9 reports statistics for the training datasets.Source: Table 9 · p. 19

C LIMITATIONS

• Performance on longer video sequences has not been thoroughly evaluated.The authors propose temporal-compression tokenizers as future work.Source: C LIMITATIONS · p. 20
• Noisy or semantically misaligned demonstrations can lead to inconsistencies in generated videos.The authors identify scaling or developing robust models for imperfect demonstrations as future work.Source: C LIMITATIONS · p. 20

D ETHICS STATEMENT

• The authors identify privacy concerns with widespread video analysis and generation, while noting possible biases in Internet-sourced datasets.The datasets are publicly available and reviewed for personally identifiable or offensive content; filtering was used to minimize inappropriate generation.Source: D ETHICS STATEMENT · p. 20