Source-linked AI summary
TV2TV: A Unified Framework for Interleaved Language and Video Generation
Xiaochuang Han, Youssef Emad, Melissa Hall, John Nguyen, Karthik Padthe, Liam Robbins, Amir Bar, Delong Chen, Michal Drozdzal, Maha Elbayad, Yushi Hu, Shang-Wen Li, Sreya Dutta Roy, Jakob Verbeek, XuDong Wang, Marjan Ghazvininejad, Luke Zettlemoyer, Emily Dinan
TL;DR
Video generation models can struggle with semantic branching and repeated high-level reasoning. TV2TV interleaves text and video generation, with reported improvements across game and sports evaluations; future work includes improving the training text’s granularity and accuracy across more domains.
Problem
Video generation models can struggle with complex outputs requiring semantic branching or repeated high-level reasoning about what should happen next.
Source: §1 · p. 1Method
TV2TV decomposes video generation into interleaved text and video generation, jointly optimizing text cross-entropy and video flow losses.
Source: §1 · p. 3, §2.2 · p. 6Results
Across settings, TV2TV improved game visual quality and controllability and sports visual quality and prompt alignment; game videos were preferred 91% of the time over a comparable T2V baseline.
Source: §1 · pp. 2–3, §6 · p. 20Takeaways & Limitations
Interleaving text and video offloads much semantic complexity to text generation and enables more flexible and effective user control during generation.
Source: §6 · p. 20Future work could improve the granularity and accuracy of interleaved training text across more video domains.
Source: §4.3 · p. 17
Abstract
from arXiv · showhide
Video generation models are rapidly advancing, but can still struggle with complex video outputs that require significant semantic branching or repeated high-level reasoning about what should happen next. In this paper, we introduce a new class of omni video-text models that integrate ideas from recent LM reasoning advances to address this challenge. More specifically, we present TV2TV, a unified generative modeling framework which decomposes video generation into an interleaved text and video generation process. TV2TV jointly learns language modeling (next-token prediction) and video flow matching (next-frame prediction) using a Mixture-of-Transformers (MoT) architecture. At inference time, TV2TV decides when to alternate between generating text and video frames, allowing the model to "think in words" about subsequent content before ``acting in pixels'' to produce frames. This design offloads much of the responsibility for deciding what should happen next to the language modeling tower, enabling improved visual quality and prompt alignment of generated videos. It also enables fine-grained controllability, allowing users to modify the video generation trajectory through text interventions at any point in the process. In controlled experiments on video game data, TV2TV demonstrates substantial improvements in both visual quality and controllability. TV2TV also scales to natural videos, as we show by augmenting sports videos with interleaved natural language action descriptions using vision-language models (VLMs). Training TV2TV on this corpus yields strong visual quality and prompt alignment, showcasing the model's ability to reason about and generate complex real-world action sequences. Together, these results highlight TV2TV as a promising step toward video generation with open-ended textual reasoning and control.
1 Introduction
Video models can struggle with outputs requiring semantic branching or repeated high-level reasoning. TV2TV interleaves text planning with video generation, allowing users to steer the trajectory through textual prompts.
1 Introduction
- TV2TV jointly learns language modeling and video flow matching, decomposing generation into interleaved text plans and chunks of video frames.At inference, it alternates between generating text and video, letting the model plan subsequent frame content in words.Source: §1 · pp. 1, 3
- The text component takes on much of the semantic decision-making, while users can inspect, edit, or steer the textual plan at any timestep.This provides a way to modify the video-generation trajectory through text.Source: §1 · p. 2
- TV2TV uses dedicated text and video transformer towers with global self-attention over the multimodal sequence.The text tower is initialized from a pretrained language model.Source: §1 · p. 2
- In holistic sports-video preference evaluations, TV2TV is favored 54.0% of the time over T2V (34.7% unfavorable) and 53.3% over Think2V (41.3% unfavorable).The model is trained on 8K hours of sports videos augmented with interleaved natural-language action descriptions using VLMs.Source: §1 · p. 3
- TV2TV videos are preferred over a T2V baseline 91% of the time, with a 19 point improvement in fine-grained instruction-following accuracy over Think2V.These controlled results use CS:GO gameplay data.Source: §1 · p. 3
2 TV2TV
TV2TV decomposes video generation into an interleaved sequence of text planning and video-frame generation, jointly learning language modeling and video flow matching. Its modality-specific towers, temporally ordered data, and inference-time switching support this factorization and allow users to steer generation with text.
2 TV2TV
- TV2TV factorizes video generation into interleaved text planning and video segments, jointly learning next-token prediction and video flow matching under temporal causality.Text and video use dedicated towers, while inference flexibly alternates between the two modalities.Source: §2 · p. 3
2.1 Data representation
- Video is encoded as 0.25-second latent frame chunks, and text segments and frame chunks are ordered chronologically, with text first when timestamps match.Beginning-of-frame and end-of-frame tokens delimit video chunks and enable transitions between text and video generation.Source: §2.1 · p. 4
- Each video chunk appears as a noisy representation followed by a clean one, so training can denoise current frames while conditioning on clean historical context.The paper says this avoids the limitation of a single-representation monotonic-noise approach for interaction with interleaved text.Source: §2.1 · p. 5
2.2 Architecture and optimization
- TV2TV uses modality-specific transformer towers and attention masks, then projects hidden states to text logits or predicted video flow and jointly optimizes text cross-entropy and video-flow losses.Its hybrid masking uses causal attention for text and block-causal attention for noisy and clean video tokens; noisy video tokens cannot be attended by future tokens.Source: §2.2 · pp. 5–6
2.3 Inference
- Inference defaults to autoregressive text generation; a beginning-of-frame token triggers video-chunk generation from noise through ODE-solver steps before text generation resumes.Generation terminates at an end-of-sequence token or when the context length is exceeded, and sliding windows support sequences beyond the trained context length.Source: §2.3 · p. 7
2.4 Task formulation
- Interleaved text planning is used as semantic guidance for video generation, allowing users to intervene through text and modify the generation trajectory at any point.For video games, controller actions can supply textual plans; for general-domain videos, the paper explores adding interleaved action descriptions with vision-language models.Source: §2.4 · p. 7
3 Controlled Experiments with Video Game Data
This section tests whether interleaved text planning improves gameplay-video quality and permits mid-generation user steering, using controller actions as plans aligned with gameplay.
3.1 Modeling details
- The controlled comparison uses a shared autoregressive video framework: T2V has no interleaved text, while Think2V generates its full text action plan before video; all models use a 3B-MoT backbone.Source: §3.1 · p. 8
3.2 Data details
- Models train on 95 hours of CS:GO footage with controller actions grouped across four frames per video latent; TV2TV places each action text before its associated frame block.Source: §3.2 · pp. 8–9
3.3 Evaluation set-up
- Human evaluations compare matched-frame short and long videos and test intermediate action interventions against Think2V at 1s and 3s, scoring intervention correctness and visual quality.Source: §3.3 · pp. 9–10
3.4 Results
- TV2TV wins 91% vs. 1% (8% ties) against T2V and 48% vs. 14% (38% ties) against Think2V, with larger gains on long videos.The pairwise visual-quality evaluations compare short and long videos.Source: Figure 4 · p. 10, §3.4 · p. 10
- TV2TV scores 78% vs. 59% for intervention correctness against Think2V, with stronger control for jump and left-click and similar results for reload and backward.TV2TV also produces higher visual quality during interventions; without intervention, models generate the action only 17–22% of the time.Source: §3.4 · p. 10, Figure 5 · p. 11
4 Scaling TV2TV to Real World Data
TV2TV extends interleaved text-and-video generation to real-world sports by augmenting videos with VLM-generated action descriptions. In sports evaluations, it improves alignment and holistic preference against external and controlled baselines, while its visual-quality comparisons vary by model.
4.1 Interleaved data augmentation pipeline
- Sports data augmentation converts real-world video into interleaved training data through sourcing, segmentation, filtering, and VLM captioning, yielding 8K hours with meta-captions and differential action descriptions.The source sports domain was selected for dynamic motion and rich action content.Source: §4 · p. 11, Figure 6 · p. 12, §4.1 · pp. 12–13
4.2.1 Modeling and data details
- The sports model uses an 8B-MoT backbone with modality-specific 8B-parameter towers, a Llama-3.1-8B text initialization, and 250K training steps.Training uses 320 × 192 video at 16 FPS and samples 6.1 seconds (98 frames) per step per device.Source: §4.2.1 · p. 13
4.2.2 Evaluation set-up
- Professional external annotators on Turing compare TV2TV videos with external video-generation models and controlled T2V/Think2V baselines, rating prompt alignment, visual quality, real-world fidelity, and overall preference; external-model comparisons comprise 300 pairwise tasks.Source: §4.2.2 · pp. 13–14
4.2.3 Results
- On sports prompts, TV2TV outperforms external models in prompt alignment, real-world fidelity, and holistic preference; visual quality surpasses Cosmos2, matches MAGI-1, and trails WAN 2.2 5B.The external models were not specifically tuned for sports, so the authors include them as out-of-domain baselines.Source: Figure 7 · p. 15, §4.2.3 · p. 15
- TV2TV’s holistic-preference win rates are 54% vs. 35% against T2V and 53% vs. 41% against Think2V, with the latter comparison not statistically significant.Alignment win rates are 20 points higher than T2V and 12 points higher than Think2V; the latter is also not statistically significant, while fidelity and visual quality are similar across controlled baselines.Source: Figure 8 · p. 15, §4.2.3 · pp. 15–16
4.2.4 Qualitative analysis
- Qualitative examples show model-generated interleaved text planning subsequent frames, while users can inspect, edit, or steer the intermediate plan to modify video generation at any timestep, including in sports-trained 8B-MoT examples.Source: §4.2.4 · p. 16
4.3 Discussion
- The sports gains over non-interleaved methods are less pronounced than the CS:GO gains, potentially reflecting sparser synthetic captions and VLM hallucinations; the authors identify caption granularity and accuracy as future-work targets.Sports captions are inserted every 1.9 seconds on average, whereas CS:GO uses frame-level ground-truth actions.Source: §4.3 · pp. 16–17
5 Related Work
Related work spans unified multimodal models, action-conditioned and autoregressive video generation, longer-video prompting, and dense captioning. TV2TV combines these threads through interleaved text-video generation and a sports-focused captioning pipeline.
Related work
- Unified multimodal models progress from Flamingo’s vision-language cross-attention to autoregressive and hybrid approaches, including modality-specific losses, decoupled visual pathways, and Mixture-of-Transformers.The cited examples include Flamingo (Alayrac et al., 2022), Emu2 (Sun et al., 2023), Chameleon (Chameleon Team, 2024), Transfusion (Zhou et al., 2024), Janus (Ma et al., 2025; Chen et al., 2025c), and BAGEL (Deng et al., 2025).Source: §5 · p. 17
- Action-conditioned game models often use discrete, task-specific actions, while other work explores physical action spaces or learned latent actions; TV2TV targets open-ended text-grounded actions across domains.Examples of other settings include navigation and full-body control (Bar et al., 2025; Lu et al.); Genie learns latent actions (Bruce et al., 2024).Source: §5 · pp. 17–18
- Autoregressive video models predict frames or frame blocks, but cited approaches generally do not generate interleaved text; Cosmos also does not condition on the entire history.VideoPoet instead treats video generation as next-token prediction and can use KV caching.Source: §5 · p. 18
- TV2TV can extend video through sliding windows, with continuation prompts generated by the model or inserted by users, complementing multi-prompt video-generation methods.The cited full-sequence diffusion approach treats the video as one tensor and faces prohibitively large attention computations for long-form and interactive generation.Source: §5 · pp. 18–19
- TV2TV’s sports-data pipeline follows dense-captioning work but emphasizes feature-based adaptive segmentation and motion-, face-, and VLM-based filtering tailored to interleaved training corpora.The passage relates its action-change descriptions to differential captioning work, including ShareGPT4Video (Chen et al., 2024b), ProgressCaptioner (Xue et al., 2025b), and CI-VID (Ju et al., 2025).Source: §5 · p. 19
6 Conclusion
The conclusion presents TV2TV as an interleaved text-video framework that shifts semantic planning to text generation and supports user control. Experiments on games and augmented real-world data show gains in generation quality and controllability over baselines.
6 Conclusion
- TV2TV decomposes video generation into an interleaved text and video generation process.The framework generates both modalities in an interleaved manner.Source: §6 · p. 20
- Interleaving offloads much of video generation’s semantic complexity to text generation and enables more flexible user control during generation.Source: §6 · p. 20
- Controlled video-game experiments show TV2TV outperforming baselines in generation quality and controllability, and the paper demonstrates scaling to real-world data using VLM-generated interleaved action descriptions.Source: §6 · p. 20
A.1 Model configuration details
The appendix points to a model-configuration table for video-game experiments and a separate table of key TV2TV training and inference hyperparameters.
A.1 Model configuration details
- Table 6 lists key training and inference hyperparameters for TV2TV.Source: §A.1 · p. 25
- Table 6’s model-configuration details also cover baselines for the video-game experiments.Source: Table 6 · p. 25
- All model variants in the video-game experiments use a 3B-MoT Transfusion architecture.Source: Table 6 · p. 25
A.2 Human Evaluation Task
The evaluation instructions define pairwise visual-quality judgments and pointwise checks of whether an intervention caption reflects the relevant video interval.
A.2 Human Evaluation Task
- Pairwise evaluation asks annotators which video has better visual quality, with examples of weak quality including flicker, wall-jumping, static movement, and random map jumps.Ghost-like or translucent characters are explicitly not penalized.Source: §A.2 · p. 25
- Pointwise alignment checks whether Caption A reflects the clip between START and STOP, while allowing specified immediate post-STOP actions and broader context.The instructions also offer correct, incorrect, and unsure responses, including an unsure option when the player is inactive.Source: §A.2 · p. 26
- Visual quality is also rated as strong, moderate, weak, or none, with examples of poor quality and an instruction not to penalize translucent characters.The question asks about the overall visual quality of the video.Source: §A.2 · p. 26
B Additional details: Scaling TV2TV to Real World Data
The real-world data appendix gives VLM prompts for filtering sports videos and producing concise, temporally differentiated captions.
B.1 VLM-based quality filtering prompt
- The quality-filtering prompt treats meaningful sports content as high quality and talking-head footage, jitter, or static imagery as low quality.It requests a score from 1 to 10 in JSON and presents frames sampled from the video's start, middle, and end.Source: §B.1 · p. 26
B.2 VLM-based differential captioning prompt
- The differential-captioning prompt asks for separate descriptions of each chronological segment, focusing on visible actions without anticipating future events.It says to avoid repetition, use an empty description when nothing meaningfully changes, keep descriptions under 50 words, and not hallucinate.Source: §B.2 · p. 27
- The differential-captioning output is formatted as one labeled description for each segment in the supplied video sequence.The requested labels run from segment 1 through segment N.Source: §B.2 · p. 27
B.3 Model configuration details
The section points readers to Table 7 for key training and inference hyperparameters.
B.3 Model configuration details
- Table 7 contains the key training and inference hyperparameters.The passage refers readers to the table but does not state the parameter values.Source: §B.3 · p. 27
B.4 Human Evaluation Task
The real-world human evaluation uses external annotators and asks them to compare generated videos across alignment, fidelity, quality, and holistic preference.
B.4 Human Evaluation Task
- Professional external annotators evaluate the video pairs through the Turing platform, and Figure 9 shows a similar evaluation interface.The figure is described as covering alignment, fidelity, quality, and overall preference.Source: §B.4 · p. 27, Figure 9 · p. 27
- The evaluation questions cover prompt alignment, real-world fidelity, visual quality, and holistic preference.The prompts ask which video better matches the language instruction, better reflects the real world, has better visual quality, and is preferred overall.Source: §B.4 · pp. 27–29
- The model-configuration table states that all real-world experiment variants use an 8B-MoT Transfusion architecture.The table describes TV2TV and baseline configurations for the sports-data experiments.Source: Table 7 · p. 28
- Each pairwise question offers choices ranging from a significant advantage for either video to a marginal advantage or an unsure/equal judgment.This response scale is shown for fidelity, visual quality, holistic preference, and prompt alignment.Source: §B.4 · p. 28