Source-linked AI summary
LongVU-TTT: Causal Test-Time Training for Visual Resampling in Long Video Understanding
Mahmoud Ahmed, Sameh Abdulah, Olatunji Ruwase, Sam Ade Jacobs, Mathis Bode, Mohamed Elhoseiny
TL;DR
Long-video MLLMs need to model temporal change before visual-token budgets discard frame evidence. LongVU-TTT inserts a causal convolutional TTT resampler with grouped 2D fast weights and hybrid frame selection before the LLM. Controlled studies find effective temporal aggregation, while explicit frame retention remains necessary for distant evidence.
Problem
Long-video MLLMs must model temporal change while linearly growing visual tokens make quadratic attention prohibitively expensive and compression can precede explicit change modeling.
Method
LongVU-TTT inserts a convolutional TTT resampler between the vision encoder and LLM, using grouped 2D fast weights and hybrid uniform-and-change-aware frame selection.
Results
Controlled operator and retention studies show that causal 2D fast weights are effective temporal aggregators, while LongVU-TTT compresses up to 512 frames to 128 LLM frames.
Takeaways & Limitations
Fast weights contextualize video features but do not replace explicit evidence for long-horizon reasoning.
Takeaways & Limitations
Fast-state benefits attenuate beyond half of the video, whereas explicit evidence helps at every tested gap.
Abstract
from arXiv · showhide
Long-video MLLMs must model temporal change before a limited visual-token budget removes most frame evidence. We introduce LongVU-TTT, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the vision encoder and the LLM. Its grouped 2D fast weights adapt to each video and contextualize frame features before compression, while a hybrid uniform-and-change-aware selector retains explicit visual evidence for downstream reasoning. Under controlled conditions, TTT-Conv improves over TTT-MLP by up to +2.12 and bidirectional Mamba2 by up to +3.04 on MLVU, and it is stronger than attention- and fixed-state recurrent resamplers across three benchmarks. Analysis shows that the fast weights behave as a temporal aggregation state rather than a reliable long-horizon episodic memory: their benefit attenuates as evidence becomes more distant, motivating explicit frame retention. LongVU-TTT processes up to 512 frames before reducing them to 128 LLM frames and achieves competitive performance across five video understanding benchmarks.
1 INTRODUCTION
LongVU-TTT addresses long-video temporal modeling and visual-token bottlenecks by inserting causal convolutional TTT before compression. It combines grouped 2D fast weights with change-aware frame selection and evaluates these choices through controlled comparisons.
- Long-video MLLMs face linearly growing token counts and prohibitively expensive quadratic attention when frames are modeled independently.
- Existing compression methods reduce frame features before explicitly modeling temporal change, while recurrent alternatives lack explicit 2D local-neighborhood interactions.
- LongVU-TTT inserts a convolutional TTT resampler that adapts grouped 2D fast weights to contextualize video features before the LLM bottleneck.
- The paper emphasizes controlled comparisons among similarly scaled models sharing the same trained baseline rather than claiming broad state-of-the-art performance.
- The hybrid selector combines gradient-direction shift and alignment-loss delta to retain scene transitions and within-scene change without an auxiliary learned scorer.
- The controlled study compares temporal Transformers, gated recurrence, Mamba2, and TTT variants, while retention diagnostics distinguish temporal aggregation from episodic storage.
2 RELATED WORK
Related work reduces long-video cost through compression and recurrent temporal modeling, but prior approaches often flatten visual structure or rely on fixed fast-weight formulations. LongVU-TTT instead preserves 2D spatial structure while placing temporal processing at the projector stage.
- Video MLLMs typically encode frames independently and delegate temporal modeling to LLM self-attention, whose cost becomes prohibitive for long sequences.
- Compression methods include uniform pooling, token merging, differential token dropping, and structured redundancy removal with visual or text-guided signals.
- Recurrent video models reduce attention cost through linear-complexity sequence operators, but flattened scans do not explicitly encode 2D local-neighborhood interactions.
- Test-Time Training uses continuously evolving fast weights instead of fixed recurrent hidden states, while recent variants remain focused on 1D MLP-based sequence modeling.
- Its temporal operator is inserted between the ViT encoder and LLM, while the existing MLP projector performs modality-alignment mapping after pooling and frame selection.
- LongVU-TTT processes frames recurrently with grouped convolutions that preserve 2D spatial structure and complement ViT global representations.
3 METHODOLOGY
LongVU-TTT adds a causal convolutional resampler between the vision encoder and LLM, adapting 2D fast weights to contextualize frames before compression. It combines parallelized causal updates with change-aware and uniform frame selection to process long videos within a fixed visual-token budget.
- Architecture and fast weights: LongVU-TTT inserts a TTT-Conv resampler between the vision encoder and LLM, preserving frame feature grids while adapting grouped convolutional fast weights to temporal change.The fast weights are carried across chunks, reset at video boundaries, and adapted using a self-supervised alignment objective without answer labels.
- Causal materialization: Parallel Causal Materialization preserves causal ordering while materializing per-frame fast-weight states simultaneously within each chunk.Prefix sums and torch.vmap enable parallel computation, while asynchronous activation checkpointing rematerializes states during backpropagation to trade activation memory for compute.
- Architecture and fast weights: TTT-Conv applies each frame through its corresponding preceding-state convolutional weights, producing a temporally contextualized representation before the LLM.The mapping uses two grouped Conv2d kernels with a SiLU activation, and the current frame’s own update is excluded from its causal state.
- Importance scoring and selection: LongVU-TTT selects top-ranked salient frames first, then fills the remaining budget uniformly, reducing up to 512 frames to 128 LLM frames without an auxiliary learned scorer.The fixed-budget selection occurs after scores for the full video are available, while causality applies to fast-weight updates.
- Importance scoring and selection: The selector combines Gradient-Direction Shift and Alignment-Loss Delta into a normalized importance score that identifies scene-level and within-scene visual changes.The two signals are weighted with α = 0.4 after min-max normalization, with ϵ preventing division by zero for constant signals.
4 EXPERIMENTS
Experiments evaluate LongVU-TTT across five benchmarks, controlled temporal-operator and selection studies, evidence withholding, and system-efficiency tests. Results support causal 2D fast-weight contextualization, hybrid frame selection, explicit evidence retention, and scalable long-video training.
- Main results: LongVU-TTT improves over the same-pipeline LLaVA-Video reference on all five benchmarks, with the largest gains on LongVideoBench and LVBench.
- Temporal operators: TTT-Conv is strongest across all three controlled benchmarks when temporal modules share the same projector location and training conditions.
- Temporal operators: Cross-chunk state carry improves all benchmarks, while 1 × 1 and shuffled-grid controls reduce the advantage, isolating benefits beyond convolution alone.
- Temporal context and selection: 512-frame temporal processing followed by 128-frame hybrid selection improves over the frame-independent 512-to-128 baseline, while direct 128-frame processing is not uniformly beneficial.
- Temporal context and selection: Both TTT-derived selection signals improve over uniform sampling, and their weighted combination is strongest; importance-ranked frames are supplemented with uniform sampling.
- Evidence retention: Fast-state recovery gains are largest at short and medium temporal gaps, attenuate beyond half the video, and remain weaker than retaining explicit evidence at every gap.
- System efficiency: Asynchronous CPU activation offloading raises the maximum trainable frame count from 64 to 256 on eight GPUs, while batch-tiled ViT processing reaches 2.23× speedup at 64 GPUs and 87% scaling efficiency.
5 CONCLUSION
LongVU-TTT addresses long-video spatiotemporal redundancy by contextualizing frame features with causal convolutional TTT before selecting visual evidence for the LLM. Controlled studies characterize its fast weights as temporal aggregators rather than replacements for long-horizon episodic storage.
- LongVU-TTT inserts a convolutional TTT layer between the ViT encoder and LLM to address spatiotemporal redundancy before visual-token compression.
- Grouped-convolution fast weights preserve 2D spatial structure while causal updates propagate temporal information before LLM processing.
- Inner-loop gradient signals provide an importance measure that, combined with uniform sampling, compresses 512 frames to 128 while retaining explicit evidence.
- Retention studies show that causal 2D fast weights aggregate temporal information but do not replace long-horizon episodic storage.
- Batch-tiled ViT processing and asynchronous CPU activation offloading extend trainable sequence length by 4× and reduce training step time by up to 2.23× at 64 GPUs.
A.1 TRAINING DATA
Training uses two stages and progressively exposes LongVU-TTT to longer and more complex video content. Stage 1 emphasizes short-video projector learning, while Stage 2 jointly trains on mixed image-video data with up to 512 frames.
- Training proceeds in two stages with different data mixtures designed to expose the model progressively to longer and more complex video content.
- Stage 1: Stage 1 trains the TTT layer and MLP projector on LLaVA-CC3M-Pretrain-595K and 200K short-video instruction samples, capping videos at 16 frames.
- Stage 1: The short-video constraint gives fast weights multiple distinct gradient updates per sample and keeps projector-pretraining memory manageable.
- Stage 2: Stage 2 jointly trains all model components on 1.5M LLaVA-Video and LLaVA-OneVision samples, with videos sampled at 1 fps up to 512 frames.
- Stage 2: Image samples remain part of the mixed image-video Stage 2 training distribution.
A.2 TRAINING SETUP
The training setup uses A100 40GB GPUs and scales from 16 GPUs in Stage 1 to 64 GPUs in Stage 2, with component-specific learning rates in the second stage.
- All experiments use A100 40GB GPUs with NVLink intra-node and InfiniBand inter-node connections.
- Stage 1: Stage 1 runs on 16 GPUs for one epoch with global batch size 512 and a 5 × 10−4 learning rate for the TTT layer and MLP projector.
- Stage 2: Stage 2 scales to 64 GPUs for one epoch with global batch size 128 and component-specific learning rates.
A.3 SYSTEM IMPLEMENTATION DETAILS
The implementation uses memory-saving strategies to support long-video processing without changing the temporal update rule or attention computation.
- Memory optimization: Checkpointed TTT materialization rematerializes per-frame fast-weight states during backward instead of retaining them all.This trades activation memory for parallel compute.
- Memory optimization: LLM activations are offloaded to host memory and prefetched in reverse layer order during backward computation.Transfers overlap with computation on a dedicated CUDA stream.
A.4.1 TTT-CONV LAYER
This section defines the controlled temporal-resampler comparisons, input-length behavior, inference scaling, and hybrid importance signals used to analyze LongVU-TTT.
- TTT-Conv layer: Algorithm 1 computes projections and gradients, applies per-group learning rates and momentum, materializes causal per-frame states, normalizes weights, and applies them to queries.Weight and momentum states carry across chunks to preserve temporal continuity.
- Controlled variants: All variants occupy the same pipeline position and use identical training conditions, including Csize = 8, enabling controlled comparisons.The variants are inserted between the ViT encoder and spatial pooling layer.
- Controlled variants: TTT-MLP performs one joint update per chunk, whereas causal TTT-MLP materializes a distinct prefix-update state for each frame.Non-causal TTT-Conv likewise aggregates one update per chunk and shares it across that chunk’s queries.
- Context length: Increasing pre-aggregation context beyond 128 frames produces only small, non-monotonic MLVU changes at a fixed 128-frame LLM budget.This motivates distinguishing temporal contextualization from persistent episodic storage.
- Inference speed: TTT-Conv is slower than TTT-MLP and Gated DeltaNet but remains substantially faster than the temporal Transformer at every tested context length.Recurrent fast-weight operators scale approximately linearly, while the temporal Transformer scales quadratically with frame count.
- Frame importance: The gradient-direction shift detects abrupt scene transitions, while the alignment-loss delta captures sustained within-scene changes.Their combination selects both sudden transitions and gradual meaningful visual changes.
- Frame importance: In a long lecture video, alignment-loss peaks identify later poster content, gestures, and visual elements that gradient-direction shifts largely miss.The combined score retains both early transitional frames and later content-rich moments.