Source-linked AI summary
Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing
Baifeng Shi, Stephanie Fu, Long Lian, Hanrong Ye, David Eigen, Aaron Reite, Boyi Li, Jan Kautz, Song Han, David M. Chan, Pavlo Molchanov, Trevor Darrell, Hongxu Yin
TL;DR
Long, high-resolution video understanding is limited by models that process redundant pixels across frames, creating a scaling gap for MLLMs. AutoGaze autoregressively selects reconstructive multi-scale patches before ViT processing, reducing computation while preserving information. It cuts visual tokens by 4×–100×, accelerates ViTs and MLLMs by up to 19× and 10×, and improves performance on the new HLVid benchmark.
Problem
Video MLLMs process redundant pixels across frames, making long, high-resolution video understanding computationally difficult.
Method
AutoGaze autoregressively selects a minimal set of multi-scale patches that reconstructs video within a user-specified threshold before ViT processing.
Results
4×–100× fewer visual tokens and up to 19× ViT and 10× MLLM speedups accompany improved benchmark performance, including 52.6% on HLVid versus a 42.5% baseline.
Takeaways & Limitations
AutoGaze enables MLLMs to process 1K-frame, 4K-resolution videos while maintaining or improving downstream performance.
Takeaways & Limitations
AutoGaze does not account for most camera motion and cannot anticipate future frames using intuitive physics knowledge.
Abstract
from arXiv · showhide
Multi-modal large language models (MLLMs) have advanced general-purpose video understanding but struggle with long, high-resolution videos -- they process every pixel equally in their vision transformers (ViTs) or LLMs despite significant spatiotemporal redundancy. We introduce AutoGaze, a lightweight module that removes redundant patches before processed by a ViT or an MLLM. Trained with next-token prediction and reinforcement learning, AutoGaze autoregressively selects a minimal set of multi-scale patches that can reconstruct the video within a user-specified error threshold, eliminating redundancy while preserving information. Empirically, AutoGaze reduces visual tokens by 4x-100x and accelerates ViTs and MLLMs by up to 19x, enabling scaling MLLMs to 1K-frame 4K-resolution videos and achieving superior results on video benchmarks (e.g., 67.0% on VideoMME). Furthermore, we introduce HLVid: the first high-resolution, long-form video QA benchmark with 5-minute 4K-resolution videos, where an MLLM scaled with AutoGaze improves over the baseline by 10.1% and outperforms the previous best MLLM by 4.5%. Project page: https://autogaze.github.io/.
1. Introduction
AutoGaze addresses the computational burden of processing redundant video pixels by autoregressively selecting informative multi-scale patches before ViT processing. It substantially reduces visual tokens and accelerates models while supporting long, high-resolution video understanding and motivating HLVid as a stress test.
- Method: AutoGaze removes redundant static-region patches, focuses on moving objects, adapts to scene changes, and uses finer granularity for detailed regions.These behaviors support low gazing ratios while preserving reconstruction quality.
- Motivation: Modern video models process every pixel in every frame despite spatiotemporal redundancy, limiting their ability to scale to long, high-resolution videos.Static backgrounds often need to be viewed only once, yet current models process them repeatedly.
- Motivation: Existing MLLM efficiency methods prune tokens only in the LLM or rely on heuristics and costly search, leaving major ViT bottlenecks.These limitations prevent efficient scaling to longer and higher-resolution videos.
- Method: AutoGaze autoregressively selects a minimal set of multi-scale patches that reconstructs each frame within a user-specified reconstruction-loss threshold.It is a 3M-parameter lightweight model trained with next-token prediction and reinforcement learning on reconstruction rewards.
- Results: 4×-100× fewer patches and up to 19× ViT and 10× MLLM speedups preserve downstream MLLM performance across videos with different frame rates and resolutions.For 30-FPS 4K videos, AutoGaze can use 1% of patches.
- Benchmark: HLVid provides 268 questions on details in up to 5-minute, 4K-resolution videos, requiring visual perception at 1K-2K resolution.It is introduced to stress-test scalability beyond benchmarks that focus on long videos without high resolution.
- Results: Scaling an MLLM with AutoGaze to 1K frames and 4K resolution improves HLVid performance from 42.5% to 52.6% and beats the previous best MLLM by 4.5%.The result demonstrates improved performance in the high-resolution, long-form setting targeted by HLVid.
2. Related Work
Video understanding research progressed from specialized supervised or self-supervised encoders toward general-purpose video MLLMs. However, current models typically remain limited to short, low-resolution clips because scaling spatiotemporal processing is costly.
- Prior video understanding: Classical video understanding used 3D-ConvNets, early transformers, masked auto-encoding, predictive coding, and vision-language pre-training.These approaches include both supervised and self-supervised video encoders and pre-training algorithms.
- Recent MLLMs: Recent MLLMs extend video encoders to general-purpose video question answering and captioning.This marks a shift from specialized video representations toward broader multimodal tasks.
- Scaling limitation: Most current video MLLMs operate on short, low-resolution clips because scaling to higher spatiotemporal dimensions is costly.The passage identifies computational cost as the central scaling constraint.
3. AutoGaze for Efficient Video Understanding
AutoGaze autoregressively selects multi-scale patches that reconstruct each frame within a user-specified threshold, stopping when sufficient patches have been selected. Its lightweight design combines learned gazing, multi-stage training, multi-scale selection, and tiled inference for long, high-resolution videos.
- Model Design: AutoGaze selects patch indices autoregressively and finds the smallest per-frame gazing lengths whose optimal reconstruction loss falls below a user-specified threshold.The system predicts reconstruction loss during decoding and stops gazing for a frame once the threshold is met.
- Model Design: 3M parameters comprise AutoGaze’s convolutional encoder and autoregressive transformer decoder, whose vocabulary consists of patch indices rather than words.The decoder conditions later-frame gazing on previously encoded frames and selected patches.
- Model Design: Multi-scale gazing assigns coarse patches to low-detail regions and finer patches where needed, reducing patch counts while preserving reconstruction quality.The downstream ViT must accept multiscale patches, and multi-token prediction accelerates gazing with little performance loss.
- Training Pipeline: AutoGaze uses next-token-prediction pre-training on curated gazing sequences followed by reinforcement-learning post-training with reconstruction loss as reward.The training data include approximately reconstruction-minimizing sequences, while RL improves on their sub-optimal gazing quality.
- Downstream Usage: AutoGaze processes videos of any resolution and duration by tiling them into 16×224×224 spatiotemporal blocks, then merging gazed positions for downstream ViTs and MLLMs.This tiled procedure supports scaling to 1K-frame and 4K-resolution videos without additional training.
- Downstream Usage: HLVid contains 268 question-answer pairs about details in videos up to 5 minutes long at 4K resolution, requiring high-resolution visual perception.The benchmark was introduced to stress-test scalability where existing benchmarks focus on long duration but not high resolution.
4. Experiments
Experiments show that AutoGaze selects informative patches based on motion and detail, generalizes across unseen video styles and semantics, and substantially reduces the patches and latency required for video processing. These efficiency gains enable higher-resolution and longer-video MLLM scaling while preserving or improving benchmark performance.
- 4.1. What is AutoGaze paying attention to?: AutoGaze prioritizes moving regions and uses finer scales for more detailed patches.Across scales, it more frequently selects patches with higher optical flow; finer-scale selections also correspond to higher Laplacian variance.
- 4.1. What is AutoGaze paying attention to?: AutoGaze tracks changing regions in videos with unseen semantics, object categories, styles, textures, and illumination.The reported examples include CCTV, robot grasping, object swapping, and style-transferred videos.
- 4.2. Efficiency of ViT and MLLM with AutoGaze: ∼1% patches suffice for 30-FPS, 4K-resolution videos to reach reconstruction loss 0.7, with typical videos requiring 4×-100× fewer patches.The reconstruction-loss threshold of 0.7 usually causes less than 0.5% downstream MLLM performance degradation across benchmarks.
- 4.2. Efficiency of ViT and MLLM with AutoGaze: 19× and 10× are the maximum reported latency reductions for ViTs and MLLMs, respectively, when using the gazing ratio for reconstruction loss 0.7.AutoGaze varies the selected patch count to trade latency against the user’s reconstruction requirement.
- 4.3. Scaling MLLMs with AutoGaze: AutoGaze enables MLLMs to scale to 1K frames and 4K resolution while the baseline runs out of memory.Scaling to 4K resolution especially improves HLVid, which requires high-resolution video processing.
- 4.4. Comparing to Token Pruning Baselines: AutoGaze improves ViT latency by 4× in addition to LLM latency improvements, whereas token-reduction baselines leave ViT latency unchanged.The comparison uses 128-frame videos and a 6.25% selection ratio; both approaches retain performance comparable to the no-reduction baseline.
- 4.5. Ablations: 2.3× is the efficiency improvement attributed to multi-scale gazing, while decoding 10 tokens balances latency and gazing ratio.Decoding more tokens at a time lowers latency but requires a higher gazing ratio.
5. Conclusion
AutoGaze removes redundant video patches to improve ViT and MLLM efficiency, enabling 1024-frame 4K video understanding and stronger benchmark performance. HLVid introduces a 5-minute, 4K video QA benchmark where AutoGaze surpasses the previous state of the art by 4.5%.
- AutoGaze reduces visual tokens by 4×–100× and accelerates ViTs and MLLMs by up to 19× and 10×, respectively.The framework selects a minimal set of multi-scale patches that reconstructs the video within a user-specified threshold.
- AutoGaze enables 1024-frame 4K-resolution video understanding and improves performance on video benchmarks.
- HLVid is a 5-minute, 4K high-resolution video QA benchmark introduced for long-form video understanding.
- 4.5%: an MLLM with AutoGaze outperforms the previous SOTA model on HLVid.
A. Additional Details of AutoGaze Model Design
AutoGaze combines causal spatiotemporal visual encoding with a reconstruction-based objective to select sparse patches while preserving video information.
- AutoGaze uses a convolutional vision encoder, visual connector, and transformer decoder, with causal spatiotemporal features from the current and previous two frames.The encoder uses 2D patch embedding followed by 3D convolution over spatial and temporal dimensions.
- AutoGaze is trained to select as few patches as possible while keeping reconstruction loss below a specified level.The reconstruction objective combines a distance function with a video reconstruction model based on a video-finetuned masked autoencoder.
B. Additional Details of AutoGaze Training Pipeline
The AutoGaze training pipeline combines gazing-sequence pre-training, reconstruction-reward reinforcement learning, and diverse natural and synthetic video data with frame-wise gaze-ratio sampling.
- NTP pre-training: AutoGaze is pre-trained on about 250K videos with paired gazing sequences for 150 epochs.Training uses batch size 256 and learning rate 5e-4.
- RL post-training: RL post-training uses GRPO with reconstruction rewards that incorporate current and subsequent frames.Training samples two frames for reconstruction rewards and uses a 0.7 reconstruction-loss threshold during gazing rollouts.
- Training data: Training videos combine exocentric and egocentric natural videos with synthetic camera-motion videos derived from high-resolution image datasets.
- Training data: Frame-wise gazing ratios are sampled from a Dirichlet distribution biased toward the first frame.The first frame uses alpha 10 and subsequent frames use alpha 3 to reflect its greater amount of new information.
C. Additional Details of HLVid Benchmark
HLVid contains high-resolution, long-form video QA designed to require fine visual perception, while accompanying analyses quantify motion and visual detail in candidate patches.
- Analysis methodology: Optical-flow analysis measures motion in candidate patches using the pixelwise maximum of forward and backward flow.
- Analysis methodology: Patch detail is measured with Laplacian variance, where lower variance indicates smoother coloring and higher variance indicates busier textures.
E. Additional Quantitative Results
Additional experiments identify a reconstruction-loss threshold that preserves downstream performance and evaluate AutoGaze for streaming video. The results show that AutoGaze substantially increases processing speed and can support real-time video understanding.
- Reconstruction threshold: A reconstruction loss below 0.7 causes less than 0.5% performance drop across videos with different frame counts.Visualizations show that losses above 0.7 usually produce visible artifacts, motivating 0.7 for MLLM experiments.
- Reconstruction threshold: The selected reconstruction-loss threshold is 0.7, balancing downstream performance and efficiency.Table 5 reports little performance drop at this threshold.
- Streaming efficiency: AutoGaze achieves up to ∼16× higher maximum FPS on streaming videos and enables real-time processing that is infeasible without it.The evaluation varies video FPS and resolution and compares ViTs and MLLMs with or without AutoGaze.
F. Additional Qualitative Results
Qualitative examples show AutoGaze operating across diverse video domains and out-of-distribution cases. Its selections track informative content across lectures, sports, cartoons, surveillance, driving, robotics, and multi-view videos.
- Qualitative coverage: AutoGaze is demonstrated on lectures, sports streams, cartoons, films, surveillance footage, nighttime driving, robot arms, and split-view videos.The examples span multiple visual domains and out-of-distribution use cases.
- Qualitative coverage: The examples include picture-in-picture, fisheye-lens, warehouse, black-and-white, and other specialized video settings.These cases test behavior under varied viewpoints, distortions, and visual styles.
- Out-of-distribution behavior: AutoGaze continues tracking moving objects when an object is swapped midway through a video.The object-swapping example is presented as an out-of-distribution use case.
G. Limitations
AutoGaze has two identified limitations: it does not adequately account for most camera motion and cannot anticipate future frames using intuitive physics. These boundaries affect shifted scenes and motion extrapolation beyond observed frames.
- Camera motion: AutoGaze does not account for most camera motion and may retain patches that are redundant up to a scene shift.A panning example illustrates this limitation.
- Future-frame prediction: AutoGaze cannot anticipate future frames using intuitive physics, such as a ball continuing to fall.The causal VideoMAE is not trained with this physics knowledge.
- Scope boundaries: Camera pans and physics-driven future motion mark practical boundary cases for AutoGaze’s patch selection and reconstruction.The limitation examples concern scene shifts and extrapolating motion beyond the observed frames.
C: SONOS
Qualitative examples illustrate AutoGaze’s multi-scale selection across varied scenes and motion patterns. The method tracks relevant objects, preserves fine details when needed, and can omit unnecessary fine-scale patches.
- Lectures: AutoGaze focuses on a moving cursor and lecturer’s face in a picture-in-picture whiteboard lecture.The example shows attention shifting after the first frame.
- Fine-scale selection: AutoGaze uses finer scales to capture small or moving details, including gymnasts, cards, hands, people, and text.The examples cover uneven bars, Blackjack, walking figures, blinking, and chalkboard writing.
- Adaptive patch selection: AutoGaze skips the finest scale when it is unnecessary for the specified reconstruction loss.Other examples show tracking through lens distortion, selecting patches in both views, and following swapped objects.
- Limitations: AutoGaze does not perfectly track movement when panning across a static image and can select regions from different parts of previous frames.The example identifies a failure case in camera-motion handling.
- Limitations: VideoMAE cannot reconstruct subsequent frames of a falling ball to reflect continued motion from earlier frames.This example illustrates the limitation in predicting future frames from physics knowledge.