Source-linked AI summary
Swift Sampling: Selecting Temporal Surprises via Taylor Series
Dahye Kim, Bhuvan Sachdeva, Karan Uppal, Naman Gupta, Vineeth N. Balasubramanian, Deepti Ghadiyaram
TL;DR
Long-form video contains redundant frames alongside sparse, informative temporal surprises, but common sampling methods often treat these frames alike or require costly auxiliary models and tuning. Swift Sampling uses Taylor residuals in VLM latent space to select surprising frames, achieving consistent top performance across video tasks and especially strong gains on long videos.
Problem
Long-form video sampling must identify sparse, informative temporal surprises without treating redundant and pivotal frames identically or incurring auxiliary-model and tuning costs.
Method
Swift Sampling is a training-free algorithm that scores frames by Taylor-series residuals in VLM latent space, without auxiliary models or video-specific tuning.
Results
Swift Sampling achieves state-of-the-art performance across video question answering, token compression, and over ten reasoning tasks; on long LVB videos, it reaches 54.3 versus MaxInfo’s 50.8 (+3.5 points).
Takeaways & Limitations
Swift Sampling offers a lightweight, architecture-agnostic drop-in for long-form video pipelines such as token compression and captioning.
Takeaways & Limitations
Because Swift Sampling is query-agnostic, it may select visually surprising but semantically irrelevant frames, including shot transitions.
Abstract
from arXiv · showhide
While most frames in long-form video are redundant, the critical information resides in temporal surprises: moments where the actual visual features deviate from their predicted evolution. Inspired by the human brain's predictive coding, we introduce Swift Sampling, an elegant, training-free frame selection algorithm that automatically identifies high-information moments in a video. Specifically, we model a video as a differentiable trajectory in the visual latent space and compute the velocity and acceleration of its features. Then, we apply Taylor expansion to project the expected path of subsequent frames. Frames that diverge sharply from this predicted manifold are identified as temporally surprising frames and selected for sampling. Unlike prior training-free methods that rely on auxiliary networks or video-specific hyperparameter tuning, Swift Sampling is incredibly lightweight, adding only 0.02x additional computational cost over baseline making it 30x cheaper overhead than leading baselines. Across three long-video question answering benchmarks and 10 different downstream tasks, Swift Sampling outperforms uniform sampling and prior query-agnostic baselines. It is especially powerful for long videos with limited frame budgets improving accuracy by up to +12.5 points.
1 Introduction
Swift Sampling addresses temporal redundancy in long-form video by using predictive coding to identify sparse frames whose visual latent features diverge from their expected trajectory. The training-free method uses Taylor-series residuals without auxiliary models or video-specific tuning, targeting efficient frame selection for video understanding.
- Motivation: Long-form video contains extended predictable intervals punctuated by sparse informative transitions, but most VLMs still use uniform sampling and treat frames identically.This overlooks temporal structure and redundant frames’ unequal information content.
- Contributions: The authors report that Taylor-expansion-based temporal-surprise detection is robust enough to replace expensive prior methods as a drop-in component.The stated goal is to bridge low-level temporal motion with high-level LLM reasoning.
- Method: Swift Sampling scores frames by Taylor-series residuals in the VLM’s latent space, operationalizing predictive coding without auxiliary models or video-specific tuning.It treats adjacent-frame visual latent features as points on a trajectory and measures divergence from the expected path.
- Contributions: Swift Sampling achieves state-of-the-art performance against uniform sampling and prior training-free methods across VLM backbones and diverse video-understanding tasks.The listed evaluation areas include video question answering, token compression, and more than ten other tasks.
- Method: Frames with large prediction residuals represent temporal surprises, such as a seal suddenly emerging from the ice, and are selected for sampling.The residual compares the predicted next-frame feature with the actual feature on the latent trajectory.
2 Related Work
Prior work on long-video understanding improves temporal modeling, multimodal fusion, and multi-scale encoding, while frame selection spans training-based and training-free approaches. Other efficiency methods adjust or merge tokens, whereas Swift Sampling filters redundant frames before tokenization and builds on derivative-based temporal prediction.
- Video large language models and long video understanding: Video large language models perform well on short videos, but long-video understanding remains difficult because of the large number of input frames.Prior solutions improve temporal modeling, multimodal fusion, and multi-scale encoding.
- Frame selection for long video understanding: Frame-selection methods for long videos follow training-based and training-free directions, using task losses, candidate ranking, pseudo-labels, reinforcement or self-learning, and keyframe annotations.
- Frame selection for long video understanding: Tokenization methods dynamically adjust token counts or merge redundant spatial and temporal tokens, whereas Swift Sampling retains informative frames before tokenization.Filtering redundant frames at the input level makes Swift Sampling complementary to tokenization-based efficiency methods.
- Taylor series for video understanding: Taylor-series methods model local behavior through value, velocity, acceleration, and higher-order terms, supporting predictive representations for video understanding and generation.Taylor Video uses higher-order Taylor residuals as a motion representation that replaces or complements RGB frames for action classifiers.
3 Swift Sampling: Our Approach
Swift Sampling selects a fixed budget of informative video frames by predicting latent visual features with a truncated Taylor expansion and ranking temporal-surprise residuals. It preserves diverse important events by selecting local residual maxima rather than only globally high-residual frames.
- Method overview: Swift Sampling selects K ≤ T frames that are most informative for downstream video models using Taylor-series prediction of latent visual features.The method is organized around a Taylor predictor and a Taylor-residual informativeness score.
- Taylor predictor: Taylor predictors estimate future features from preceding frames by approximating derivatives with backward finite differences.An n-th-order derivative uses the current frame and n preceding frames, enabling closed-form prediction from observations.
- Taylor residual: Temporal surprise is the prediction-error magnitude, with large Taylor residuals indicating informative deviations and small residuals indicating redundancy.This criterion follows predictive coding, which equates informativeness with prediction error relative to prior context.
- Taylor residual: Taylor residuals provide a tractable informativeness surrogate because, under an isotropic Gaussian innovation model, self-information increases monotonically with residual magnitude.The Gaussian interpretation is presented as a motivation rather than a claim about the vision encoder’s projections.
- Local Maxima Selection: The algorithm selects the K largest residuals among local maxima, then fills remaining slots from non-maxima when fewer than K local maxima exist.Local-maxima selection avoids concentrating all frames in high-motion bursts and helps represent subtler critical events.
4 Experiments
Experiments across long-video VQA benchmarks show that Swift Sampling consistently improves accuracy, especially for longer videos, while adding negligible inference cost. It also strengthens complementary selection and compression methods and benefits reasoning-intensive tasks.
- Experimental Setup: Experiments evaluate Swift Sampling on Video-MME, MLVU, and LongVideoBench using LLaVA-OneVision and LLaVA-Video backbones with 128 candidates reduced to K = 32 frames.Frame representations come from mean-pooled spatial tokens at the vision encoder’s first transformer layer (ℓ = 0).
- Main Results: +6.8 points on LVB videos longer than 20 minutes (47.5 →54.3), with similarly strong gains on other long-duration subsets.Additional improvements include +6.4 points on LVB videos longer than 10 minutes and +4.2 points on MLVU videos longer than 30 minutes.
- Efficiency: 1.02× inference cost preserves competitive performance against query-agnostic baselines, adding only 0.02× overhead versus approximately 1.8× for methods using separate vision encoders.Swift Sampling reuses the target Video LLM’s existing vision encoder, specifically its first few layers, to compute frame representations.
- Combining with AKS: +2.1% on Video-MME and +0.5% on MLVU versus standard AKS are achieved while reducing inference cost from 2.06× to 1.59×.Swift Sampling acts as a high-speed pre-filter before more expensive query-based scoring, including reductions from 256 to 128 candidates.
- Task Analysis: +5.4% in Spatial Reasoning, +3.9% in Action Reasoning, +2.8% in Temporal Reasoning, and +2.2% in Action Recognition show strongest gains on reasoning-intensive tasks.The analysis attributes these gains to selecting frames that distinctively capture motion, while global temporal continuity tasks can regress.
5 Analysis of Swift Sampling
Swift Sampling’s analysis identifies low-order Taylor expansion, tight frame budgets, global mean pooling, and early-layer key features as effective design choices. In particular, N = 3 balances accuracy across benchmarks, while temporal-surprise selection delivers its largest gains on long videos under constrained budgets.
- Choice of Feature Pooling: Global mean pooling provides the best balance across benchmarks for spatial aggregation before computing Taylor residuals.The analysis sweeps patch-grid resolutions from S = 1 to S = 14 using SigLIP’s 14 × 14 token grid.
- Taylor Expansion Order: N = 3 provides the best overall balance in VQA accuracy across Video-MME, LongVideoBench, and MLVU.Accuracy improves sharply from N = 1 to N = 3, then largely saturates; MLVU shows only marginal improvement at N = 6.
- Choice of Feature Layer and Type: Mean-pooled key features from the earliest vision-encoder layers produce the lowest average residuals and the most favorable accuracy–cost trade-off.The study compares self-attention projection Wk features with hidden encoder-block outputs.
- Frame Budget: +12.5 points is Swift Sampling’s gain over uniform sampling for videos exceeding 30 minutes at K = 4.The gain is +10.4 points at K = 2, and Swift Sampling consistently outperforms uniform sampling across frame budgets.
6 Conclusion and Future Work
Swift Sampling concludes as a lightweight, training-free framework that selects keyframes via Taylor residuals and integrates readily into long-video understanding pipelines. Its query-agnostic design can select semantically irrelevant visual surprises, motivating task-sensitive and multimodal extensions.
- Conclusion: Swift Sampling identifies keyframes via Taylor residuals while reusing only early VLM vision-encoder layers.This design supports a training-free framework for long-video understanding.
- Conclusion: 0.02× overhead is added by Swift Sampling, 30× less than prior baselines.The method is positioned as a lightweight drop-in for token compression and captioning pipelines.
- Future Work: Swift Sampling’s query-agnostic selection can favor visually surprising but semantically irrelevant frames, including shot transitions.This limitation arises from prioritizing efficiency.
- Future Work: Future work will adapt the Taylor signal to be task-sensitive and extend the framework to audio and spatio-temporal modalities.These directions target more context-aware video understanding.