Source-linked AI summary
SLAP: The Semantic Least Action Principle for Variational Video-Language Modeling
Xiang Fang, Wanlong Fang
TL;DR
Sparse frame sampling leaves LVLMs blind to causal transitions, while existing generative approaches can violate object persistence over long gaps. SLAP addresses this by optimizing a query-conditioned latent trajectory with a Semantic Lagrangian and discrete Euler-Lagrange equations, reporting reduced object vanishing and faster inference. Its error bound grows quadratically with gap size, limiting very long unanchored intervals.
Problem
Sparse sampling creates temporal blind spots and existing generative methods can produce hallucination flickering or object-permanence failures.
Method
SLAP models latent video states on a semantic manifold and solves interpolation as a two-point boundary-value problem by minimizing a Semantic Lagrangian with kinetic and query-conditioned potential terms.
Results
SLAP reduces object vanishing rates by 68% versus state-of-the-art diffusion inpainting, achieves a 177x inference-latency speedup, and drops 3.4% rather than 10% when using 10% of frames.
Takeaways & Limitations
The reported results support using least-action latent trajectories to improve temporal consistency and efficiency in sparse-video LVLM reasoning.
Takeaways & Limitations
As the gap size T grows, the interpolation error bound grows quadratically, motivating reasonably dense anchors or hierarchical solving for very long videos.
Abstract
from arXiv · showhide
In the era of Large Video-Language Models (LVLMs), the computational necessity of sparse frame sampling creates a fundamental ``temporal gap'', rendering models blind to critical causal transitions. Existing solutions relying on generative hallucination (e.g., latent diffusion) or autoregressive extrapolation often fail to maintain semantic consistency over long horizons, suffering from object vanishing and energetic instability. We propose a paradigm shift from probabilistic generation to variational mechanics with the \textbf{Semantic Least Action Principle (SLAP)}. Drawing a rigorous isomorphism between classical mechanics and semantic dynamics, we model the latent video trajectory as a path on a Riemannian manifold governed by a Semantic Lagrangian. By formulating the interpolation task as a Boundary Value Problem (BVP) solved via the discrete Euler-Lagrange equations, SLAP naturally enforces object persistence without pixel-level rendering. Extensive experiments show the effectiveness of our proposed SLAP.
1. Introduction
SLAP frames sparse-video interpolation as variational dynamics on a latent semantic manifold rather than probabilistic generation. Its Lagrangian Bridge solves a boundary-value problem to balance temporal continuity with text-guided semantic alignment and preserve objects across missing intervals.
- Motivation: Sparse sampling below 0.5 frames per second creates temporal blind spots that hide causal transitions between observed video states.The missing trajectory must be inferred from boundary conditions and linguistic priors.
- Motivation: Generative approaches can produce hallucination flickering and violate object permanence during occlusions.An object may disappear because statistically common empty scenes are favored over occluded-object states.
- Semantic Least Action Principle: SLAP models video as a particle moving through a high-dimensional Riemannian manifold with semantic inertia and query-induced semantic forces.The formulation treats temporal interpolation as action minimization rather than likelihood maximization.
- Lagrangian Bridge: The Lagrangian Bridge solves the temporal gap as a two-point boundary-value problem using discrete Euler-Lagrange equations for a global latent trajectory.This replaces autoregressive next-token prediction with trajectory optimization constrained by the boundary states.
- Lagrangian Bridge: A kinetic term penalizes discontinuous motion, while a text-conditioned potential term steers the trajectory toward semantic alignment without pixel-level rendering.The design targets object persistence, text-guided dynamics, and lower computational cost than pixel-space diffusion or autoregressive decoding.
2. Related Work
Prior LVLMs manage temporal information through pooling or token concatenation, but sparse sampling and quadratic attention leave much of the physical process unobserved. SLAP instead uses a Lagrangian Bridge to optimize a query-conditioned latent trajectory between fixed anchors.
- Existing LVLM Strategies: Temporal pooling compresses frame features but can discard event order and causal structure.Related architectures commonly use frozen visual encoders followed by Q-Formers or linear adapters before temporal aggregation.
- SLAP Framework: The SLAP framework encodes sampled frames as fixed anchors, maps the query into latent space, and optimizes intermediate states through a boundary-value Lagrangian Bridge.The optimized trajectory combines action minimization with semantic conditioning before downstream reasoning.
- Existing LVLM Strategies: Token concatenation preserves frame order but becomes prohibitively expensive for long videos because self-attention scales quadratically.A one-minute 30fps video with 256 tokens per frame would require approximately 460,000 tokens.
- Sparse Sampling Bottleneck: Aggressive sparse sampling typically retains only 8 to 32 frames per video, leaving most physical processes unobserved and encouraging linguistic-prior-based hallucination.The resulting temporal aliasing makes descriptions of between-frame events effectively inferential rather than directly observed.
- Kinematic Naivety: Standard Transformers model next-token probabilities without explicitly penalizing discontinuous semantic transitions or teleporting objects.SLAP addresses this missing conservation-like constraint by reintroducing motion cost through a kinetic-energy term.
3. Method: The SLAP Architecture
SLAP models sparse-video interpolation as a variational boundary-value problem on a semantic latent manifold. Its Lagrangian Bridge combines text-guided potential dynamics, smoothness constraints, and Euler-Lagrange optimization to produce stable trajectories while preserving semantic continuity.
- Architecture: SLAP maps video frames into a latent Riemannian manifold where temporal interpolation is solved as a two-point boundary-value problem.The visual encoder induces the manifold geometry, while the latent trajectory is optimized between fixed endpoints.
- Architecture: The visual encoder defines latent states, while the text encoder defines a potential field that guides trajectories toward query-compatible semantic regions.The text-conditioned potential supplies semantic forces without requiring direct pixel-space generation.
- Potential Field Network: A learned proxy potential Pθ replaces costly LLM gradient evaluation and recovers the true semantic energy up to a constant under the stated assumptions.The proxy enables gradients approximately 10^4× faster than backpropagation through a large language model.
- Inference Procedure: The discrete Euler-Lagrange solver balances kinetic smoothness against textual alignment, with convexity conditions yielding a single optimal semantic trajectory.The guarantee depends on the semantic potential’s curvature remaining bounded relative to the kinetic term.
- Training the Potential Field: Sobolev regularization penalizes large potential gradients, flattening the field near valid states so trajectory optimization avoids unstable forces and off-manifold cliffs.The regularizer constrains gradient magnitude because erratic potential surfaces can destabilize the Euler-Lagrange solver.
- Limitations: The interpolation error bound grows quadratically with temporal-gap duration, motivating reasonably dense anchors or hierarchical solving for very long videos.This is the paper’s explicit scope boundary for increasingly long gaps.
4. Experiments
SLAP is evaluated on synthetic occlusion, MSR-VTT QA, sparsity, action-centric questions, ablations, and computational scaling. Results emphasize object persistence, robustness under extreme frame sparsity, stronger action understanding, and lower computational cost.
- Evaluation Protocol: The Tunnel Test evaluates object identity through synthetic occlusion using accuracy, persistence score, and semantic drift across 1,000 held-out videos.Lower semantic drift indicates better preservation of the occluded object's identity.
- Tunnel Test: Stable Video Diffusion achieves competitive accuracy but has high semantic drift of 0.28 and often generates realistic empty tunnels instead of preserving the object.Replacing the object embedding with an empty-tunnel embedding requires a discontinuous manifold jump and incurs a kinetic-energy penalty.
- Tunnel Test: Video-LLaMA 3 experiences context drift in long tunnels, whereas SLAP uses the exiting-object boundary condition to constrain the trajectory toward the correct semantic path.Without fresh visual tokens, Video-LLaMA 3 drifts toward generic language-model priors and may describe tunnel lights or traffic instead of the tracked object.
- MSR-VTT QA: SLAP outperforms Video-LLaMA 3 by 12% on action-centric queries under extreme sparsity, where questions require trajectory understanding rather than static object attributes.The reported example interprets the transition from standing to lying down through the semantically meaningful intermediate state of falling.
- Ablation and Efficiency: The full Semantic Lagrangian requires both kinetic and potential terms, while SLAP uses 0.15 TeraFLOPs per inference compared with approximately 150 Joules per Stable Video Diffusion query.The complexity analysis contrasts SLAP's low-dimensional latent optimization with diffusion's resolution-dependent denoising and autoregressive attention costs.
5. Conclusion
The paper identifies Kinematic Naivety in LVLMs and reports that SLAP achieves state-of-the-art performance against failures in object permanence and causal consistency.
- SLAP addresses Kinematic Naivety, where video frames are treated as statistically independent tokens rather than states in a continuous physical process.
- Sparse sampling can produce failures in object permanence and causal consistency during occlusions or long temporal gaps.
- SLAP achieves state-of-the-art performance, supporting the effectiveness of the proposed method.