Source-linked AI summary
SemanticMoments: Training-Free Motion Similarity via Third Moment Features
Saar Huberman, Kfir Goldberg, Or Patashnik, Sagie Benaim, Ron Mokady
TL;DR
Video representations and benchmarks often favor static appearance over semantic motion, limiting motion-similarity retrieval. The paper introduces SimMotion benchmarks and SemanticMoments, which summarizes temporal statistics of pretrained semantic features; across the benchmarks, it outperforms existing approaches and supports motion-centric video understanding.
Problem
Existing video representations and retrieval benchmarks often favor static appearance and scene context, while action labels and common objectives under-specify semantic motion dynamics.
Method
SemanticMoments is a training-free representation that computes multiple temporal moments over pretrained semantic features to encode motion.
Results
SemanticMoments consistently retrieves videos with similar motion and outperforms existing state-of-the-art approaches across the SimMotion benchmarks.
Takeaways & Limitations
Temporal statistics in pretrained semantic feature spaces provide a motion-centric representation evaluated through dedicated synthetic and real-world benchmarks.
Takeaways & Limitations
Performance remains challenging for subtle or long-horizon motions, and the method’s ceiling is bounded by the quality of available pretrained video backbones.
Abstract
from arXiv · showhide
Retrieving videos based on semantic motion is a fundamental, yet unsolved, problem. Existing video representation approaches overly rely on static appearance and scene context rather than motion dynamics, a bias inherited from their training data and objectives. Conversely, traditional motion-centric inputs like optical flow lack the semantic grounding needed to understand high-level motion. To demonstrate this inherent bias, we introduce the SimMotion benchmarks, combining controlled synthetic data with a new human-annotated real-world dataset. We show that existing models perform poorly on these benchmarks, often failing to disentangle motion from appearance. To address this gap, we propose SemanticMoments, a simple, training-free method that computes temporal statistics (specifically, higher-order moments) over features from pre-trained semantic models. Across our benchmarks, SemanticMoments consistently outperforms existing RGB, flow, and text-supervised methods. This demonstrates that temporal statistics in a semantic feature space provide a scalable and perceptually grounded foundation for motion-centric video understanding.
1. Introduction
Motion similarity remains largely unsolved because existing video representations and benchmarks favor static appearance and scene context over semantic temporal dynamics. The paper introduces SimMotion benchmarks and SemanticMoments to evaluate and represent motion more directly.
- Motivation: Existing video retrieval methods often produce visually similar but dynamically unrelated results because their representations favor static appearance and scene context.The bias is attributed to training data and objectives that encourage models to exploit appearance cues.
- Motivation: Action-recognition labels can be recovered from a single frame, making them an imperfect proxy for motion.Identical actions may have distinct motions, while different actions may share similar dynamics.
- Benchmarks: SimMotion-Synthetic uses video pairs with identical motion but controlled differences in viewpoint and visual style to expose appearance sensitivity.Existing methods often fail to retrieve videos sharing the same motion, while large-scale annotated training data is prohibitively expensive to collect.
- Method: SemanticMoments computes higher-order temporal statistics over patch-level embeddings from pretrained semantic models, producing a compact motion descriptor without optical flow, labels, or additional training.The method uses statistics such as variance and skewness to summarize structured semantic change over time.
- Results: Across the SimMotion benchmarks, SemanticMoments consistently retrieves videos with similar motion and outperforms existing approaches.The evaluation includes SimMotion-Real, a small human-annotated benchmark designed for perceptual motion similarity independent of appearance.
2. Related Work
Prior video-representation methods use action, multimodal, or self-supervised objectives, but these objectives can preserve appearance more readily than temporal dynamics. Related work also explores motion disentanglement and semantic visual features, motivating a different use of pretrained representations.
- Video Representation Learning: Video retrieval methods commonly rely on pretrained features learned through action recognition, multimodal supervision, or self-supervision.These supervision modes define the main families of prior video representation learning.
- Action Recognition: Action-recognition architectures intended to separate appearance and motion can still learn static cues because both streams predict labels often defined by objects or scenes.The shared training objective undermines the intended decoupling of RGB and optical-flow pathways.
- Multimodal Supervision: Multimodal methods achieve strong retrieval results, but language supervision can under-specify motion in descriptions such as “a person walking”.The supplied passage identifies motion specification as a limitation of language supervision.
- Self Supervision: Self-supervised methods often make appearance consistency the simplest route to minimizing their objectives, preserving appearance even when motion changes.The related work discusses contrastive, masked-prediction, and predictive objectives as sources of this bias.
- Disentangling Motion from Appearance: Motion-disentanglement approaches use mechanisms including PCA, attention trajectories, inter-frame differences, and sparse temporal attention, while DINO-based methods show semantic features can capture motion-relevant structure.The paper adopts this semantic-feature insight for motion similarity rather than generative motion transfer.
3. The “SimMotion” benchmarks
The SimMotion benchmarks evaluate perceptual motion similarity through controlled and real-world video pairs rather than action classification. Their designs isolate appearance and viewpoint changes while preserving or contrasting motion, enabling targeted analysis of representation failures.
- Benchmark Motivation: Action categories such as walking, jumping, or dancing are insufficient for fine-grained motion similarity because they can be identified from single frames.The benchmark instead targets motion structure and dynamics beyond coarse action names.
- Benchmark Design: SimMotion benchmarks use relative comparisons with intra-class pairs for fine-grained similarity and inter-class pairs for broader action-type discrimination.This composition evaluates both detailed motion matching and separation of different motion types.
- Benchmark Design: SimMotion-Synthetic systematically varies non-motion factors while holding underlying motion constant, whereas SimMotion-Real tests generalization to curated in-the-wild scenarios.The real-world benchmark is manually curated and smaller because human annotation is time-consuming.
- SimMotion-Synthetic: The synthetic benchmark contains 250 triplets, or 750 videos, with each triplet comprising a reference, a same-motion positive, and a similar-appearance hard negative.Its five categories vary static objects, dynamic objects, dynamic appearance, scene style, and viewpoint.
- SimMotion-Synthetic: Synthetic positives preserve motion through temporally synchronized generation, while hard negatives share the base appearance but use different motion.The generation pipeline controls appearance, subject, viewpoint, and style while maintaining aligned temporal transformations for positives.
- Evaluation: Retrieval treats all other benchmark videos as additional negatives, providing a diverse pool and increasing motion-discrimination difficulty.This creates a controlled basis for evaluating motion representations.
- SimMotion-Real: SimMotion-Real contains 40 reference-positive-negative examples, with positives matching underlying motion despite changed appearance or context and negatives sharing appearance but differing in motion.The negative pairs come from different short clips of the same source video.
4. Analysis
The analysis finds that conventional embeddings do not consistently capture fine-grained motion under style-preserving changes. Higher-order moments over semantic features instead produce motion-sensitive embeddings that better group motion-equivalent variants.
- Analysis Setup: Action-recognition and video-text objectives under-specify temporal structure, while masked autoencoding can remain biased toward appearance.The analysis therefore hypothesizes that embeddings learned from these objectives are suboptimal for motion similarity.
- Controlled Analysis: Controlled walking sequences vary only in style while preserving identical dynamics, isolating motion similarity as the factor of interest.The evaluation compares pairwise cosine-similarity heatmaps over video embeddings.
- Baseline Analysis: Existing methods show partial grouping but fail to consistently isolate shared motion across different styles.The heatmaps reveal sensitivity to appearance-preserving style changes.
- Moment Analysis: Higher-order moments such as variance and skewness yield motion-sensitive embeddings that correctly cluster motion-equivalent variants while distinguishing different walking styles.This contrasts with the inconsistent fine-grained motion capture of self-supervised image and video encoders.
5. SemanticMoments
SemanticMoments constructs motion-centric video descriptors by combining multiple temporal moments of pretrained feature embeddings instead of collapsing temporal information into one pooled vector.
- M+ represents each video with a structured set of temporal descriptors, with each moment capturing a distinct temporal characteristic.
- Patch-wise temporal moments compute statistics from d-dimensional features extracted for each spatial patch across video frames.
- The first moment encodes average patch appearance, while higher-order moments capture temporal variation and directional asymmetry of change.
- Per-patch moments are spatially aggregated into one descriptor for each statistical moment, summarizing distinct aspects of temporal variation.
- The final embedding concatenates weighted moment vectors, with SemanticMoments using the first three moments so average appearance is combined with motion magnitude and polarity.
- The representation uses pretrained semantic backbones and is training-free, with uniform 32-frame sampling and minimal additional computational cost.
6. Experiments
Experiments show that SemanticMoments improves motion-focused retrieval across controlled, real-world, and gesture benchmarks by using semantic temporal statistics rather than relying on appearance or flow correspondence. Ablations support localized, multi-order moments, while real-world failures remain for difficult motions and limited backbones.
- Evaluation Protocol: The evaluation uses motion-focused retrieval, comparing established multimodal, RGB, flow-based, and self-supervised video representations.Video embeddings are normalized and ranked by cosine similarity, with closest-video retrieval success rate emphasized.
- SimMotion-Synthetic: SimMotion-Synthetic holds motion fixed while varying appearance, viewpoint, and style to expose representation sensitivity to non-motion factors.The benchmark reports retrieval accuracy across five motion-preserving edit categories.
- SimMotion-Synthetic: SemanticMoments achieves the best or competitive performance across synthetic categories, with strong gains for Dynamic Object and View.It remains near flow-level in Static Object and Dynamic Appearance while avoiding explicit flow.
- SimMotion-Real: On SimMotion-Real, SemanticMoments attains the best overall scores despite unsynchronized motion, camera changes, timing variation, and scene complexity.All methods struggle in this setting; flow loses effectiveness when clips are semantically related but not temporally aligned, while RGB and CLIP remain appearance-dominated.
- Jester: SemanticMoments consistently improves gesture metrics across backbones on Jester using K=20 majority-vote and weighted kNN evaluation.The benchmark tests gesture-level separability without training an additional classifier.
- Ablation Studies: Ablations favor multi-order localized moments on raw patch embeddings, with retrieval improving up to 32 uniformly sampled frames before gains saturate.Additive fusion offers compact precision, while concatenation can broaden recall at higher dimensionality.
- Limitations: The method remains limited by difficult motions, training-free adaptation, and the quality of available semantic backbones.Failures persist for fine gestures, long-horizon actions, multi-agent interactions, subtle dynamics, and motion defined by waiting.
7. Conclusion
The paper introduces motion-centric video similarity and two benchmarks that separate motion from appearance. It presents SemanticMoments as a training-free representation that improves motion alignment, while real-world results show that human-level motion perception remains unresolved.
- Conclusion: The paper defines motion-centric video similarity and introduces SimMotion-Synthetic and SimMotion-Real as focused evaluations of motion perception.Together, the benchmarks expose systematic limitations of current video representations.
- Conclusion: SemanticMoments encodes motion through temporal statistics of pretrained semantic features and consistently outperforms prior approaches across multiple backbones.The representation is training-free despite its broad evaluation gains.
- Conclusion: Real-world results indicate that current models remain far from human-level motion perception, leaving motion-aware video understanding an open challenge.The paper positions its benchmarks and representation as an initial step toward perceptually aligned motion understanding.