Source-linked AI summary
Real-World Repetition Estimation by Div, Grad and Curl
Tom F. H. Runia, Cees G. M. Snoek, Arnold W. M. Smeulders
TL;DR
Repetition estimation is challenging because real videos can have varied motion, changing viewpoints, and non-stationary dynamics, while prior work often assumes static and stationary motion. The paper derives viewpoint-dependent recurrence cases from 3D flow and measures foreground flow signals using wavelets. On QUVA Repetition, its method achieves lower repetition-counting MAE than the deep-learning and Fourier-based alternatives reported.
Problem
Repetition estimation remains challenging because motion patterns and continuity vary, viewpoint matters, and existing work often assumes static, stationary motion.
Source: §1 · p. 1Method
The paper derives 18 viewpoint-dependent 2D perception cases from 3D flow differentials and estimates repetition using wavelet-based flow signals over segmented motion foreground.
Source: §1 · pp. 1–2Results
On QUVA Repetition, the method scores MAE 23.2, versus 48.2 for and 38.5 for.
Source: §6.4 · p. 8Takeaways & Limitations
The authors report that flow-based signals over motion foreground segmentation are effective for recurrence estimation in realistic video.
Source: §7 · p. 8Analysis is inevitably hard when the camera viewpoint changes between frontal and side views.
Source: §3.4 · p. 4
Abstract
from arXiv · showhide
We consider the problem of estimating repetition in video, such as performing push-ups, cutting a melon or playing violin. Existing work shows good results under the assumption of static and stationary periodicity. As realistic video is rarely perfectly static and stationary, the often preferred Fourier-based measurements is inapt. Instead, we adopt the wavelet transform to better handle non-static and non-stationary video dynamics. From the flow field and its differentials, we derive three fundamental motion types and three motion continuities of intrinsic periodicity in 3D. On top of this, the 2D perception of 3D periodicity considers two extreme viewpoints. What follows are 18 fundamental cases of recurrent perception in 2D. In practice, to deal with the variety of repetitive appearance, our theory implies measuring time-varying flow and its differentials (gradient, divergence and curl) over segmented foreground motion. For experiments, we introduce the new QUVA Repetition dataset, reflecting reality by including non-static and non-stationary videos. On the task of counting repetitions in video, we obtain favorable results compared to a deep learning alternative.
1. Introduction
The paper addresses repetition estimation in realistic video, where motion patterns and continuity vary and camera motion complicates recurrence. It develops a flow-based account of 2D perception of 3D periodicity and evaluates a wavelet-based method on a new dataset.
1. Introduction
- Figure 1 illustrates oscillatory translation from two viewpoints and constant rotation, with abstractions of perceived 2D flow.Source: Figure 1 · p. 1
- Prior repetition methods report good results for well-localized static motion with strongly periodic, stationary dynamics, unlike many real-world videos.The paper also identifies motion-pattern variety and viewpoint, including camera motion, as challenges.Source: §1 · p. 1
- The theory derives 18 cases of 2D repetitive perception from three 3D motion types, three motion continuities, and two viewpoint extremes.The motion types derive from divergence, gradient, and curl acting on the 3D flow field.Source: §1 · p. 1
- The proposed method outperforms deep-learning state-of-the-art on repetition counting in the new QUVA Repetition dataset.The dataset is designed to include non-static and non-stationary videos.Source: §1 · p. 2
- The method measures diverse flow-based signals over segmented motion foreground and uses wavelets to handle non-stationary motion.It automatically selects the most discriminative signal based on self-estimated quality assessment.Source: §1 · pp. 1–2
2. Related Work
Earlier repetition work uses frequency analysis and other signal representations, while the paper reframes visual repetition from 3D flow and its projection into 2D. It also contrasts its realistic-video focus with prior stationary-motion settings and a synthetic-data-trained counting network.
2. Related Work
- Fourier-based methods are unable, and were not intended, to handle non-stationary motion, which the paper describes as common in the real world.Other approaches extract frequency information through peak detection or singular value decomposition.Source: §2 · p. 2
- A spatiotemporal filter bank can work online when tuned correctly, but its reported experiments use stationary motion and require manual tuning.The paper instead adopts the continuous wavelet transform for time-frequency decomposition and non-stationary repetition.Source: §2 · p. 2
- The paper derives 18 cases of 2D repetitive perception from 3D flow, motion differentials, and viewpoint, extending the prior 2D motion taxonomy.Pogalin et al. identify four visual motion types and three continuity cases in the 2D field of view.Source: §2 · p. 2
- Levy and Wolf train a repetition-counting network on synthetic sequences with four motion types; their 100-video YTSegments set is mostly static-viewpoint and stationary.The paper proposes QUVA Repetition to include more non-static and non-stationary examples.Source: §2 · p. 2
3. Theory
The theory characterizes 3D intrinsic periodicity through flow-field motion and continuity, then describes how viewpoint and camera or motion changes affect its 2D perception. It distinguishes 18 recurrent 2D cases and addresses non-stationary signals with local wavelets.
3.1. 3D Intrinsic Periodicity
- Three motion types and three motion continuities organize intrinsic 3D periodicity; mixtures can occur, so the dominant measurable periodicity is selected.For robustness to illumination changes, the gradient of the flow is preferred over the flow itself; the continuity types are constant, intermittent, and oscillating.Source: §3.1 · pp. 2–3
3.2. 2D Recurrence of 3D Intrinsic Periodicity
- Nine combinations of basic motion type and continuity form the 3D cases, while 2D recurrence concerns their observed repetition in the image.Repetitive appearance can also create recurrence for constant translation or expansion; rotational symmetry can multiply the observed recurrence rate.Source: §3.2 · p. 3
3.3. The Viewpoint
- Two extreme viewpoints—frontal and side—summarize 18 cases, while intermediate views can skew the flow signals without changing recurrence if they remain measurable.For translation and rotation, perpendicular views are indistinguishable; expansion also reduces to frontal and perpendicular cases.Source: §3.3 · pp. 3–4, Figure 3 · p. 4
3.4. Non-Static Repetition
- Independent camera motion must be removed before analyzing repetitive object motion, and a camera shift between frontal and side views makes analysis difficult.Camera motion mounted on or following the target can instead reflect the object's periodic dynamics.Source: §3.4 · p. 4
3.5. Non-Stationary Repetition
- Because realistic repetitive signals may accelerate or decay in frequency, the method does not assume stationarity and uses local wavelets.A signal is stationary when its period length is constant over time.Source: §3.5 · p. 4
4. Repetition Estimation
The method localizes foreground motion, represents it with flow-based signals, then uses time-frequency decomposition to estimate repetitions and select a discriminative signal. Wavelets and local-frequency integration accommodate non-stationary motion.
4. Repetition Estimation
- The three-stage method localizes the target, represents it with time-varying signals, then decomposes those signals to estimate repetition and select a discriminative signal.Figure 4 illustrates foreground segmentation, optical flow, signal extraction, and time-frequency decomposition.Source: §4 · p. 4, Figure 4 · p. 5
- Six signals measure average-pooled flow and its differentials over segmented foreground motion, combining two zeroth-order and four differential measurements.The flow differentials include ∇xFx, ∇yFy, divergence, and curl.Source: §4 · p. 5
- For side-view oscillating or intermittent motion, ∇F can carry the strongest repetitive signal, while flow F is stronger for constant motion appearance.The most discriminative signal can be difficult to select, motivating the min-cost selection step.Source: §4 · pp. 5–6
- The continuous wavelet transform decomposes each signal into a time-frequency representation that reveals changes in repetitive dynamics over time and scale.The Morlet wavelet’s localized energy is represented by the scalogram |Wn(s)|^2.Source: §4 · p. 5
- Integrating local frequencies over time estimates the repetition count and allows the method to handle changing frequencies, including acceleration or transient phenomena.A local frequency is approximately s^-1 for the Morlet wavelet.Source: §4 · p. 6
- The min-cost algorithm selects the signal with the lowest-cost path through the wavelet spectrum, favoring high-power paths with local regularity.Inverting wavelet power creates the cost surface; locally smooth paths can remain useful for non-stationary signals.Source: §4 · p. 6
5. Datasets, Evaluation and Implementation
QUVA Repetition adds more complex scenes and variable motion to repetition-counting evaluation, with cycle-bound annotations from untrimmed videos. The wavelet-based method outperforms a Fourier baseline on most idealized signals from the dataset.
5. Datasets, Evaluation and Implementation
- QUVA Repetition is designed to be more difficult than YTSegments in scene complexity, repetitive appearance, and cycle-length variation.The dataset is presented as a more realistic and challenging benchmark for repetition estimation.Source: §5 · p. 6
- The dataset contains 100 untrimmed YouTube videos; two annotators labeled intervals and cycles, and the 100 intervals with the highest overlap were retained.Final intervals use the intersection of the annotators’ temporal annotations.Source: §5 · p. 6
- Evaluation reports mean absolute error (MAE) and off-by-one accuracy (OBOA), also called count-within-1 accuracy.MAE is defined relative to the ground-truth count.Source: §5 · p. 6
- Implementation uses foreground motion segmentation, EpicFlow, Gaussian derivative filters, and a Morlet wavelet with logarithmic scales.Before wavelet transformation, signals are mean-filtered and linearly detrended.Source: §5 · pp. 6–7
- 83 of 100 videos: wavelet-based counting outperforms the Fourier-based baseline on idealized signals; Fourier measurements show notable error with high cycle-length variation.Time-localized wavelets are less sensitive to non-stationary repetition.Source: Figure 6 · p. 7
- The comparisons include Pogalin et al.’s Fourier-based method and Levy and Wolf’s deep-learning method using its public code and pretrained model.The Fourier baseline uses an object tracker initialized with a manually drawn box.Source: §5 · p. 7
6. Experiments
On QUVA Repetition, the proposed method outperforms Fourier-based and deep-learning alternatives, while wavelets handle non-stationarity better than periodograms and diverse flow signals address varied appearances.
6.1. Fourier versus Wavelets
- Wavelet-based counting outperforms periodograms on idealized signals and is less sensitive to cycle-length variation.Added Gaussian noise (σ = 0.5) has a minor negative effect on both methods.Source: §6.1 · p. 7
6.2. Value of Diverse Signals
- No single flow signal handles all repetitive appearances; their joint diversity gives a good lower-bound, with vertical flow Fy best overall and selected most often by the oracle.The authors attribute the vertical-flow bias to sports videos in which gravity often acts as an opposing force.Source: §6.2 · p. 7, Table 2 · p. 7
6.3. Video Acceleration Sensitivity
- Under artificial 2× acceleration halfway through videos, our method suffers less and achieves lower MAE than Levy & Wolf.Levy & Wolf excels on its own dataset, but Figure 7 shows greater difficulty with non-stationary acceleration.Source: §6.3 · pp. 7–8, Figure 7 · p. 8
6.4. Comparison State-of-the-Art
- On QUVA Repetition, our method obtains an MAE of 23.2, versus 38.5 for Fourier-based and 48.2 for deep learning; on YTSegments, leads with 6.5 versus our 10.3.On YTSegments, our method still outperforms the Fourier-based approach.Source: Table 3 · p. 8, §6.4 · p. 8
7. Conclusion
The conclusion categorizes intrinsic periodic motion through flow-field differentials and identifies 18 recurrent 2D perception cases. It reports that flow-based signals over segmented foreground motion support recurrence estimation in realistic video.
7. Conclusion
- Translation, rotation, or expansion and constant, intermittent, or oscillatory continuity yield 18 fundamental cases of recurrent perception across two viewpoint extremes.The categorization uses divergence and curl for motion type and the time-varying flow gradient for continuity.Source: §7 · p. 8
- Flow-based signals over motion foreground segmentation are effective for recurrence estimation in realistic video, including complex appearances, camera motion, and non-stationarity.A viewpoint change from side to front leaves partial continuity in ∇× F but prevents a good signal and count in one example.Source: §7 · p. 8, Figure 8 · p. 8