Counting Out Time: Class Agnostic Video Repetition Counting in the Wild

• Five email addresses under google.com are listed for debidatta, yusufaytar, tompson, sermanet, and zisserman.Source: §1 · p. 1

1. Introduction

• A lack of large-scale labeled repetition-video data has limited research, because video text rarely describes repeated activity and fine-grained manual labeling is tedious and expensive.The introduction contrasts repetition labels with large action-recognition datasets such as Kinetics.Source: §1 · pp. 1–2
• The method synthesizes repetition videos by repeating clips from existing videos at different periods, then combines synthetic and real data to train the model.Synthetic clips provide period and count annotations, while augmentation is intended to help transfer to real repetitions.Source: §1 · p. 2
• RepNet uses a temporal self-similarity matrix as a narrow intermediate representation to predict repetition periods across known and unseen video classes.The representation reduces the feature map from 512 channels to one and constrains period prediction to the self-similarity matrix.Source: §1 · p. 2
• The paper studies per-frame repetition counting via period estimation and count conversion, plus per-frame binary detection of whether a frame belongs to a repeating pattern.The introduction motivates these tasks with applications including exercise tracking and biological-event measurement.Source: §1 · pp. 1–2
• RepNet substantially outperforms state-of-the-art methods on existing repetition-counting and periodicity-detection benchmarks, and Countix is about 90 times larger than the previous largest dataset.The paper also introduces RepNet and synthetic-video generation and augmentation as contributions.Source: §1 · p. 2

2. Related Work

• Earlier periodicity work leveraged temporal autocorrelation, and repetition counting built on predicting cycle length; RepNet adds a temporal self-similarity matrix to that approach.The related-work passage contrasts RepNet with, which did not use a TSM.Source: §2 · pp. 2–3
• Prior video-period estimation used autocorrelation-based periodograms or wavelet transforms on hand-designed optical-flow features, while temporal self-similarity matrices served recognition, gait analysis, and periodic-segment identification.The cited prior periodic-segment approach used special filters with a TSM based on Improved Dense Trajectories.Source: §2 · p. 2
• RepNet uses a temporal self-similarity matrix as an intermediate information bottleneck in an end-to-end network, rather than only for prior recognition or periodic-segment tasks.The related-work discussion says earlier synthetic counting patterns were hand-designed and insufficiently diverse, motivating realistic repetition videos made from existing datasets.Source: §2 · p. 2

3. RepNet Model

• RepNet uses an encoder, a temporal self-similarity layer, and a period predictor to estimate per-frame period lengths and periodicity.The self-similarity matrix is the intermediate representation between the learned components.Source: §3 · p. 3

3.1. Encoder

• The encoder extracts per-frame convolutional features, adds local temporal context, and pools spatial dimensions into frame embeddings.The temporal context helps distinguish similar-looking frames with different motion; pooling removes the need to explicitly track a region of interest.Source: §3.1 · p. 3

3.2. Temporal Self-similarity Matrix (TSM)

• The temporal self-similarity matrix compares frame embeddings pairwise, forming a one-channel information bottleneck that also makes predictions temporally interpretable.Pairwise similarity uses negative squared Euclidean distance followed by row-wise softmax.Source: §3.2 · p. 3

3.3. Period Predictor

• The period predictor takes per-frame self-similarity representations and outputs a period-length estimate and a binary periodicity classification for each frame.Real-world matrices vary across uniform, accelerating, decelerating, and interrupted repetitions.Source: §3.3 · p. 3, Figure 3 · p. 4

3.4. Losses

• Period length is trained as a multiclass classification task over discrete lengths, while periodicity uses binary cross-entropy.The experiments use 64 input frames and vary frame rates to predict larger periods.Source: §3.4 · p. 4

3.5. Inference

• Inference sums per-frame counts, computed from periodicity and period-length predictions; evaluating at 1×, 2×, 3×, and 4× speeds covers longer periods.Counting benchmarks contain only periodic segments, so periodicity is set to 1 for those experiments.Source: §3.5 · p. 4

4. Training with Synthetic Repetitions

• Synthetic repetitions from unlabeled videos provide a cheaper, more scalable alternative to manually annotating period boundaries in real videos.The generated data supplies period-length and periodicity labels for training.Source: §4 · p. 4

4.1. Synthetic Repetition Videos

• The pipeline samples a random-length clip from a Kinetics [23] video, repeats it, optionally reverses it, and adds neighboring non-repeating segments.Using real videos is intended to provide more realistic and diverse training data than synthetic patterns.Source: §4.1 · pp. 4–5

4.2. Camera Motion Augmentation

• Camera augmentation smoothly varies affine viewing-frame rotation, translation, and scale; without it, training loss falls but the model does not transfer to real repetition videos.The authors identify this augmentation as vital to effective training with synthetic videos.Source: §4.2 · p. 5, Figure 5 · p. 5

5. Countix Dataset

• QUVA contains 100 class-agnostic test videos with realistic challenges, but its size limits semantic coverage and supervised deep-model training.The cited challenges include occlusion, camera movement, and changes in action speed.Source: §5 · p. 5
• Countix covers diverse real-world repetition settings, including camera and object motion, varied periods and counts, and changing action speed.The dataset is about 90 times larger than the previous largest repetition-counting dataset, QUVA.Source: §5 · p. 5
• Countix is drawn from Kinetics [23] classes selected for likely repetitions, with crowdsourced segment and count labels aggregated by median after overlap checks.The dataset retains Kinetics train/validation/test splits to avoid data leakage when using Kinetics-pretrained models.Source: §5 · p. 5

6. Experiments

• Figure 6 presents examples of repeating videos from the Countix dataset and shows their distribution by count and period length labels; Table 1 reports statistics for Countix and QUVA Repetition datasets.Source: Figure 6 · p. 6, Table 1 · p. 6

6.1. Benchmarks and Evaluation Metrics

• The evaluation follows established PERTUBE periodicity metrics and reports OBO and MAE on QUVA and Countix validation sets.OBO counts predictions within one of the ground truth as correct; MAE is the normalized absolute count difference.Source: §6.1 · p. 6

6.2. Implementation Details

• Training uses an ImageNet-pretrained ResNet-50 encoder, ADAM, learning rate 6 × 10−6, batch size 5 videos, and 400K steps.Each video in a batch has 64 frames; ablations use synthetic repetition data unless stated otherwise.Source: §6.2 · p. 6

6.3. Ablations

• Adding the temporal self-similarity matrix improves performance across metrics and validation datasets, and synthetic-only training generalizes to real repetitions.RepNet performs similarly on Countix with synthetic or Countix training data, but Countix-only training is worse on QUVA; combining datasets gives the best OBO error.Source: §6.3 · p. 6, Table 2 · p. 7
• The transformer outperforms LSTM, temporal CNN, and 2D-CNN alternatives, while more camera-motion augmentation improves performance on both datasets.The comparisons use synthetic repetition data.Source: §6.3 · p. 7, Table 3 · p. 7, Table 4 · p. 7

6.4. Evaluation on Benchmarks

• AUC 0.969 on PERTUBE; RepNet outperforms prior methods on periodicity detection and repetition counting on established benchmarks.The paper also reports Countix test results as evidence that RepNet is an effective baseline.Source: §6.4 · p. 7, Table 5 · p. 7, Table 6 · p. 7

6.5. Qualitative analysis

• The TSM encodes action acceleration and deceleration; learned embeddings show quasi-sinusoidal patterns and similar states across different periods.A common failure is half-counting when model motion cues differ from annotators’ repetition conventions, as with juggling.Source: §6.5 · pp. 7–8, Figure 7 · p. 7

Applications

• Consecutive differences in predicted repetition rates can track speed changes, including whether someone exercising speeds up or slows down.The paper points to exercise tracking as an application.Source: §7 · p. 8
• RepNet can estimate frequencies of repeating phenomena such as heartbeats, including from Eulerian-magnified videos without explicit training on those videos.The paper describes echocardiogram examples on its project webpage.Source: §7 · p. 8
• Higher input frame stride can extend period estimation to slow events; learned embeddings can support cross-period retrieval and identify repeating sections for self-supervised learning.The examples include yearly satellite-image changes and potential use with existing self-supervised objectives.Source: §7 · p. 8

8. Conclusion

• Synthetic training data combined with temporal self-similarity yields a class-agnostic model that detects periodicity and predicts counts across diverse actors and sensors.The conclusion lists objects, humans, animals, Earth, standard cameras, ultrasound, and laser microscopes.Source: §8 · p. 8
• The demonstrated scope is simple repetitions; multiple simultaneous signals and temporal arrangements such as dance steps and music remain next steps.Source: §8 · p. 8

A.1. Counting on Videos with Different Sensors

• The same synthetic-data-trained model estimates heartbeat rates from echocardiograms and rotation rates from laser-microscope videos, and counts in Eulerian-magnified videos without further training.The echocardiogram estimates are reported close to device-measured heart rates.Source: §A.1 · p. 11, A. Qualitative Results · p. 11

A.2. Physics Experiments with RepNet

• The predicted long-to-short pendulum period ratio is 1.566, compared with an approximate physics-based estimate of 1.612 from video-measured lengths.The authors describe period estimates as a possible replacement for stopwatch measurements in the experiment.Source: §A.2 · p. 11

A.3. Consistent Multi-view Period Predictions

• RepNet recovers the Tacoma Bridge collapse frequency robustly from different viewpoints.Source: §A.3 · p. 11

A.4. Inspecting Changes over Periods

• From 25 years of Arctic ice-cover satellite imagery, RepNet predicts a period of roughly one year and marks frames about a year apart.The passage notes that ice cover visibly decreases over the years.Source: §A.4 · p. 11

A.5. Video Galleries

• The appendix provides galleries of 1D PCA embedding projections and learned temporal self-similarity matrices.Source: §A.5 · p. 11

B.1. Removing Countix Classes from Synthetic Data

• Removing Countix classes from synthetic training has marginal performance impact, supporting generalization without those classes in the synthetic pool.Source: §B.1 · p. 11, Table 8 · p. 11

B.2. ImageNet Pre-training

• Training from scratch is 4% worse than ImageNet pretraining, yet still exceeds current state-of-the-art repetition-counting methods.ImageNet initialization without further encoder training is also sufficient for repetition counting; Figure 9 shows frames one period apart depicting declining Arctic ice cover.Source: §B.2 · p. 11, Table 9 · p. 11, Figure 9 · p. 12

B.3. Camera Motion Augmentations

• Omitting any tested camera-motion augmentation worsens OBO error by about 1.5%–2% and MAE error by about 13%–26%.The authors therefore use all tested augmentation techniques in subsequent experiments.Source: §B.3 · p. 12, Table 10 · p. 12

B.4. Varying Number of frames

• Using N = 64 input frames gives the best reported performance, and this setting is used throughout the main-paper experiments.Table 11 reports results for varying the number of frames.Source: §B.4 · p. 12, Table 11 · p. 13

B.5. Other Architectural Choices

• Other tested architectural choices have only a minor impact on overall performance.Table 12 reports architecture variations with synthetic-data training.Source: §B.5 · p. 12, Table 12 · p. 13

C.1. Detailed Architecture

• The appendix provides a detailed RepNet architecture, with table notation distinguishing 2D and 3D convolution filters and fully connected layer channels.The notation uses [n × n, c] for 2D filters, [n×n×n, c] for 3D filters, and [c] for fully connected layer channels.Source: §C.1 · p. 13, Table 13 · p. 14

C.2. Architectures of Alternative Baselines

• Alternative baselines include a 2D CNN for clip-level period classification, a 512-unit LSTM, and a seven-layer dilated temporal CNN.The 2D CNN uses convolutional layers with channels [32, 64, 128, 256, 512] and 3 × 3 filters; a ResNet50 gave similar performance.Source: §C.2 · p. 13

C.3. Combining Period Length and Periodicity Outputs during Inference

• RepNet processes consecutive N-frame windows to produce per-frame periodicity and period estimates, then sums thresholded per-frame counts into a video count.A frame contributes 1/l_i when its periodicity score exceeds threshold T, and zero otherwise.Source: §C.3 · p. 13

D.1. Countix Details

• Countix’s class list and descriptions of the repetitions in those classes are provided in Table 14.Source: §D.1 · p. 13, Table 14 · p. 15