Source-linked AI summary
PuTR-CouT: Counting-by-Tracking in Camera-Trap Image Sequences
Fagner Cunha, Juan G. Colonna, Eulanda M. dos Santos
TL;DR
Camera-trap sequence counting is difficult because low frame rates, changing appearance, and scarce labels complicate tracking individuals across images. PuTR-CouT addresses this with transformer-based learned association trained on synthetic sequences, and achieves competitive counting while supporting multi-species predictions and track-level verification. An improved MaxBoxCount baseline achieves the highest reported benchmark score.
Problem
Camera-trap sequence counting lacks labels in most datasets and is challenged by low frame rates, large inter-frame displacement, appearance changes, and occlusions.
Method
PuTR-CouT uses transformer-based learned association to connect detections into tracks, classifies tracks individually, and counts unique tracks per species.
Results
The improved MaxBoxCount achieves the highest reported score on iWildCam 2021, while PuTR-CouT delivers competitive counting results.
Takeaways & Limitations
PuTR-CouT supports multi-species counting and direct verification through track analysis, unlike MaxBoxCount.
Takeaways & Limitations
The synthetic-label strategy assumes that sequences with at most one detection per image contain one individual, and detections above 0.4 are treated as ground truth during training.
Abstract
from arXiv · showhide
Species identification in camera trap images has been widely studied, but key ecological modeling tasks such as species abundance or density estimation also require counting individual animals. However, the lack of counting labels in most datasets and low frame rates (typically ~1 frame per second) make sequence-level tracking and count estimation particularly challenging. In this work, we present PuTR-CouT, a counting-by-tracking framework built on a transformer-based learned association mechanism for sequence-level animal counting in camera trap images. To address the scarcity of annotated tracking data, we generate synthetic training data by exploiting structural priors, such as static backgrounds and short temporal bursts, to heuristically create pseudo-tracking labels in a weakly supervised manner. The resulting tracker associates detections across frames, using these tracks to estimate per-species counts. We also refine the MaxBoxCount heuristic used by the top solutions of the iWildCam 2021 challenge as a strong baseline, setting the highest score reported to date. When evaluated on the iWildCam 2021 benchmark, our framework PuTR-CouT delivers competitive counting results compared to the improved MaxBoxCount, with the added capability of multi-species predictions and track-level verification.
1. Introduction
Camera-trap counting requires sequence-level reasoning because repeated images can duplicate individuals, while low frame rates, appearance changes, and scarce labels hinder tracking. PuTR-CouT addresses this with synthetic training data, transformer-based association, and track-based multi-species counting.
- Motivation: Sequence-level counting is needed because counting detections independently can over- or underestimate individuals appearing across repeated images.The same individual may recur across images, while different individuals may appear within one sequence.
- Challenges: Low frame rates, appearance changes, occlusions, and missing tracking or counting labels make camera-trap animal counting challenging.Sequences may be captured at about 1 frame per second, producing large displacements and low bounding-box overlap.
- Benchmark: The iWildCam 2021 benchmark evaluates sequence-level counting at low frame rates without publicly available training count labels.Top solutions used MaxBoxCount but restricted predictions to one species per sequence.
- Contribution: PuTR-CouT uses a transformer-based learned association mechanism and synthetic pseudo-tracking labels to estimate per-species counts from tracks.The synthetic data exploit camera-trap structural priors in a weakly supervised setting.
- Contribution: The paper improves MaxBoxCount to the highest reported benchmark score and develops tracking-based counting with multi-species predictions.The framework also supports species classification using multiple views of each individual.
2. Related Work
Related work spans tracking-by-detection methods and sequence-counting heuristics. Transformer trackers learn associations from detected bounding-box tokens, while prior camera-trap counting commonly uses maximum bounding-box counts.
- Multi-object tracking: Multi-object tracking detects objects, associates them across frames, and assigns unique identities, commonly separating detection from temporal association.The literature includes geometric methods such as SORT and more recent learned association approaches.
- Transformer tracking: PuTR models associations with learned attention and treats each detected bounding box as a token rather than tokenizing uniform image patches.Frames without detections generate no object tokens, apart from an initial sequence token.
- Sequence counting: Prior sequence-counting methods estimate animal numbers using heuristics such as the maximum number of bounding boxes across frames.This approach addresses counting in camera-trap image sequences without requiring full track identities.
3. Improving the MaxBoxCount heuristic as a strong counting baseline
The paper revisits MaxBoxCount as a strong baseline by updating its detection and classification components while retaining its sequence-level counting strategy.
- Baseline: MaxBoxCount estimates sequence counts from the maximum number of detected bounding boxes in any image, assuming one species per sequence.The revised baseline updates detection and classification models.
- Detection and classification: MegaDetectorV1000redwood1 generates bounding boxes for classifier training and heuristic evaluation, using thresholds of 0.3 and 0.4 respectively.The lower training threshold accommodates noisy detections, while the higher counting threshold is more conservative.
- Detection and classification: Species classification uses a Swin-B model with 224×224 inputs and standard crop, resize, and augmentation preprocessing.Augmentations include random crops, horizontal flipping, and RandAugment.
- Implementation: Training uses ImageNet-1k initialization, AdamW, 30 epochs, learning-rate warmup, cosine decay, and label smoothing.The initial learning rate is 10^-5 and weight decay is 10^-7.
- Inference: Sequence-level species predictions average weighted image-level ensembles across non-empty images in each burst.The ensemble combines full-image, mirrored full-image, bounding-box, and mirrored bounding-box predictions.
4. Counting-by-Tracking in Camera Trap Image Sequences
PuTR-CouT counts animals by associating detections into tracks, classifying each track, and counting unique tracks per species. It enables weakly supervised training through synthetic sequences constructed from camera-trap priors.
- Counting-by-tracking: Counting-by-tracking associates detections across sequence images and obtains animal counts by counting unique tracks.This reframes sequence counting as a multi-object tracking problem.
- Framework: PuTR-CouT classifies each track using its associated bounding boxes and estimates per-species counts from unique tracks.The framework is built on PuTR and uses synthetic pseudo-labels because annotated camera-trap tracking datasets are usually unavailable.
- Training challenge: Learned-association trackers require tracklet annotations, which are infeasible to obtain in most camera-trap projects.This motivates synthetic training data rather than relying only on geometric trackers.
- Synthetic data: The synthetic generator begins with sequences containing at most one detected box per image and assumes one individual throughout each sequence.The authors explicitly accept noise from this simplifying assumption.
- Synthetic data: Base-case tracks from the same location and coarse timeslot are merged to simulate multi-individual sequences while limiting background and visual inconsistencies.PuTR combines cropped-box tokens, avoiding reconstruction of composite images.
- Synthetic data: The generator uses MaxBoxCount as a proxy for unknown individual counts and samples synthetic sequence composition from fitted and categorical distributions.The procedure includes a Negative Binomial count model and species-count probabilities.
- Tracker adaptation: PuTR uses a six-block transformer with multi-head self-attention, positional encodings, and causal association between each box and the previous frame.Camera-trap bursts are fixed to 10 frames during training and inference, with padding for shorter sequences.
- Tracker adaptation: MegaDescriptor embeddings are used for bounding-box tokens to address high appearance variation in wildlife imagery.The embeddings are precomputed and projected to the PuTR model dimension.
5. Experiments and Results
Experiments on iWildCam 2021 compare improved heuristic and tracking-based counting methods using MCRMSE, with lower values indicating better performance. PuTR-CouT performs strongly, while sequence reordering helps it and removing affinity components does not improve results.
- Evaluation setup: The experiments use the iWildCam 2021 MCRMSE metric and report separate public and private test scores.The test labels are unavailable publicly, so evaluation uses late submissions to the Kaggle platform.
- Improved MaxBoxCount: Replacing only the classifier reduces MaxBoxCount error by 12.4%, from 0.0293278 to 0.0256999.Using MegaDetectorV1000 further reduces error to 0.0238785, an approximately 18.5% improvement over the winning solution.
- Tracker comparison: Track-level classification is slightly worse than sequence-level classification on the private score but has a larger advantage on the public score.ByteTrack is the weakest evaluated tracker yet still surpasses the winning solution.
- Tracker comparison: PuTR-CouT outperforms the evaluated trackers with error 0.0248642, nearly 9% better than BoT-SORT using the same re-ID model.Its Swin-B configuration reduces private-score error by approximately 15.2% versus the winning solution, while MegaDescriptor achieves the best public score at 0.0250704 versus 0.0254706 for improved MaxBoxCount.
- Ablations: Reordering sequences using EXIF timestamps reduces PuTR-CouT error by approximately 3%, whereas it increases ByteTrack and BoT-SORT private error by less than 1%.Filtering detections classified as empty increases error.
- Ablations: Disabling the additional affinity-matrix components does not improve performance, indicating that transformer representations alone remain competitive in this setting.The ablation concerns components introduced in the original PuTR implementation.
6. Conclusions
PuTR-CouT addresses sequence-level animal counting without widespread count labels through transformer-based tracking trained with weakly supervised synthetic data. It achieves competitive performance while supporting multi-species counting and track-level verification.
- PuTR-CouT uses a transformer-based multi-object tracker with learned association for counting animals in image sequences.
- Synthetic training data exploit structural priors in camera-trap sequences to create pseudo-tracking labels under weak supervision.
- PuTR-CouT achieves competitive performance against traditional multi-object trackers and MaxBoxCount.
- The approach supports counting multiple species within one sequence, unlike MaxBoxCount.
- Track analysis enables direct verification of the resulting counts.