Source-linked AI summary
CFC26: Building Evaluations for Deployment in Sonar-Based Fish Counting
Madison Van Horn, Suzanne Stathatos, Sevan Brodjian, Justin Kay, Kai Van Brunt, Erik Young, Sara Beery, Pietro Perona, Michael Hobley
TL;DR
Sonar-based fish counting remains difficult to evaluate for real deployment, where performance can shift at new rivers and existing metrics may not reflect ecological counting goals. CFC26 expands the benchmark and adds deployment-oriented protocols and metrics. Its evaluations reveal directional undercounting that nMAE can miss, while the dataset remains limited to riverine sonar fish counting.
Problem
Method
CFC26 expands the sonar fish-counting dataset and evaluates in-distribution and held-out-river performance with new counting metrics nMNE and nMANE.
Source: §1 · p. 2, §3.1.1 · p. 6, §3.1.2 · p. 7Results
Across locations, nMANE is negative in most conditions, revealing dominant-direction undercounting that nMAE can conceal.
Source: §5 · p. 9Takeaways & Limitations
Directional bias in passage counts can lead to over- or under-estimation of fish passage, making it relevant to ecological interpretation of deployment results.
Source: §7 · p. 10CFC26 covers riverine sonar fish counting in North America, so its findings may not generalize to other sensing modalities or marine settings.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Accurate counts of fish populations are essential for monitoring ecosystems and making decisions for conservation, harvest limits, and river management. In many field deployments, these counts are obtained from sonar videos that experts must inspect manually, making large-scale monitoring slow, expensive, and difficult. Automated counting systems have been proposed as a scalable alternative, but so far this promise has been difficult to realize in deployment. Developing evaluations that better reflect real-world deployment conditions will help close this gap and drive improvements in counting performance in practice. To achieve this, we propose Caltech Fish Counting 2026 (CFC26), a new dataset with improved evaluation protocols that simulate deployment to seen and novel rivers, testing in-distribution and out-of-distribution performance. We expand the CFC22 dataset to eleven deployment locations across nine Pacific salmon river systems from Alaska to Northern California, capturing a broader geographical and ecological range of data. We additionally introduce two new evaluation metrics for counting, nMNE and nMANE, that mirror the operational workflow that experts use and help surface errors that frame-level detection metrics do not capture. Using these metrics, we reveal previously-concealed directional bias in upstream vs downstream counts, which would have direct and significant ecological implications if undetected. The dataset and code are publicly available on CaltechDATA and GitHub.
1 Introduction
CFC26 addresses the gap between benchmark performance and field deployment by expanding the dataset and revising evaluations to reflect temporal change, new rivers, and operational counting goals. Field deployment of CFC22-performing models exposed larger performance drops at more ecologically divergent locations and showed that nMAE does not reveal directional bias.
1 Introduction
- Deployment of models that performed well on CFC22 revealed larger performance drops at locations farther from the training river, while CFC22’s OOD rivers did not fully stress-test ecological shift.The authors report that deployment results did not hold and that the failure intensified with broader ecological shift.Source: §1 · p. 2
- The CFC22 metric nMAE measures absolute crossing error but cannot show directional bias or whether counts are over- or under-estimated.The authors identify this mismatch between the metric and fisheries managers’ operational goals through deployment experience.Source: §1 · p. 2
- CFC26 expands CFC22 from 1,567 to 4,363 annotated videos and from 515,933 to over 1M bounding boxes across eleven deployment locations spanning nine ecologically distinct river systems.Source: §1 · pp. 2–3
- CFC26 uses temporal data splits and ID and held-out-river OOD protocols to better simulate deployment, where conditions change over time and locations may be unseen.The protocols make generalization measurable and directly simulate deployment conditions practitioners face.Source: §1 · p. 3
- CFC26 introduces nMNE and nMANE to detect directional bias in aggregate passage counts, alongside a broader dataset intended to support meaningful OOD evaluation.The dataset covers eleven deployment sites across nine ecologically distinct river systems.Source: §1 · p. 3
2 Related Work
Prior work established sonar fish-analysis tasks and counting benchmarks, but many datasets were narrow in geography, scale, or annotation, and common counting metrics emphasize frame-level outcomes. CFC26 follows benchmark work that treats real-world variation and deployment-relevant evaluation as central to measuring progress.
2 Related Work
- Benchmark conclusions can change when refreshed test sets expose missed structure, and robustness gains depend on which natural distribution shifts are evaluated.CFC26 applies this view by emphasizing deployment variation, not annotation volume alone.Source: §2 · p. 3
- Prior sonar datasets often covered narrow settings; CFC22 broadened evaluation to detection, tracking, and counting but had limited OOD evaluation.Examples include a single-river, 524-clip classification dataset and Kenai-only work split across sensors within one river.Source: §2 · p. 3
- Most counting benchmarks use per-image or per-frame cardinality metrics, while sonar salmon-counting methods commonly detect fish frame by frame or count tracks built from detections.The operational target is aggregate passage at an unseen river, where frame accuracy matters insofar as it supports reliable deployment-level counts and exposes aggregation failures such as directional bias.Source: §2 · p. 3
- Low-SNR video benchmarks share the challenge that appearance is weak and motion carries the signal; CFC26 additionally targets aggregate passage estimates at unseen rivers.The cited neighboring examples include camouflaged-animal and medical-video datasets.Source: §2 · p. 4
3 The CFC26 Dataset and Benchmark
CFC26 broadens sonar fish-counting evaluation with geographically and operationally diverse data, improved annotations, deployment-oriented splits, and counting metrics that capture directional bias. Its protocol evaluates transfer to unseen rivers alongside performance where target-location data is available.
3 The CFC26 Dataset and Benchmark
- CFC26 contains 4,363 clips and over 1M bounding boxes across eleven locations and nine river systems, spanning diverse fish sizes, densities, and sonar ranges.It nearly doubles CFC22’s scale; locations cover fish sizes from 2.03e-04 to 1.23e-02 normalized area and densities from 0.42 to 7.69 fish per frame.Source: §3 · p. 4, Table 1 · p. 6
- Chronologically ordered, non-overlapping train, validation, and test splits use natural river-specific temporal gaps to approximate evaluation on future deployment data.The target is approximately 60% training and at least 10% each for validation and test, with exact proportions depending on deployment duration and gap structure.Source: §3 · p. 4, Figure 3 · p. 5
- Re-annotation of Kenai-Channel recovered 55.1% additional tracks, while CFC26’s pipeline uses echograms to help identify small or low-contrast fish and applies quality control.Videos with substantial unresolved annotation problems or missing annotations may be excluded; 2,794 videos were excluded, primarily for annotation quality.Source: §3 · p. 5
3.1.1 Generalization Evaluation
- The primary OOD regime holds out entire locations during training to simulate first deployment at a new river, while ID evaluation includes labeled data from the tested location.Interpreting OOD alongside ID performance helps distinguish weak transfer from broader performance problems.Source: §3.1.1 · p. 6
3.1.2 Counting Metrics
- The evaluation follows fish biologists’ workflow by linking frame-level annotations into tracks and counting their crossings of a virtual center line as upstream or downstream passage.The upstream direction varies by deployment, so counts use upstream and downstream rather than left and right.Source: §3 · p. 4, §3.1.2 · pp. 6–7
- nMAE measures normalized total crossing error, while nMNE and nMANE capture net movement error and directional bias that total-error metrics can obscure.Negative nMANE indicates systematic undercounting in the dominant direction; positive values indicate overcounting.Source: §3.1.2 · p. 7
4 Experimental Setup
The experiments vary access to target-river data to assess performance under deployment conditions. They compare balanced sampling and three training regimes, using YOLOv5-medium with ByteTrack and reporting counting errors across locations.
4 Experimental Setup
- The experiments test how performance changes as target-river access ranges from full inclusion to complete exclusion from training.The central question is how much performance degrades when a model has never seen the target river.Source: §4 · p. 7
- Balanced Sampling (ID) achieves the lowest nMAE at most locations, while holding out the target river substantially degrades performance at harder sites such as Eel and Dungeness.Directional errors are negative across nearly all conditions, with the largest bias at Chemainus and Eel; Chemainus’s nMNE of 3.67 under Balanced Sampling (OOD) is truncated.Source: Figure 4 · p. 8
- Balanced Data Sampling fixes training at 10k fish-containing samples per location, upsampling locations with fewer qualifying frames to control location imbalance.Klamath contributes 24.8% of bounding boxes; validation uses 3k samples per location.Source: §4 · p. 8
- The regimes train on the target river only, all rivers, or all rivers except the target; OOD is the primary test of first deployment without target data.Target River (ID) measures performance when training is limited to a single deployment location.Source: §4 · p. 8
- The detector and tracker are YOLOv5-medium and ByteTrack, with detector architecture selected using validation-set mAP50.Training uses SGD with initial learning rate 0.01, final learning rate 0.0001, momentum 0.937, and weight decay 0.0005.Source: §4 · p. 8
5 Results
CFC26 results distinguish robust locations, distribution-sensitive locations, and persistent detection or directional-bias failures. Target-location data helps where detection is tractable, while nMANE reveals directional errors that nMAE can obscure.
5 Results
- Balanced Sampling (ID) achieves the lowest or near-lowest nMAE at most locations, with clearer gains where errors are moderate, including Chemainus at 0.39.Chemainus improves from 0.43 under Balanced Sampling (OOD) and 0.49 under Target River (ID); Elwha’s ID nMAE is near 0.15.Source: §5 · p. 8
- Mad’s nMAE falls from 0.80 under Balanced Sampling (OOD) to 0.33 under Balanced Sampling (ID), while its OOD nMANE is −0.11.Unlike Eel, Mad improves substantially with target-location training; the smaller directional error indicates more balanced errors across movement directions.Source: §5 · pp. 8–9
- Across locations, nMANE is negative in most conditions; at Chemainus it ranges from −0.47 to −1.0 despite moderate nMAE values.At Eel, nMANE reaches approximately −0.80 under Balanced Sampling (ID), −1.0 under Balanced Sampling (OOD), and −0.80 under Target River (ID).Source: §5 · p. 9
- At Chemainus, Balanced Sampling (ID) has the best nMAE, 0.39, but the worst directional error, with nMANE at −1.0.Selecting by nMAE alone would miss severe undercounting of dominant movement.Source: §5 · p. 9
- Kenai-Left and Kenai-Right have nMAE generally below 0.07 and 0.03, respectively, across training conditions, with nMANE close to zero.Eel remains difficult even with target-location data: Balanced Sampling (ID) yields nMAE of 0.80, and some conditions reach 1.0.Source: §5 · pp. 8–9
- Across locations, results separate robust performance, sensitivity to training distribution, and persistent difficulty or directional bias, each requiring different evaluation conditions or metrics.Held-out-river conditions expose distributional failure; directional metrics expose bias that magnitude error alone can miss.Source: §5 · p. 9
6 Limitations
The CFC26 benchmarks have limits in statistical certainty, geographic and task scope, and data coverage. They are a static snapshot rather than a continuously updated data engine.
6 Limitations
- Experiments use single runs without variance estimates or error bars, so results should be interpreted as directional rather than definitive.Full multi-seed evaluation across conditions and eleven locations was impractical; Klamath contributes 53.9% of clips, and Balanced Sampling only partly offsets location imbalance.Source: §6 · p. 9
- CFC26 covers sonar-based fish counting in North American river systems; other sensing modalities, marine settings, species, and some river conditions may not generalize from it.Low signal-to-noise and ambiguous visibility can affect annotations; it is not recommended for fine-grained species classification, re-identification, or appearance-based recognition independent of motion cues.Source: §6 · p. 9
- CFC26 is a static release, not a continuously updated data engine, motivating evaluation frameworks that are refined as deployment conditions change.The dataset is a snapshot of current deployment conditions.Source: §6 · pp. 9–10
7 Conclusion
CFC26 updates the evaluation goalpost using lessons from field deployment. Its expanded data and metrics target outstanding gaps in representing deployment conditions and ecological counting outcomes.
7 Conclusion
- Field deployment exposed gaps in CFC22’s evaluation, prompting CFC26 to capture conditions that matter more directly to real deployment.The authors describe CFC22 as enabling initial progress, but not a full snapshot of reality.Source: §7 · p. 10
- CFC26 expands coverage to nine rivers, nearly triples video clips, and provides more than one million high-quality bounding-box annotations linked to individual tracks.The expanded data highlights distribution sensitivities across sonar deployments.Source: §7 · p. 10
- The authors argue that evaluation frameworks, like methods, must be carefully designed, tested, and improved for progress to translate to real needs.They describe CFC26 as an important step in the machine-learning deployment lifecycle.Source: §7 · p. 10
A.1 Model Selection and Experimental Setup
The experiments select detectors using validation mAP50 while emphasizing counting metrics as the more direct measure of deployment performance. A transformer-based detector shows similar overall detection and counting performance to YOLOv5m.
Detector Architecture Selection
- Four YOLO variants were trained for two epochs, with the final detector selected by validation-set mAP50 after performance quickly plateaued.The variants were YOLOv5s, YOLOv5m, YOLO26s, and YOLO26m.Source: §A.1 · p. 14
- mAP50 is used for efficient architecture selection, but tracking errors, fragmented tracks, and duplicate identities can separate frame-level detection quality from counting accuracy.The mAP50 ranking is generally consistent with the counting-metric rankings.Source: §A.1 · p. 14
Detector Architecture Robustness
- YOLOv5m and RT-DETRv2-S achieve test mAP50 of 0.755 and 0.748, nMAE of 0.080 and 0.091, and nearly identical nMANE of −0.051 and −0.050.The authors report similar overall detection and counting performance despite the architectural difference.Source: §A.1 · p. 14, Table 3 · p. 15
Tracking and Inference Configuration
- ByteTrack counting performance varied little across tested hyperparameters, with average nMAE changing by 0.93% for min hits, 1.39% for IoU threshold, and 4.99% for max age.The reported directional biases were robust to reasonable ByteTrack hyperparameter variations, although alternative trackers were not evaluated.Source: §A.1 · p. 15
Target-River Training
- Target-river training monitors per-location validation mAP50 at intervals of 10% of an epoch and selects the checkpoint with the best aggregate validation mAP50.Runs that plateau or saturate early are manually terminated.Source: §A.1 · p. 15, Table 4 · p. 15
A.2 Sampling Strategy Selection
The sampling comparison favors the 10k-frame upsampling configuration for counting, even though the no-upsampling configuration achieves higher mAP50. The selected procedure uses the same balanced sampling design for ID and OOD experiments.
Sampling Strategy Selection
- The selected balanced-sampling procedure uses 10k training frames and 3k validation frames per location, with upsampling for locations below the training target.The budgets were based on the median number of fish-containing frames per location; a 1k-frame variant stalled without validation-loss improvement.Source: §A.2 · p. 15
- ID and OOD experiments use the same sampling procedure and fixed random seed 42; only ID training includes the target river.Source: §A.2 · p. 15
- 10k no-upsampling achieves higher mAP50 (0.77) than 10k upsampling (0.74), but upsampling produces lower counting errors across locations.The study selects upsampling because counting accuracy, rather than mAP50, is the operative evaluation criterion.Source: §A.2 · pp. 15–16, Table 5 · p. 16
A.3 Additional Training Conditions
The additional training conditions compare performance with full multi-location coverage, training restricted to CFC22 locations, or training from one Kenai deployment. The Kenai-only condition also tests transfer across banks within the same river.
All Locations
- All Locations combines training and validation splits from every location, establishing an upper bound when all deployment locations contribute training data.Source: §A.3 · p. 16
Kenai Leftbank Only
- Kenai Leftbank Only trains on Kenai Leftbank and evaluates both there and on Kenai Rightbank to test within-river cross-bank generalization.Source: §A.3 · p. 16, Table 6 · p. 16
CFC22 Locations
- CFC22 Locations restricts training to Elwha, all Kenai deployments, and Nushagak using the updated CFC26 splits.Source: §A.3 · p. 16
A.4 Additional Generalization Results
Generalization varies by river: training restricted to CFC22 locations can help at a geographically similar site but fails at more distinct rivers, while target data and broader training diversity have different benefits across locations. CFC26 adds broader deployment coverage and protocols, and its counting metrics distinguish error types relevant to practitioner decisions.
A.4 Additional Generalization Results
- At Chemainus, Target River Only performs worse than Balanced Sampling (OOD), and both are less accurate than Balanced Sampling (ID).The authors report that location-specific data alone may be insufficient when sparse, while combining target data with multi-location training performs best.Source: §A.4 · p. 17
- At Nushagak, Kenai-Leftbank Only yields directional error of approximately −0.65 despite low error under most conditions.The result indicates that shared Alaskan geography alone is insufficient for directional generalization.Source: §A.4 · p. 17
A.5 CFC22 vs CFC26
- CFC26 expands CFC22 to 2.5x as many clips, 2x as many frames, 2x as many bounding boxes, and six additional deployment locations.It also introduces chronological non-overlapping splits, held-out-river evaluation, directional metrics nMNE and nMANE, and a standardized benchmark.Source: §A.5 · p. 17
- Synthetic data generation and augmentation for sparse, far-range, or low-SNR fish are proposed as future directions but are beyond the scope of this benchmark and dataset paper.Source: §A.5 · p. 17
A.6 Metric Interpretation and Practitioner Workflow
- nMAE of 0 means every fish crossing was counted correctly, while nMAE of 1 means total counting errors equal the number of ground-truth crossings.The errors at nMAE 1 may be missed or hallucinated crossings.Source: §A.6 · p. 17
- nMNE of 0 indicates perfect net-passage estimation, while −1 indicates all true crossings were missed and +1 indicates hallucinated counts equal ground-truth crossings.Source: §A.6 · p. 17
- nMANE measures undercounting relative to each clip's dominant flow, whereas nMNE measures it relative to the dominant direction across all clips.nMNE suits net-passage priorities; nMAE better reflects individual-passage priorities, while nMANE helps assess systematic under- or over-counting, especially when directions vary.Source: §A.6 · p. 17
A.4 Additional Generalization Results
- At Dungeness, CFC22 Locations achieves nMAE of approximately 0.20 versus approximately 0.80 for Balanced Sampling (OOD) and Kenai-Leftbank Only, while CFC22 Locations reaches approximately 1.0 at Eel and 0.73 at Mad.The results suggest the narrower training set contains relevant regional signal for Dungeness but is unreliable for ecologically distinct rivers.Source: §A.4 · p. 16, Figure 5 · p. 18