Source-linked AI summary
Towards Visual Re-Identification of Fish using Fine-Grained Classification for Electronic Monitoring in Fisheries
Samitha Nuwan Thilakarathna, Ercan Avsar, Martin Mathias Nielsen, Malte Pedersen
TL;DR
Electronic Monitoring video exceeds manual-review capacity, and occlusion can make individual fish difficult to track across appearances. The paper evaluates Swin-T and ResNet-50 for fish re-identification in a controlled setting, finding viewpoint consistency more important than feature completeness.
Problem
EM video volume exceeds manual-review capacity, while occlusion and changing views can cause identity loss; aquatic Re-ID remains sparse for visually similar fish.
Source: §1 · pp. 1–2Method
The paper evaluates Swin-T and ResNet-50 for fish Re-ID in a controlled, closed-world setting, using custom resize-and-pad transformations and dataset-specific normalization.
Source: §2.2 · pp. 3–4, §2 · p. 3Results
Viewpoint consistency mattered more than feature completeness, and non-occluded queries retrieved targets across occlusion states; errors were almost exclusively intra-species.
Source: §4 · p. 8Takeaways & Limitations
The authors recommend prioritizing hard-negative examples of visually similar fish under varied viewpoints and acquiring high-quality queries or best shots.
Source: §4 · p. 8
Abstract
from arXiv · showhide
Accurate fisheries data are crucial for effective and sustainable marine resource management. With the recent adoption of Electronic Monitoring (EM) systems, more video data is now being collected than can be feasibly reviewed manually. This paper addresses this challenge by developing an optimized deep learning pipeline for automated fish re-identification (Re-ID) using the novel AutoFish dataset, which simulates EM systems with conveyor belts with six similarly looking fish species. We demonstrate that key Re-ID metrics (R1 and mAP@k) are substantially improved by using hard triplet mining in conjunction with a custom image transformation pipeline that includes dataset-specific normalization. By employing these strategies, we demonstrate that the Vision Transformer-based Swin-T architecture consistently outperforms the Convolutional Neural Network-based ResNet-50, achieving peak performance of 41.65% mAP@k and 90.43% Rank-1 accuracy. An in-depth analysis reveals that the primary challenge is distinguishing visually similar individuals of the same species (Intra-species errors), where viewpoint inconsistency proves significantly more detrimental than partial occlusion. The source code and documentation are available at: https://github.com/msamdk/Fish_Re_Identification.git
1 Introduction
Electronic Monitoring video volume exceeds manual-review capacity, while occlusion and changing views make individual-fish tracking difficult. The paper evaluates deep-learning approaches for fish re-identification and identifies methodological choices and failure modes for this task.
1 Introduction
- EM video volume exceeds manual-review capacity, with poor image quality, occlusion, and varied recording conditions compounding the bottleneck.Conveyor-belt fish may be occluded or move out of view, making it difficult to tell whether an individual is appearing for the first or second time.Source: §1 · p. 1
- Aquatic Re-ID remains sparse, and prior aquatic work has focused mainly on species with distinct visual patterns rather than similar-looking commercial fish.The paper describes its work as the first in-depth study of re-identifying similarly looking commercial fish species.Source: §1 · p. 2
- The study evaluates fish Re-ID on AutoFish in a simulated conveyor-belt setting, using ground-truth annotations to isolate feature extraction from upstream detection errors.The annotations establish a theoretical performance ceiling for the Re-ID component.Source: §1 · p. 2
- The paper compares Swin-T with ResNet-50 to assess the architectural advantage of self-attention for fine-grained fish Re-ID.Both backbones are modified to produce shared 512-dimensional embeddings for metric learning.Source: §1 · p. 2, Figure 3 · p. 3
- The authors identify custom image transforms and hard triplet mining as methodological strategies that improve Re-ID performance.The introduction also flags analysis of Swin Transformer's failure modes in partially occluded scenarios.Source: §1 · p. 2
2 Methodology
The methodology evaluates fish re-identification across visually similar commercial species, comparing CNN and Transformer architectures in a closed-world setting. It combines fish-preserving preprocessing and hard triplet mining with controlled tests of viewpoint and occlusion robustness.
2 Methodology
- Aquatic Re-ID research has focused mainly on species with distinctive patterns, while this study evaluates ResNet-50 and Swin-T for multiple visually similar commercial fish in a closed-world setting.The authors note that localized markers can be obscured in EM scenarios.Source: §2 · p. 3
2.1 Dataset and Data preparation
- AutoFish contains 1,500 images of 454 fish across six North Sea species and a miscellaneous category, with nonoverlapping fish IDs across the train, validation, and test splits.Each fish has 40 images across four arrangement-and-viewpoint conditions; crops use instance masks with two-pixel padding.Source: §2.1 · p. 3
2.2 Model Architectures and Preprocessing
- Both architectures were adapted with a shared 512-dimensional embedding head, and fish crops were resized without changing aspect ratio and padded to 224×224.The preprocessing also applies dataset-specific channel normalization.Source: §2.2 · pp. 3–4
2.3 Training
- Training used triplet margin loss with hard-positive and hard-negative mining; preliminary experiments favored hard over semi-hard mining alongside the custom transforms.A PK sampler ensured batches contained multiple fish IDs and instances per ID.Source: §2.3 · p. 4, §2.4 · p. 4
2.4 Experimental Design
- Experiments compared batch sizes 16, 32, 64, and 256, then tested the selected model under identical, viewpoint-shift, occlusion, and combined-challenge conditions.The combined challenge pairs viewpoint change with occlusion.Source: §2.4 · p. 4
2.5 Evaluation
- Evaluation ranked gallery images by Euclidean distance between L2-normalized embeddings and reported R1 accuracy and mAP@k, with k=39 true matches per query.One image per fish ID was randomly selected as the query, with the remaining instances forming the gallery.Source: §2.5 · pp. 4–5
3 Results
Swin-T achieved the strongest fine-grained fish Re-ID results, while analysis of the best model found viewpoint changes more damaging than partial occlusion. Errors were almost exclusively between individuals of the same species.
3.2 Model architecture comparison for Re-ID
- 90.43% R1 accuracy and 41.65% mAP@k were Swin-T’s peak results, versus ResNet-50’s peaks of 70.21% R1 and 13.56% mAP@k.Swin-T peaked at batch size 16 and ResNet-50 at batch size 256.Source: Figure 5 · p. 6, §3.2 · p. 6
3.1 Preliminary experiments and Baseline
- 3.19% R1 and 0.27% mAP@k were Swin-T’s zero-shot results; hard mining beat semi-hard, and custom transforms improved Swin-T retrieval, motivating their adoption for main experiments.Custom transforms slightly decreased ResNet-50 R1 under hard mining, but improved its mAP@k.Source: §3.1 · pp. 5–6
3.2 Model architecture comparison for Re-ID
- Swin-T embeddings formed separated identity clusters with minimal positive-negative distance overlap, whereas ResNet-50 embeddings overlapped and were poorly clustered.The analysis used test-set KDE plots and t-SNE visualizations.Source: §3.2 · p. 6, Figure 6 · p. 7
3.3 In-Depth Analysis
- 95.42% R1 and 78.48% mAP@k under partial occlusion exceeded 78.59% R1 and 69.24% mAP@k for an unoccluded opposite-side view; the combined challenge performed worst.The analysis reports that viewpoint consistency was the critical limiting factor across query subsets.Source: §3.3 · pp. 6–7, Table 3 · p. 7
- Rank-1 errors were almost exclusively intra-species confusions, with inter-species errors nearly absent.The error analysis identifies visually similar individuals of the same species as the main Re-ID challenge.Source: Table 4 · p. 7, §3.3 · p. 7
- The non-occluded Separated-Initial query achieved 78.48% mAP@k against a Touched-Initial gallery, above the 77.59% for a Touched-Initial query against itself.The same pattern recurred in the flipped scenario, where the reported mAP@k values were 79.87% and 77.64%, respectively.Source: §3.3 · p. 7
4 Discussion and Conclusion
The discussion attributes Swin-T’s performance to its global-context modeling and small-batch robustness, while noting a fixed-margin caveat for noisy real-world data. It also emphasizes viewpoint consistency and high-quality queries for fish Re-ID.
4 Discussion and Conclusion
- Swin-T’s shifted-window attention captures global context and subtle distributed markers, while its layer normalization supports small-batch performance where ResNet-50 batch normalization may become unstable.The discussion links the transformer’s advantage to global information propagation and sample-wise normalization.Source: §4 · p. 8
- A fixed hard triplet margin of 0.5 worked for clean laboratory data, but may hinder convergence in noisy fishing environments with blood or debris.The authors propose testing margins for noisy domains or exploring adaptive margins.Source: §4 · p. 8
- Viewpoint consistency mattered more than feature completeness: preserving the lateral view supported matching under occlusion, while flipping removed access to asymmetric features.The discussion also reports that a non-occluded query retrieved targets in occluded scenarios more successfully than an occluded query did against a similarly occluded gallery.Source: §4 · p. 8
- Because errors were almost exclusively intra-species, the authors recommend collecting hard negatives—visually similar same-species fish across varied viewpoints—and prioritizing high-quality queries.They connect reliable re-identification to catch data used in stock assessments, selectivity studies, and gear development.Source: §4 · p. 8