Source-linked AI summary
Temperate Fish Detection and Classification: a Deep Learning based Approach
Kristian Muri Knausgård, Arne Wiklund, Tonje Knutsen Sørdalen, Kim Halvorsen, Alf Ring Kleiven, Lei Jiao, Morten Goodwin
TL;DR
The paper addresses the labor-intensive analysis of underwater imagery and the difficulty of classifying fish in noisy, variable natural environments. It proposes a two-step pipeline that detects fish with YOLOv3 and classifies them with CNN-SENet, using transfer learning from ImageNet and Fish4Knowledge. The system reaches 86.96% mean average precision for detection, 99.27% accuracy on Fish4Knowledge, and 83.68% average accuracy for temperate fish.
Problem
Underwater recordings are abundant but labor-intensive to process, while natural environments introduce noise and variability in backgrounds, visibility, depth, habitat, and fish orientation.
Method
The approach separately detects fish with YOLOv3 and classifies each detected fish with CNN-SENet, using transfer learning and no image pre-processing beyond resizing.
Results
86.96% mean average precision is reported for detection, alongside 99.27% accuracy on Fish4Knowledge and 83.68% average accuracy for temperate fish.
Takeaways & Limitations
The authors conclude that the approach can detect individual fish in complex environmental conditions and provide a useful tool for automatic fish-species analysis.
Takeaways & Limitations
The temperate-fish classification accuracy is constrained by the comparatively smaller dataset and high variation in image data.
Abstract
from arXiv · showhide
A wide range of applications in marine ecology extensively uses underwater cameras. Still, to efficiently process the vast amount of data generated, we need to develop tools that can automatically detect and recognize species captured on film. Classifying fish species from videos and images in natural environments can be challenging because of noise and variation in illumination and the surrounding habitat. In this paper, we propose a two-step deep learning approach for the detection and classification of temperate fishes without pre-filtering. The first step is to detect each single fish in an image, independent of species and sex. For this purpose, we employ the You Only Look Once (YOLO) object detection technique. In the second step, we adopt a Convolutional Neural Network (CNN) with the Squeeze-and-Excitation (SE) architecture for classifying each fish in the image without pre-filtering. We apply transfer learning to overcome the limited training samples of temperate fishes and to improve the accuracy of the classification. This is done by training the object detection model with ImageNet and the fish classifier via a public dataset (Fish4Knowledge), whereupon both the object detection and classifier are updated with temperate fishes of interest. The weights obtained from pre-training are applied to post-training as a priori. Our solution achieves the state-of-the-art accuracy of 99.27\% on the pre-training. The percentage values for accuracy on the post-training are good; 83.68\% and 87.74\% with and without image augmentation, respectively, indicating that the solution is viable with a more extensive dataset.
1 Introduction
Underwater cameras generate valuable but labor-intensive ecological recordings, while natural environments make fish classification difficult because of complex backgrounds, turbidity, and changing light. The paper proposes a separate deep-learning pipeline using YOLOv3 for detection and CNN-SENet for temperate fish classification, supported by transfer learning and no image pre-processing beyond resizing.
- Motivation: Manual processing limits how much underwater imagery can be analyzed despite the large volumes collected by high-resolution cameras.This constrains the ecological value that can be extracted from recordings of remote areas and fish behavior.
- Challenges: Natural underwater classification is challenged by background complexity, turbidity, and light propagation as depth increases.Controlled-environment methods may achieve high accuracy but do not readily transfer to wild settings.
- Prior Work: Prior natural-environment detection and classification approaches included Fast R-CNN with a mean average precision of 81.4% and another approach reporting 85.08% accuracy on 1420 test images.These methods used pre-training and, in some cases, image de-noising or other processing steps.
- Prior Work: Previous Fish4Knowledge work reported 98.64% accuracy using image filters, fish-shape extraction, background removal, and CNN-SVM classification.Earlier solutions commonly pre-processed images to reduce noise or outline fish contours.
- Contribution: The proposed method separately detects fish with YOLOv3 and classifies them with a CNN using the Squeeze-and-Excitation architecture.The approach targets temperate fish in underwater images and videos captured in natural environments.
- Contribution: Transfer learning uses Fish4Knowledge to learn generic fish features before further training on a newly collected temperate-fish dataset.The method requires no image pre-processing beyond resizing and was presented as an application not previously used for temperate fish detection and classification.
2 Datasets and Deep Learning Approaches
The approach separates fish detection from species classification, using YOLOv3 to detect fish and a CNN-SENet component to classify cropped fish. It combines independent training datasets and transfer learning with temperate-species data collected in natural, variable underwater conditions.
- Deep Learning Approaches: YOLOv3 detects fish in each video frame and passes rectangular fish subframes to a CNN-SENet classifier.The two components are trained independently, separating detection from species categorization.
- Deep Learning Approaches: YOLOv3 was selected because it offers a speed/accuracy tradeoff suitable for real-time video detection.The object detector processes a live video stream and outputs fish objects without requiring species classification at that stage.
- Deep Learning Approaches: The detection model uses ImageNet initialization and a temperate dataset containing 619 images with 1,943 annotated fish.The dataset was designed around shallow-water fish assemblages along Southern Norway’s coast.
- Deep Learning Approaches: Detection ignores species labels, while species information is used in the independent classification stage.Only a fraction of Cod images is shared between detection and classification training, making the datasets nearly non-overlapping.
- Datasets: Fish4Knowledge provides classification pre-training, while a novel Southern Norway temperate dataset supports post-training.The Fish4Knowledge dataset contains 27,230 images across 23 species and is highly imbalanced, with species counts ranging from 25 to 12,112 images.
- Datasets: The temperate dataset combines GoPro recordings from three Norwegian locations and represents abundant Northern European coastal fish species.Its images vary in depth, visibility, habitat, fish orientation, and camera distance, while frames often contain multiple fish.
3 Object Detection and Classification
The paper separates species-independent fish detection from species classification, using YOLOv3 for detection and CNN-SENet for classifying detected fish. The classifier uses squeeze-and-excitation features, while transfer-learning and tuned training settings support classification performance.
- Fish Detection: YOLOv3 detects fish independently of species recognition, separating object localization from subsequent classification.The separation allows distinct training data and more fine-grained species classification.
- Species Classification: CNN-SENet classifies detected fish species using a convolutional network with an added squeeze-and-excitation architecture.The SE block adaptively recalibrates channel-wise feature responses.
- Species Classification: The SE block combines squeeze-based global average pooling with excitation-based feature recalibration before subsequent network processing.This design uses contextual information outside the receptive field and recalibrates channel responses adaptively.
- Species Classification: The network uses three fully connected layers, dropout, batch normalization, and a final softmax classifier for species prediction.The final fully connected layer has C neurons corresponding to the number of fish species.
- Species Classification: 99.27% testing accuracy with batch normalization exceeded 98.35% without batch normalization in the reported configuration.The study used 50% dropout and a learning rate of 0.001 without decay after parameter trials.
4 Experiments, Results and Discussion
The experiments evaluate the proposed system in two stages: fish detection first and fish classification second.
- Experimental Design: The proposed approach was verified through separate experiments for fish detection and fish classification.Detection performance was assessed before classification performance.
4.1 Fish Detection
YOLOv3 localizes individual fish in underwater video frames using IoU-based detection evaluation. It achieves strong average overlap and performs across substantially different live-stream conditions, though some fish and seaweed are misdetected.
- Detection Method: Individual fish are localized in video frames with a YOLOv3-based object detector and evaluated using Intersection over Union.IoU is the Jaccard-index measure of overlap used for object detection and segmentation verification.
- Detection Results: 0.6802 average IoU and 0.9934 per-class IoU were reported for fish detection.The per-class value indicates that only a tiny percentage of background objects were erroneously detected as fish.
- Detection Results: 86.96% peak precision was obtained during detection evaluation on a dataset split into 70% training and 30% verification data.The IoU, training loss, and mean average precision were tracked during training.
- Detection Results: The detector correctly handled live-stream footage with variable lighting and unseen camera angles, but ignored some corner fish and detected seaweed as fish in one case.The live stream came from an underwater camera near a semi-submerged restaurant in southern Norway.
4.2 Species Classification
Species classification compares CNN-SENet with established networks across Fish4Knowledge pre-training and temperate-fish post-training experiments. The experiments use held-out data and examine augmentation, training setup, and transfer-learning initialization.
- Experimental Design: Classification performance is compared between CNN-SENet, Inception-V3, ResNet-50, Inception-ResNet-V2, and a CNN-SENet variant without SE blocks.The ablation examines how removing SE blocks affects the relationship between fish-image colors and other feature layers.
- Experimental Design: Three experiments cover Fish4Knowledge pre-training, temperate-fish post-training, and post-training with image augmentation.Each relevant dataset was divided into training, validation, and testing subsets.
- Experimental Design: The benchmarked networks were trained for 50 epochs using 200 × 200 RGB images, except Inception-ResNet-V2, which used 299 × 299 RGB images.Testing images were kept outside the training process for independent verification.
- Pre-training: Fish4Knowledge pre-training used 19149 training images, 4126 verification images, and 4126 testing images, with evaluation based on the highest validation-accuracy epoch.The selected configuration used 50 epochs and batch size 16.
- Post-training: Temperate-fish post-training used 712 training images, 155 verification images, and 155 testing images across four fish classes.The process used 50 epochs and batch size 8, loading pre-training weights before post-training.
- Post-training: Transfer learning starts post-training from generic fish-identification features learned on the larger Fish4Knowledge dataset.The final fully connected layer is changed from 23 output neurons to four for the temperate dataset.
- Data Augmentation: Image augmentation expands the dataset with rotations, shifts, scaling, shearing, and horizontal flips to expose the network to transformed images.These operations are intended to preserve labels while increasing training variation.
4.3 Results
Pre-training on Fish4Knowledge achieved accuracy comparable to or exceeding prior state-of-the-art results, while post-training on temperate fish was more challenging. Image augmentation improved post-training accuracy across the benchmarked algorithms.
- The pre-training testing accuracy was on par with or exceeded previous state-of-the-art solutions.
- 99.15% test accuracy was achieved by CNN-SENet with Squeeze-and-Excitation, nearly matching Inception-V3.CNN-SENet required roughly one-third of Inception-V3's epoch runtime, while its non-SE version was faster but slightly less accurate.
- Inception-ResNet-V2 had the lowest test accuracy and longest training time, partly associated with its higher 299×299 input resolution.The other tested networks used 200×200 inputs, and upscaling may have negatively affected accuracy.
- The post-training experiment was more challenging than pre-training, with limited data, low visibility, and unclear fish silhouettes.These conditions characterize the temperate species dataset used for post-training.
- 85.42% was the highest average testing accuracy without image augmentation, achieved by Inception-V3.CNN-SENet performed a few percentage points lower but had significantly better training time per epoch.
- Image augmentation improved post-training results for all benchmarked algorithms, while CNN-SENet accuracy increased by approximately four percentage points.Training time per epoch did not change notably with augmentation.
5 Conclusions
The proposed system combines YOLOv3 detection with CNN-SENet classification for automatic temperate fish analysis. It achieved strong Fish4Knowledge performance and lower but promising accuracy on the smaller, more variable temperate-fish dataset.
- YOLOv3 detects individual fish in complex environmental conditions, while CNN-SENet classifies their species.
- 86.96% mean average precision was achieved for object detection, and 99.27% accuracy was achieved on Fish4Knowledge without augmentation or preprocessing.
- 83.68% average accuracy was obtained for temperate fish classification.The paper attributes the lower accuracy to the smaller dataset and high variation in image data.
- The detection algorithm was successfully tested in real time on a live 25 FPS Full HD underwater video stream.