FiVOS: A Fish Segmentation Algorithm Based on Interactive Video Object Segmentation and Filter Enhancement

Introduction

• The study presents interactive fish video object segmentation tailored to aquaculture to improve fish-characterization extraction and support behavior and health analysis.The method is intended to improve extraction accuracy and efficiency and support intelligent aquaculture management.Source: §1 · p. 3
• Background subtraction and optical flow have difficulty capturing fish motion information along the depth axis, which affects the accuracy of behavior.Pedersen et al. developed an RGB approach for comprehensive capture of fish swimming information; this method is prone to target loss during detection and tracking.Source: §1 · p. 3
• Interactive video object segmentation uses simple user interactions, including points, scribbles, or boxes, to segment specific objects at low annotation cost.Earlier approaches [25] reran forward computations at each interaction round, while MA-Net computed all-frame embeddings initially and used shallow branches in later rounds.Source: §1 · pp. 4–5
• Current fish behavior methods underuse dynamic correlations between frames and lack individual-object focus, contributing to target confusion and feature loss in multi-object scenes.Interactive video object segmentation can focus on specific objects for more targeted tracking.Source: §1 · p. 5
• Interactive video segmentation methods that perform well on public datasets face fish-farming data scarcity, water-flow and lighting interference, and limited accuracy after repeated correction.Concentrating fish of the same species and growth stage also creates high intra-class similarity, making individuals difficult to distinguish during propagation.Source: §1 · p. 5

• Long-sequence propagation can lose target masks, and the proposed filtering scheme is intended to address errors in high-similarity and challenging settings.Source: §1 · pp. 5, 7
• FiVOS training uses two fish-specific datasets: static images of 1,350 fish instances and 23 pseudo-dynamic sequences of 10 consecutive frames each, all with pixel-level masks.The sequences capture fish motion characteristics.Source: §1 · p. 6
• A serial mask-block and noise filtering scheme detects and corrects erroneous propagated masks, using median filtering and a rule-based distance threshold.The scheme refines dynamically generated coarse masks and is intended to improve performance under high intra-class similarity and challenging conditions.Source: §1 · p. 6
• FiVOS achieves state-of-the-art performance on fish videos and is presented as a tool for fish behavior feature analysis.Source: §1 · p. 6

2.1 Dataset acquisition

• Videos covering the tank surface were recorded and processed into both datasets; frames were extracted for pixel-level manual fish-contour annotation.Source: Figure 2 · p. 8, §2.1 · p. 9
• Water averaged 21.85°C ± 1.04°C, dissolved oxygen 9.13 ± 1.21 mg/L, and pH 7.44 ± 0.18; the reported parameters were within suitable ranges.Source: §2.1 · p. 8
• Two fish-farming datasets, Fish-static and Fish-DAVIS, were created to address data scarcity, using adult rainbow trout as the observation species.Source: §2.1 · pp. 7, 9
• Fish-static contains 225 annotated images, representing multiple fish in an original image as separate instances with one target mask each.Source: §2.1 · p. 9
• Fish-DAVIS is an interactive-video dataset with sequences sampled every five frames and complete data sequences constructed every ten frames.Image data are paired with ground-truth instance masks and three randomly selected single-frame scribble annotation files.Source: §2.1 · pp. 9–10

2.2.1 FiVOS Overview

• FiVOS uses an interaction–propagation–fusion workflow: users refine a selected frame with S2M, propagate its mask bidirectionally, and fuse predictions with prior-iteration results.The interaction module provides real-time feedback before propagation; the mask-block filter is applied to predictions with multiple blocks, while single-block predictions bypass filtering.Source: §2.2.1 · pp. 10–11

2.2.2 Propagation Module

• To address STM error accumulation under high intra-class similarity, the propagation module filters unreasonable mask blocks and residual noise before subsequent propagation.The mask-block filter can also be integrated into various SVOS models.Source: §2.2.2 · p. 12

2.2.3 Space-time memory reader

• The STM reader predicts a query-frame mask by matching query features to labeled memory frames, retaining top-k similarities, and decoding the resulting features.Memory frames encode images and object masks, while query frames encode images only.Source: §2.2.3 · p. 13

2.2.4 Mask block filter

• The mask-block filter matches predicted blocks to adjacent-frame blocks by centroid distance and removes blocks exceeding a minimum-distance threshold.It uses centroid movement to suppress distant erroneous blocks, helping prevent errors from spreading into later propagation steps.Source: §2.2.4 · pp. 13–14

2.2.5 Rule-Based Thresholding Method

• The rule-based threshold sets DT from the maximum centroid movement between adjacent frames and uses it to identify erroneous detections and stabilize tracking.The threshold is a heuristic based on prior knowledge and can be estimated from mask information in the dataset.Source: §2.2.5 · pp. 14–15
• The rule-based method computes mask centroids and uses the distance between adjacent-frame centroids, with DT = 25.7 selected for Fish-DAVIS.The centroids are calculated from the target binary mask using first-order moments.Source: §2.2.5 · pp. 15–16

2.2.6 Noise filter

• The noise filter applies median filtering to remove residual small specks that may evade the mask block filter and impair later processing.Median filtering replaces a pixel with the median value in its local window.Source: §2.2.6 · p. 16
• A 3 × 3 median-filter window achieved optimal performance, while 1 × 1 left noise and 21 × 21 over-smoothed masks.The 3 × 3 window suppressed noise while preserving important details.Source: §2.2.6 · pp. 17–18, Table 3 · p. 17

3.1 Experimental confguration

• The training workflow includes three stages, and Mask Propagation training varies the maximum distance between frames from 5 to 10 before annealing it back to 5.The distance schedule is described as curriculum learning.Source: §3.1 · p. 18
• The Scribble To Mask module is trained independently on Fish-static using pretrained DeepLabV3+ ResNet-50 weights.The workflow then trains Mask Propagation on Fish-static before continuing on Fish-DAVIS with the saved weights.Source: §3.1 · p. 18
• The Fusion module takes output from the trained Mask Propagation module, while Fish-DAVIS video sequences provide its training loss.Three modules’ weights are ultimately used for validation.Source: §3.1 · p. 19
• The paper reports its experimental system and hardware configuration and model parameter settings in Tables 4 and 5.The supplied table captions identify these topics but provide no configuration values.Source: Table 4 · p. 19, Table 5 · p. 20

3.2 Evaluation metrics

• The study uses J&F and AUC-J&F to evaluate segmentation accuracy, edge consistency, and performance across different interaction counts.The evaluation also emphasizes that accuracy and interaction efficiency affect user experience.Source: §3.2 · p. 21
• J&F combines the Jaccard Index (J) and edge accuracy (F) to assess region accuracy and edge consistency.J measures intersection over union with the ground-truth mask; F assesses edge consistency.Source: §3.2 · p. 21
• J&F reflects both segmentation accuracy and edge quality, with higher values indicating stronger region accuracy and boundary consistency.A higher J&F also indicates more precise segmentation contours.Source: §3.2 · pp. 21–22
• AUC-J&F measures segmentation performance across interaction steps, with higher values indicating stable segmentation with fewer interactions.The metric is the area under the performance-versus-interactions curve.Source: §3.2 · p. 22

3.3 Comparison with other advanced tracking algorithms

• Automated evaluation begins with a scribble on a selected frame and repeats correction on the worst-performing candidate frame for at most eight rounds.The private Fish-DAVIS dataset required modifying the official API configuration for this process.Source: §3.3 · p. 22
• On the Fish-DAVIS interactive validation set’s 10-frame short video, FiVOS surpassed all competitors across various metrics; MiVOS and STCN exhibited no erroneous mask blocks, and no significant improvement was observed compared with FiVOS.Source: Table 6 · p. 23, §3.3 · p. 23
• FiVOS significantly outperformed comparison methods in the 80-frame long-sequence test, with enhanced robustness and segmentation accuracy.The authors merged Fish-DAVIS into an 80-frame sequence for this evaluation.Source: Table 7 · p. 23, §3.3 · p. 23
• Short-sequence qualitative results were comparable for three methods, while MANet, XMem, and RGMP were exceptions to that comparison.The figure caption notes target confusion for MANet and minor differences among other methods.Source: §3.3 · p. 24, Figure 6 · p. 24
• On long qualitative sequences, MiVOS lost the target at frame 99 and STCN made a recognition error at frame 117, while FiVOS remained stable.The paper identifies a 20-second, 1200-frame video for real-time human interaction testing.Source: §3.3 · pp. 23–24

3.4.2 Noise filter

• 17.6% accuracy improvement over the baseline was reported for the noise filter on Fish-DAVIS long-video validation.The short-video improvement was 0.604% over the baseline.Source: Table 9 · p. 26, §3.4.2 · p. 27

3.4.3 Rule-Based Thresholding Method

• 9.4% accuracy improvement over manually set thresholds was reported for rule-based thresholding on Fish-DAVIS long-video validation.The short-video improvement over manually set thresholds was 0.04%.Source: Table 9 · p. 26, §3.4.3 · p. 28

3.4.1 Mask block filter

• In the Fish-DAVIS long-video ablation (80 frames), validation accuracy improved by 4.6% over the baseline after adding the mask block filter.Source: §3.4.1 · pp. 25–26, Table 9 · p. 26
• Compared with baseline propagation recognition errors, the mask block filter retained high recognition accuracy past frame 75, completely preventing recognition errors, containing further error spread, and enhancing model stability.It also corrected propagation errors in longer video sequences.Source: §3.4.1 · pp. 26–27, Figure 8 · p. 27

3.4.2 Noise filter

• The noise filter removes small erroneous particles left after the mask block filter, which otherwise can accumulate during propagation.The authors report that noise filtering prevents further error propagation.Source: §3.4.2 · pp. 27–28, Figure 9 · p. 28

3.4.3 Rule-Based Thresholding Method

• Rule-based thresholding filtered erroneous masks at frames 79 and 146, whereas the manually defined mask-filter errors reappeared after frame 146.The paper describes the rule-based distance threshold as more precise and reports that it improved stability and reliability in long-sequence validation.Source: Figure 10 · p. 29, §3.4.3 · p. 29

3.4.4 Generalization Verification

• On DAVIS, incorporating the filtering scheme decreased accuracy by 0.529% and 0.792%, while the authors report the effect as minimal and statistically insignificant.They attribute the limited filter effect to high target-background contrast and low target similarity; the noise filter may also remove a small amount of mask information.Source: §3.4.4 · p. 29, Table 10 · p. 30

4.1 Robustness of the Method

• FiVOS showed stable segmentation without target confusion in real-farming footage, while MiVOS exhibited target confusion and loss.The dataset included 150 farmed fish; the figure presents MiVOS on the left and FiVOS on the right.Source: §4.1 · p. 30, Figure 11 · p. 31
• In underwater tests, FiVOS maintained segmentation performance despite partial occlusion at frames 0, 25, and 51.The authors report robustness in aquaculture and underwater scenarios, including cluttered backgrounds and fish occlusions.Source: Figure 12 · p. 31, §4.1 · p. 31

4.2 Limitations

• After occlusion-induced disappearance, later interactions removed an erroneous mask block but still produced suboptimal recognition of the correct target.The system increasingly relies on the fusion module in later propagation, making mask synthesis and correction more complex and limiting filter-based removal of unwanted blocks.Source: §4.2 · pp. 31–32, Figure 13 · p. 32
• For small targets, masks showed noticeable detail loss and recognition instability after propagation at 0.8% image area; at 0.2% image area, the target was completely lost.Source: §4.2 · p. 32

4.3 Application Prospects

• The trajectory visualization connects each frame’s mask centroid in temporal order to display fish swimming paths.The plotted points are 0.03 seconds apart; the paper describes their spacing and distribution as cues for movement state and behavior.Source: §4.3 · pp. 32–34, Figure 15 · p. 34

5 Conclusions

• FiVOS is proposed to address the difficulty of distinguishing fish with high intra-class similarity in aquaculture video segmentation.The conclusion describes the method as an interactive video object segmentation approach.Source: §5 · p. 34
• The mask block filter uses adjacent-frame mask-centroid distances and a precomputed maximum-distance threshold to filter erroneous masks early.The conclusion links this approach to misidentification and accumulated recognition errors in the STM model.Source: §5 · pp. 34–35
• Experiments report improved fish-aquaculture performance and long-term recognition accuracy and robustness, with performance comparable to the original model on the public dataset.A noise filter is added after the mask block filter to reduce the impact of minor noise.Source: §5 · p. 35
• Future work targets better filter collaboration with the fusion module, broader fish-species and color diversity, and improved small-target segmentation.The proposed directions include multi-scale feature fusion and high-resolution input pipelines.Source: §5 · p. 35