PaperlayerView exact revision on arXiv

Source-linked AI summary

Weakly Supervised Fish Segmentation through Point-Supervised Tracking and Motion-Informed Language

Vedanth Kallakuri, Katherine A. Skinner

arXiv:2610.29047v1cs.CV

TL;DR

Dense fish masks are costly to annotate, and underwater conditions make segmentation challenging. The paper combines point tracking, motion-informed language, and mask filtering; on DeepFish, it reports stronger foreground and overall mIoU than state-of-the-art baselines, with no human input at inference.

  • Problem

    Fish segmentation is difficult in noisy underwater scenes, while dense per-pixel training labels are costly and can take up to two minutes per fish to annotate.

    Source: §I · p. 1

  • Method

    The pipeline fine-tunes a point tracker, converts predicted trajectories into LLM-generated motion queries for LMPM++ mask generation, then filters masks using tracked points, background subtraction, and optical flow.

    Source: §I · p. 2

  • Results

    The pipeline surpasses state-of-the-art baselines in foreground and overall mIoU; SAM2 has higher background mIoU but requires ground-truth point annotations at inference.

    Source: 2) Quantitative Results Analysis · p. 5

  • Takeaways & Limitations

    On the FishSeg benchmark from DeepFish, the approach achieves performance competitive with state-of-the-art weakly supervised methods without human input at inference.

    Source: §VI · p. 6

    Computationally intensive optical flow, sequence processing, and per-segment LLM queries make the pipeline less suitable for real-time deployment without further optimization.

    Source: §V · p. 6

Abstract

from arXiv · show

Understanding the properties of fish populations is increasingly important. Analyzing fish length, width, and mass is integral to maintaining healthy stock in the aquaculture industry, while counting wild fish populations supports marine conservation and coastal reef restoration efforts. Deep learning-based segmentation models offer a promising means of obtaining such information at scale. Most of these models, however, require costly and time-consuming per-pixel segmentation labels for training, which can take up to two minutes per fish to annotate manually. To address this inefficiency, we propose a weakly supervised segmentation pipeline that requires only point-labeled data (one click per fish) to produce accurate segmentation masks. Our pipeline consists of four stages. First, a point-supervised tracker predicts point annotations for each frame. These points are then converted into motion-informed language expressions, which guide a language-conditioned segmentation model (LMPM++) to capture temporal consistency and produce candidate segmentation masks. Finally, optical flow and background subtraction are applied as post-processing filters to suppress noise from incorrect segmentations. Experiments on the DeepFish dataset show improvement in foreground mean intersection over union (mIoU) as compared to baseline models.

I. INTRODUCTION

Fish monitoring can support aquaculture and conservation, but underwater visual noise and costly dense labels make automated fish segmentation challenging. The paper proposes using point labels, tracking, and motion-informed language to guide mask generation.

I. INTRODUCTION

  • Underwater visual noise complicates fish segmentation, while dense per-pixel labels can take up to two minutes per fish to annotate.Fish measurements support aquaculture stock management, and wild-fish counts support conservation and reef restoration.Source: §I · p. 1
  • Weak supervision reduces annotation effort with one click per fish, but point labels provide less information than dense labels.The paper aims for performance comparable to full supervision.Source: §I · p. 1
  • The pipeline fine-tunes a point tracker on sparse labels, then uses predicted trajectories to generate motion-informed language queries for candidate mask generation.The queries describe motion and behavioral cues, including direction, speed, and fish class.Source: §I · pp. 1–2, Figure 1 · p. 1
  • Tracked points filter candidate masks, and background subtraction and optical flow further denoise them.Source: §I · p. 2
  • The contributions include point-label-only training without dense pixel annotations and LLM-generated motion descriptions as an additional inference signal.The paper evaluates the approach on the publicly available DeepFish dataset.Source: §I · p. 2

II. RELATED WORKS

Related work spans label-efficient fish segmentation and point tracking. The paper uses point trajectories to add motion and behavioral context to weakly supervised fish segmentation.

A. Fish Segmentation Methods

  • Existing fish-segmentation approaches include point-label-trained Affinity-LCFCN, prompted foundation models, and language-prompted LangSAM; the paper identifies underwater challenges as unresolved.Source: A. Fish Segmentation Methods · p. 2

B. Point Tracking

  • Point tracking predicts queried point locations across video frames and can also estimate visibility or occlusion, unlike optical flow’s dense pixel-displacement field.Source: B. Point Tracking · p. 2
  • CoTracker3 jointly tracks points using transformer attention across time and tracks, addressing independent-track limitations described for PIPs and TAPIR.Source: B. Point Tracking · p. 2
  • The pipeline fine-tunes CoTracker3 on underwater fish videos, then uses its predicted trajectories to generate motion descriptions and filter LMPM++ masks.Its pretrained synthetic data includes point tracks and occlusion labels.Source: B. Point Tracking · p. 2

C. LMPM++

LMPM++ is a motion-language video segmentation framework that generates candidate masks from language queries. This pipeline supplies it with LLM-generated descriptions of predicted fish motion, then filters and refines its candidate masks.

C. LMPM++

  • LMPM++ is a referring video object segmentation framework designed to handle motion-centric language expressions that cannot be resolved from one frame alone.It was proposed as a benchmark method for MeViS, which contains over 33,000 human-annotated motion expressions.Source: C. LMPM++ · p. 3
  • LMPM++ represents candidate objects as compact embeddings and reasons over temporal context; a temporal-level contrastive loss helps distinguish motion sequences with different orders.Source: C. LMPM++ · p. 3
  • The pipeline supplies LMPM++ with LLM-generated motion descriptions from predicted trajectories, and the model matches the query to video motion patterns to generate candidate masks.Tracked points filter candidates, followed by background subtraction and optical-flow refinement.Source: C. LMPM++ · p. 3

III. METHODOLOGY

  • The methodology has four steps: predict points, convert trajectories into language queries, generate and filter LMPM++ masks, then remove noise with optical flow and background subtraction.Source: §III · p. 3

A. Fine-Tuning CoTracker3

CoTracker3 is fine-tuned to track fish points through underwater video, including occlusions, using trajectory and visibility supervision. The training setup favors accurate point localization while retaining supervision for occlusion and reappearance.

A. Fine-Tuning CoTracker3

  • The offline CoTracker3 variant processes clips jointly in both temporal directions, supporting tracking through underwater occlusions before fish reappear.All network parameters except the visibility confidence head are fine-tuned to adapt the pretrained tracker to underwater fish videos.Source: A. Fine-Tuning CoTracker3 · p. 3
  • Training uses a 50-frame clip sampled every 10 frames from a random temporal offset, exposing the model to different video slices while managing computational load.Random offsets over iterations let the model eventually observe the full temporal extent of each training video.Source: A. Fine-Tuning CoTracker3 · p. 3
  • The tracker starts from human-annotated points and predicts each queried point’s 2D location and visibility in subsequent frames, including when fish are occluded or leave the frame.Predicted trajectories can recover points when fish reappear.Source: A. Fine-Tuning CoTracker3 · p. 3, Figure 2 · p. 4
  • The trajectory objective uses Huber loss: small errors are quadratic, while larger errors receive a linear penalty to reduce sensitivity to annotation noise or difficult frames.With high-quality annotations and relatively short sequences, δ is set so most errors fall in the quadratic region, making the loss behave similarly to Euclidean distance during typical training.Source: A. Fine-Tuning CoTracker3 · p. 3
  • Visibility is supervised with binary cross-entropy, and invisible points receive reduced weight because they are less informative but still matter for long-term tracking consistency.The final objective weights this visibility term against trajectory loss.Source: A. Fine-Tuning CoTracker3 · pp. 3–4
  • Training combines trajectory and visibility losses, with λ chosen to prioritize precise point localization while softly encouraging reliable occlusion estimation.The visibility loss for invisible points is weighted by 0.01, downweighting occluded regions while retaining supervision for occlusion and reemergence.Source: A. Fine-Tuning CoTracker3 · p. 4

B. Generating Language Queries

The pipeline converts predicted point trajectories into short language queries describing fish movement. These queries add motion information to point locations for downstream segmentation.

B. Generating Language Queries

  • Each trajectory is translated into a motion-centered query because point locations specify where a fish is, while language describes how it moves.Motion descriptions can distinguish fish in similar image regions when they move in different directions.Source: B. Generating Language Queries · p. 4
  • For each fish, GPT-4.1 generates a query for a 10-frame segment from coordinates at its first, middle, and last frames plus the video resolution.The method uses three representative frames because fish motion rarely changes drastically between consecutive frames.Source: B. Generating Language Queries · p. 4
  • The LLM is instructed to produce one 5–10-word description of positional movement, excluding numbers, coordinates, and descriptive adjectives.The prompt asks for movement across frames 1, 5, and 10 of a 1920×1080 underwater clip.Source: B. Generating Language Queries · pp. 4–5

C. Candidate Mask Generation

LMPM++ uses motion-informed language queries with video frames to generate candidate masks, which are filtered using tracked points. The point-guided step grounds the selected output in predicted fish locations.

C. Candidate Mask Generation

  • For each 10-frame segment, its motion query is applied to all corresponding RGB frames, and LMPM++ produces language-conditioned object embeddings and top-k candidate masks.The model is pretrained on MeViS, whose 33,000+ expressions use motion-focused vocabulary suited to the generated queries.Source: C. Candidate Mask Generation · p. 5
  • At inference, the pipeline keeps the union of candidate masks containing at least one tracked fish point; if none contains a point, it falls back to LMPM++’s top-ranked prediction.This point-guided filter grounds the output masks in predicted point labels rather than relying only on language-visual ranking.Source: C. Candidate Mask Generation · p. 5

D. Post-Processing

Post-processing composites predicted foreground regions onto an estimated background and uses dense optical flow to refine masks. It retains foreground pixels with significant motion to suppress static or incorrectly segmented objects.

D. Post-Processing

  • Background subtraction and optical flow are used to refine fish-mask boundaries and remove incorrectly segmented inanimate objects by distinguishing moving objects from static background regions.Dense optical flow estimates per-pixel motion, supporting boundary precision.Source: D. Post-Processing · p. 5
  • The method estimates background using the median of every fifth frame, composites each predicted foreground region onto it, and computes dense RAFT flow between consecutive composite frames.The refined mask retains pixels labeled foreground in the original mask that exhibit significant motion.Source: D. Post-Processing · p. 5

A. Evaluation

The evaluation measures segmentation overlap on DeepFish FishSeg, while tracker training uses sparse point supervision with frame-based sampling and annotation handling.

A. Evaluation

  • mIoU measures overlap between predicted and ground-truth segmentation masks across all classes, and the evaluation also considers detection capability.The test set contains 186 images from 35 video sequences.Source: A. Evaluation · p. 5

B. CoTracker3 Training

  • CoTracker3 is fine-tuned offline on fish clips with AdamW and OneCycleLR for 5,000 steps, using 256 trajectory points per sample and gradient clipping at 10.0.Each batch is one 50-frame clip sampled at 10-frame intervals; the learning rate is 5 × 10−5 and weight decay is 10−5.Source: B. CoTracker3 Training · p. 5
  • Point annotations are scaled to 384 × 512 resolution, and supervision applies only on annotated frames; missing-frame labels use the nearest annotation within a 50-frame window.Frames without annotations contribute zero loss.Source: B. CoTracker3 Training · p. 5

C. Comparison with Baselines

The comparison includes point-supervised, unsupervised, and foundation-model baselines. The paper reports that its pipeline exceeds the baselines in foreground and overall mIoU, while SAM2 requires test-time point prompts.

1) Baseline Selection:

  • The comparison covers Affinity-LCFCN, a reimplementation of Saleh et al., and foundation-model methods SAM2 [9], LangSAM, and AquaSAM.SAM2 and AquaSAM require ground-truth point prompts at test time; LangSAM combines SAM with GroundingDINO.Source: 1) Baseline Selection · p. 5

2) Quantitative Results Analysis:

  • Figure 3 compares input images, ground-truth masks, and four methods’ predictions across scenes with varying turbidity, lighting, and fish scale.The methods shown are Affinity-LCFCN [7], LangSAM [12], Saleh et al. [6], and the proposed pipeline.Source: Figure 3 · p. 6
  • Our pipeline surpasses the state-of-the-art baselines in foreground and overall mIoU, while SAM2 has higher background mIoU but requires ground-truth point prompts at inference.The pipeline uses human-annotated points only during training and predicts point labels for inference.Source: 2) Quantitative Results Analysis · p. 5, Table I · p. 6

3) Qualitative Results Analysis:

The qualitative comparisons describe how the pipeline’s tracked points, motion-informed language, and optical-flow post-processing relate to mask quality. The paper also notes computational, class-scope, and underwater-visual-noise limitations.

3) Qualitative Results Analysis:

  • The pipeline’s masks closely follow fish boundaries and suppress false positives on inanimate objects, unlike baselines described as coarse, undersegmenting, or prone to labeling background objects as fish.The proposed method grounds masks on tracked points, conditions LMPM++ on motion-informed language, and uses optical-flow post-processing to remove static regions.Source: 3) Qualitative Results Analysis · p. 6

V. LIMITATIONS AND CONSIDERATIONS

  • Dense RAFT flow, sequences up to 200 frames, and LLM queries for every 10-frame segment add inference cost, making real-time deployment less suitable without optimization.The repeated queries generate motion descriptions for every fish.Source: §V · p. 6
  • The method treats fish as one foreground class, while underwater caustics, particles, and turbidity can cause tracker points to drift and errors to reach mask generation.Point-guided filtering and background subtraction partially mitigate this; future work proposes using tracker visibility uncertainty to down-weight unreliable points.Source: §V · p. 6

VI. CONCLUSION

  • On the FishSeg benchmark from DeepFish, the approach achieves performance competitive with state-of-the-art weakly supervised methods without human input at inference.The conclusion presents the pipeline as a route toward scalable fish-analysis tools for aquaculture management, wild-population monitoring, and marine conservation.Source: §VI · p. 6
Loading 2610.29047v1…