PaperlayerView exact revision on arXiv

Source-linked AI summary

WildFin: An In-the-Wild Dataset for Fish Behavioral Recognition

Abigail G. Grassick, Jerome Tze-Hou Hsu, Ethan Lin, Ziang Liu, Max Whitton, Madelyn Hair, Liam Gutierrez, Haozheng Yu, Kristin Branson, Vivek Jayaraman, Michael A. Gil, Andrew M. Hein, Jennifer J. Sun

arXiv:2608.21281v1cs.CV

TL;DR

WildFin targets the shortage of expert-annotated underwater behavioral video needed to analyze fish in complex ecological environments. It constructs a benchmark from real-world field recordings spanning stationary and dynamic capture paradigms, then evaluates vision foundation models and adaptation strategies. The results show trade-offs between temporal reasoning and appearance-based modeling while exposing important pipeline and annotation limitations.

  • Problem

    Expert-annotated underwater behavioral video remains scarce, while wild marine imagery contains severe visual variability and fish behavior is difficult to analyze automatically.

  • Method

    WildFin builds a benchmark from ecological field data across stationary and dynamic recordings and evaluates frozen vision foundation backbones with lightweight adaptation and imbalance-mitigation strategies.

  • Results

    Current models show trade-offs: spatiotemporal architectures generally outperform image-based backbones on interaction-driven behaviors, while image models can remain competitive for appearance-dominated categories at lower computational cost.

  • Takeaways & Limitations

    WildFin provides a challenging platform for developing methods that combine temporal reasoning, appearance representations, and strategies suited to imbalanced ecological behavior data.

  • Takeaways & Limitations

    CoralCam errors propagate from detection into downstream stages, while FishFollow lacks explicit spatial localization tying behavior labels to the focal fish.

Abstract

from arXiv · show

Recent advances in field technology have led to a massive influx of in-the-wild video data for ecological science. The primary bottleneck in leveraging this data is the high cost of expert annotation. While computer vision offers a potential solution, current models frequently fail when deployed in complex marine environments. To characterize these failures, we introduce WildFin, a novel benchmark for fish behavior recognition collected and annotated by ecologists.WildFin spans two critical real-world paradigms: stationary cameras monitoring groups of fish and dynamic divers following individual subjects. The dataset represents a massive curation effort, involving 1,350 hours of fieldwork and 600 hours of expert annotation to produce 9 hours of behavioral data with over 2 million frame-by-frame labels. We benchmark modern vision foundation models and quantify tradeoffs between static and spatiotemporal architectures, revealing the substantial gap that remains between current model capabilities and the demands of real-world underwater behavioral analysis. Project website: https://team-wildfin.github.io/.

1 Introduction

WildFin addresses the scarcity of expert-annotated underwater behavioral video by combining real-world ecological recordings from stationary and dynamic capture scenarios. It provides a challenging benchmark for evaluating vision models under the variability of field data.

  • WildFin responds to a shortage of public expert-annotated underwater video benchmarks for fine-grained fish behavior recognition.Existing methods and datasets have more often focused on still images or settings that do not capture underwater behavioral complexity.
  • 9.2 hours of high-resolution video and over 2 million expert-annotated frame labels cover freely behaving fish in two real-world subsets.CoralCam contains stationary multi-agent recordings, while FishFollow contains dynamic focal-follow videos.
  • Field collection preserves dynamic lighting, occlusions, varied perspectives, and other complexities that models must overcome for scientific use.
  • WildFin introduces a video benchmark derived from ongoing ecological research across two distinct real-world capture scenarios.
  • Benchmark results expose shortcomings in current models and motivate lightweight pooling, fine-tuning, and class-imbalance mitigation strategies.

2 Related Work

Prior underwater computer-vision research has emphasized still images and object-level video annotations, while comprehensive expert behavioral labels remain scarce. WildFin extends this landscape with ecologically grounded video data that reflects the variability and imbalance of animal behavior in the wild.

  • Most publicly available underwater datasets and methods focus on still-image detection, classification, or segmentation rather than behavioral video analysis.
  • Existing underwater video datasets commonly provide bounding boxes or segmentation masks but lack comprehensive behavior annotations for fine-grained ecological analysis.
  • WildFin addresses the gap by providing ecologically relevant underwater video with expert behavioral annotations for marine-ecology and video-understanding research.
  • In-the-wild underwater data combines variable lighting, viewpoints, clutter, occlusion, turbidity, poor contrast, and diverse fish morphology and movement.
  • Long-tailed behavior distributions make rare but ecologically important actions difficult to model and motivate focal loss, class weighting, and balanced sampling.

3 Dataset

WildFin is an ecologically grounded, frame-level multilabel benchmark spanning stationary group recordings and mobile focal-follow videos. Its realistic conditions and staged annotation pipelines expose substantial limitations in tracking, localization, detection, and behavioral classification.

  • Dataset composition: 9.2 hours and 2,058,892 frames comprise two expert-annotated subsets spanning 23 behaviors across stationary CoralCam and mobile FishFollow recordings.CoralCam captures multi-agent reef scenes, while FishFollow follows individual fish through their habitats.
  • Dataset composition: WildFin’s primary task assigns a behavioral label to every frame for a tracked individual, with detection and tracking released as upstream benchmark tasks.Evaluation is aligned to expert-verified tracks at the finest temporal granularity supported by the annotations.
  • Data-collection paradigms: CoralCam uses stationary cameras across reef sites, producing dense multi-agent scenes with occlusion, small objects, textured backgrounds, suspended particles, and variable lighting.These conditions create challenges from schools of fish, benthic clutter, and underwater visual noise.
  • Data-collection paradigms: FishFollow uses a diver-held camera following one fish, introducing ego-motion, viewpoint and scale changes, and dynamic backgrounds across habitat patches.The paradigm captures interactions between the focal fish, its surroundings, and other species.
  • Annotation pipeline: The annotation pipeline combines expert detection, BoTSort tracking, manual track verification, and frame-level behavioral labeling, including a not-visible class for obscured orientations.CoralCam contains 21,116 expert-labeled boxes and 213 manually verified tracks; the detector achieved AP@0.5 of 74.5 on a held-out test set.
  • Limitations: CoralCam’s behavioral data are biased toward near-field, longer, calmer tracks because early detection errors propagate and selected tracks favor fewer occlusions and standardized backgrounds.Identity switches during occlusion remain unquantified, while fast or erratic behaviors such as charging are under-represented.

4 Benchmark Design and Protocol

WildFin benchmarks frozen vision backbones with lightweight adaptation heads, comparing static and spatio-temporal processing while addressing severe behavioral class imbalance. Evaluation uses site- and date-separated splits, task-specific inputs, and tolerance-aware macro metrics.

  • Adaptation strategy: Frozen pre-trained backbones feed lightweight pooling and classification heads, with experiments spanning foundation models, fine-tuning strategies, and class-imbalance mitigation.The adaptation design compares simple mean pooling with learnable cross-attention pooling.
  • Evaluation protocol: Approximately 70% of videos are used for training and 30% for testing, with videos separated across splits by capture site and date.This protocol is applied to both CoralCam and FishFollow.
  • Backbones and inputs: Image models process single frames, whereas video models process 16-frame clips to compare static frame-based and spatio-temporal representations.Inputs are resized to 224 × 224 pixels; CoralCam uses fish crops, while FishFollow uses full-size videos.
  • Evaluation protocol: Macro-averaged F1, precision, and recall are reported using a 7-frame tolerance, approximately ±0.1 seconds at 60 FPS.The tolerance accommodates ambiguity in event boundaries and short temporal manifestations of point-based behaviors.
  • Backbones and inputs: The backbone suite includes DINOv3 and ResNet50 for image-based processing, plus V-JEPA2 and VideoMAE models for video-based processing.The selection contrasts modern vision foundation models with a classical supervised baseline and efficient versus larger video models.
  • Class imbalance: Balanced sampling and focal loss address the severe class imbalance expected in natural behavioral datasets.Balanced sampling equalizes sampling across positive behaviors and the null class, while focal loss emphasizes rarer difficult positives.

5 Benchmark Analysis

WildFin benchmarks how architecture, temporal reasoning, fine-tuning, imbalance handling, pooling, and compute affect fish behavior recognition across CoralCam and FishFollow. Results show that no single strategy dominates: temporal models help interaction-driven behaviors, while appearance-focused or lightweight approaches remain competitive in other settings.

  • Temporal vs. Static Cues: Spatiotemporal models generally outperform image-based backbones, with VideoMAE and V-JEPA-2 achieving the highest scores on both datasets.The advantage is strongest for behaviors with pronounced temporal cues.
  • Temporal vs. Static Cues: 0.332 macro-F1: VideoMAE-L detects CoralCam Aggression, whereas image-based models fail on that class.This result highlights the value of temporal context for dynamic or interaction-driven behaviors.
  • Temporal vs. Static Cues: 0.490 versus 0.510 macro-F1: DINOv3-L remains competitive with VideoMAE-L on CoralCam Biting, while image backbones lead FishFollow Habitat.Appearance cues can therefore suffice for some behavior categories at lower computational cost.
  • Frozen Foundation Models vs Fully Fine-Tuned CNNs: Fully fine-tuned ResNet50 outperforms DINOv3-L on static CoralCam, but this advantage disappears on complex FishFollow, where frozen DINOv3-L features are better.The authors attribute the FishFollow pattern to apparent overfitting by the fully tuned ResNet50.
  • Impact of Class Imbalance Strategies: Focal loss usually improves performance by increasing recall at a modest precision cost, except for VideoMAE on CoralCam, where rare-behavior overfitting harms test performance.Naive random sampling produces near-zero performance on rare behaviors, underscoring the need for targeted sampling.
  • Effect of Pooling Strategies: Attention pooling outperforms mean pooling across nearly all backbones, although mean pooling is substantially better for V-JEPA-2-L.The authors hypothesize that V-JEPA’s predictive embedding pretraining distributes information across tokens, which attention pooling may discard.
  • Parameter Count and Performance Tradeoff: Attention pooling generally improves macro-F1 as FLOPs increase, while FishFollow shows more volatile scaling behavior.V-JEPA models follow the broader pooling gap but exhibit distinct scaling trajectories.
  • Parameter Count and Performance Tradeoff: Video-native backbones generally achieve higher F1 than image-based models at equivalent parameter counts, and VideoMAE Large reaches peak macro-F1 on both datasets.The result suggests greater parameter efficiency from temporal pretraining or video-specialized architectures.

6 Conclusion

WildFin is presented as an in-situ fish-behavior dataset designed to support ecological and computer-vision research. Its varied perspectives, lighting, backgrounds, subjects, and behaviors make field imagery challenging to automate, while motivating methods that could accelerate analysis of underwater footage.

  • Conclusion: WildFin provides expert-annotated in-situ fish behaviors as a platform for studying behavioral dynamics and developing computer-vision methods.The authors connect these behaviors to the health of marine ecosystems and interdisciplinary collaboration.
  • Conclusion: WildFin spans varied camera perspectives, backgrounds, lighting conditions, subject diversity, and behaviors that challenge automated field-image analysis.The dataset is intended to support methods development for difficult ecological imagery.
  • Conclusion: Automated analysis methods could accelerate scientific-image processing and add value to footage collected by recreational divers and citizen scientists.The stated scope includes underwater imagery from oceans, rivers, and coral reefs.

7 Dataset Details

WildFin combines two ecologically collected fish-video datasets with expert-reviewed tracking and behavior annotations. The curation process addresses difficult underwater visibility, tracking, and labeling conditions while documenting annotation quality and exclusions.

  • Dataset documentation: WildFin documents CoralCam and FishFollow recording conditions, annotation protocols, labeling challenges, and behavioral taxonomies for reproducible benchmark use.The documentation covers both datasets and includes an ethogram for their behavioral categories.
  • CoralCam: CoralCam contains 12 videos of mixed-species fish schools recorded in Curaçao under conditions selected to reduce background complexity and conspecific occlusion.The dataset focuses on brown chromis, bicolor damselfish, and juvenile bluehead wrasse.
  • CoralCam: 213 CoralCam tracks were selected for behavior annotation after expert verification that each sequence consistently followed the same individual.The selected tracks averaged approximately 1,215 frames.
  • Annotation and tracking: CoralCam bounding boxes were produced by trained annotators and expert-reviewed, yielding 21,116 verified boxes for three target species.The boxes supported object-detection and tracking models used before behavioral annotation.
  • Evaluation scope: Behavior evaluation excludes rare categories, including C-Turn on CoralCam and several FishFollow behaviors with fewer than 30 training examples.The excluded categories remain potentially useful for future retrieval or retrieval-augmented tasks.
  • Labeling challenges: A “not visible” behavior class captures frames where focal fish are facing away, out of frame, or occluded, reflecting unavoidable underwater visibility constraints.Cryptic appearance against reef backgrounds also required manual expert verification of boxes and track identities.
  • FishFollow: FishFollow comprises 8 hours of Curaçao footage collected by snorkelers following individual parrotfish across varied habitats and behavioral contexts.Videos were independently annotated by three students, converted from time-based labels to frame-level annotations, and expert-reviewed.
  • Annotation quality: 0.86 average F1 was achieved for FishFollow annotations with a 0.5-second tolerance, compared with 0.74 F1 between the two annotators evaluated on the broader annotation comparison.At the increased tolerance, average precision was 0.925 and recall was 0.84.

8 Experiment Details

Experiments use non-overlapping video-level splits, standardized preprocessing and convergence protocols, and a two-stage frozen-backbone pipeline. Evaluation reports average precision across standard IoU thresholds for held-out test data.

  • Data splits: Approximately 70% of each dataset is used for training and 30% for testing, with videos assigned exclusively to one split to prevent information leakage.A validation set is randomly selected from 10% of the training data using the same non-overlapping criteria.
  • Dataset statistics: Training and test datasets contain documented frame counts and per-frame behavior frequencies for CoralCam and FishFollow.Tables 6 and 7 summarize these split sizes and behavior distributions.
  • Training pipeline: The pipeline freezes each backbone, pre-extracts frame or 16-frame-clip features, and trains a lightweight classification head on those features.Feature extraction is processed independently for each video or track.
  • Compute: 1–10 GPU hours are required per lightweight-head experiment, while full-test inference requires approximately 1 GPU hour per pass.Feature extraction takes roughly 3 GPU hours for CoralCam and 20 GPU hours for FishFollow, depending on hardware and backbone size.
  • Training protocol: Models use recommended implementation hyperparameters, train for 40 epochs, and select the checkpoint with the highest validation mAP for test evaluation.The protocol avoids early stopping and emphasizes convergence across configurations.
  • Input preprocessing: Image models process 224 × 224 RGB inputs, whereas video models process 16-frame clips with shape (16, 3, 224, 224).Resizing and center cropping are applied consistently across training, validation, and testing.
  • Label encoding: CoralCam and FishFollow use multi-hot labels to represent simultaneous behaviors, with tolerance regions accommodating timing uncertainty.The example tolerance is ±7 frames for short-duration behavior annotations.
  • Evaluation: Final evaluation uses held-out test data and reports AP at IoU 0.50, AP at IoU 0.75, and mAP@[.5:.95].The mean AP averages thresholds from 0.50 to 0.95 in 0.05 increments.

9 Additional Experimental Results

Additional experiments compare backbone, pooling, and class-imbalance strategies across both datasets. Ablations indicate that mean pooling generally outperforms the stronger V-JEPA-2 adaptor, while uniform sampling trades performance across behavior frequencies.

  • Experimental comparisons: Tables 8–11 report combinations of backbone, pooling, and class-imbalance strategies for CoralCam and FishFollow using F1 and mAP measures.CoralCam additionally reports per-class scores, while FishFollow retains grouped scores for legibility.
  • Pooling ablation: V-Attn generally improves over simple attention pooling but still falls short of mean pooling in the reported results.The V-Attn adaptor is the stronger V-JEPA-2 adaptor from the original V-JEPA-2 code.
  • Sampling ablation: Uniform sampling often performs better on more frequent behaviors while producing poorer performance on rare behaviors.Overall performance remains comparable to the alternative sampling strategy in the ablation.

10 Accesibility

WildFin follows a documented datasheet approach and is released openly with the materials needed for reproduction and extension. The accompanying tables expose benchmark coverage and performance summaries for both datasets.

  • Documentation: WildFin follows the datasheet proposed in for documenting the dataset.This provides a structured documentation framework for the benchmark.
  • Access: The dataset is openly available for research use without categorical access restrictions, subject to appropriate citation.The project website provides access under an open research-use license.
  • Reproducibility: The release includes code, training configurations, data splits, hyperparameters, supporting scripts, and documentation of data formats and annotation protocols.These materials are intended to support reproduction and extension of the benchmark.
  • Benchmark reporting: Tables 8 and 9 summarize performance comparisons on CoralCam and FishFollow, including backbone, pooling, and class-imbalance strategies.Tables 10 and 11 provide corresponding mAP comparisons, with CoralCam including per-class AP columns.
Loading 2608.21281v1…