CoralscapesV2: Panoptic and Fine-Grained Visual Scene Understanding in Coral Reefs

1 Introduction

• Effective conservation and restoration require monitoring that scales spatially and temporally while capturing fine-grained ecological detail.Only a small fraction of the world’s reefs are actively and regularly monitored, while reef decline and recovery vary substantially across regions.Source: §1 · p. 2
• Computer vision for reefs is shifting from standardized survey imagery toward flexible analysis across diver-held, robotic, and static-camera platforms.The paper frames this shift as a need for general-purpose analysis of reef imagery.Source: §1 · p. 2
• Coralscapes provided dense general-purpose segmentation, but treated fish as one semantic class to be masked out during benthic mapping.Existing fish datasets also lack one or more of static-image context, complete annotation, coral-reef coverage, or instance masks, limiting ecological applications.Source: §1 · p. 2
• The refined label set has 95 fine-grained classes spanning taxa, coral morphologies and health status, plus new auxiliary classes; 65k fish are exhaustively annotated on each frame using video.Annotated images increase from 2075 to 2433, polygons from 174k to 270k, annotation density from 82.9% to 86.8%, and reef sites from 35 to 45.Source: §1 · p. 2
• CoralscapesV2 is the first panoptic segmentation dataset for coral reefs, supporting joint benthic-cover mapping and fish quantification from unconstrained video.The paper also identifies fish behavior and fish-reef interactions as applications.Source: §1 · p. 3

2.1 Coral Reef Image Datasets

• Dense reef masks remain relatively uncommon: other dense-mask datasets are constrained to orthomosaics, while CoralSCOP distinguishes only coral from non-coral.Most annotated reef data consists of sparse point labels used to train classification models.Source: §2.1 · p. 3
• CoralscapesV2 extends Coralscapes’ dense general-purpose frames from 39 to 95 classes and adds exhaustive fish instance masks annotated with video context.The original dataset used static annotation and represented fish as a single semantic class.Source: §2.1 · p. 3

2.2 Fish Datasets

• No existing fish dataset combines coral-reef scenes, exhaustive per-fish instance masks, and video context; CoralscapesV2 is the first to provide all three.Within reefs, prior datasets provided boxes or tracks, while masks elsewhere were often non-exhaustive or limited to salient fish.Source: §2.2 · p. 4

3.1 Increased Size, Completeness, and Quality

• CoralscapesV2 extends the original dataset with more frames, annotated polygons, and reef sites, while increasing average annotated area from 82.9% to 86.8%.The minimum annotated area per image rises from 40% to 60%, and the total number of user-annotated segmentation masks rises from 174k to 270k.Source: §3.1 · pp. 5–6, Figure 1 · p. 5, Figure 2 · p. 6

3.2 Fine-Grained Visual Categories

• The label set grows from 39 to 95 consistently annotated fine-grained visual categories.The original classes were selected to be well represented across splits; CoralscapesV2 relaxes that requirement.Source: §3.2 · pp. 6–7, Figure 5 · p. 9

More Taxa & Morphotypes

• New labels separate identifiable hard- and soft-coral taxa, add morphology classes when taxonomy is unclear, and retain catchall classes for full coverage.Additional classes include black coral ‘cirrhipathes’ and sabellid ‘feather worm’; the strategy prioritizes non-speculative labeling over taxonomic depth.Source: More Taxa & Morphotypes · pp. 7–8

Fine-Grained Coral Composite Health Classes

• Composite ‘bleached’, ‘dead’, and ‘algae covered’ health classes are provided for taxonomic classes.Source: Fine-Grained Coral Composite Health… · p. 8

Auxiliary Classes

• Three auxiliary classes distinguish human-made underwater infrastructure, shimmering water surfaces, and unobstructed water-column views.‘Background’ captures other areas where blur or visibility prevents a speculation-free assessment of coral taxon or health.Source: Auxiliary Classes · p. 8, Figure 4 · p. 8

3.3 Exhaustive Fish Instance Annotations

• Using a four-second video around each frame, annotators labeled 65k fish instances; static-frame annotation missed 56.1% of instances on matching V1 frames.Video-based labels also reveal more frames with higher fish counts and newly annotated fish concentrated in smaller polygons; motion helps distinguish fish from marine snow or dark substrate patches.Source: §3.3 · p. 10, Figure 6 · p. 10, Figure 7 · p. 11

3.4 Limitations

• Precise exhaustive fish annotation may be impossible for large, dense schools of small fish or fish at the water-column visibility limit.Source: §3.4 · p. 11, Figure 8 · p. 11
• The 95-class label set cannot be split so that every class is well represented in train, validation, and test data.Source: §3.4 · p. 11

4.1 Semantic Segmentation

• Training on CoralscapesV2 improves accuracy and mIoU on both V2 and V1 test sets.The comparison includes the V1 SegFormer baseline and models trained on the V2 95-class set and reduced 39-class set.Source: §4.1 · p. 12, Figure 9 · p. 12, Table 2 · p. 12
• Semantic segmentation compares the V1 SegFormer MiT-b5 baseline with V2-trained models using the 95-class set and reduced 39-class set.The DINOv3-pretrained ViT-B and ViT-L models with DPT heads and LoRA are also benchmarked.Source: §4.1 · p. 12

4.2 Instance Segmentation

• For ResNet-50, video models trend toward better performance with temporal context, but the Swin-L image model outperforms its video variants.The benchmark compares image, 2-frame video, and 3-frame video variants using the Video-Mask2Former decoder.Source: §4.2 · p. 13, Table 3 · p. 13

• CoralscapesV2 expands the label set to 95 fine-grained categories and adds 65k exhaustive fish-instance masks.The categories capture more taxa, coral morphologies, health status, and auxiliary classes.Source: §5 · p. 13
• 56.1% of fish instances were missed in static-frame annotations, and video context revealed at least one fish in 51% of frames thought fish-free.Despite this evidence for the value of temporal context, model-training results on the video task are inconclusive.Source: §5 · p. 13
• The dataset lays groundwork for scaling fine-grained monitoring beyond constrained surveys and studying fish behavior and fish-reef interactions.Proposed applications include joint benthic-cover mapping and fish quantification from handheld diver or robotic surveys.Source: §5 · p. 13

• The figure compares V1 fish segmentation masks with CoralscapesV2 instance masks, highlighting added fish and removed false positives.Source: Figure 10 · p. 19

• SegFormer MiT-b5 training uses AdamW at a learning rate of 6×10−5, weight decay 0.01, linear decay, 100 epochs, and batch size 4.Training uses the combined train+validation split and evaluation uses the test split.Source: SegFormer · p. 20
• Augmentations include 1024 × 1024px random resized crops, color jitter, rotations up to ±15◦, and random horizontal flips.Crop scale is 0.02–1.0; brightness, contrast, and saturation range from 0.8 to 1.2, and hue from −0.05 to 0.05.Source: SegFormer · p. 20
• The training recipe follows the best hyperparameter configuration of.Source: SegFormer · p. 20

• Frozen DINOv3 ViT-B and ViT-L encoders pretrained on LVD-1689M are adapted with LoRA (r = 16, α = 32, dropout 0.05).Source: DinoV3+LoRA+DPT · p. 20
• The DPT head uses fusion dimension 256, neck hidden sizes [96, 192, 384, 768], and reassemble factors [4, 2, 1, 0.5].Source: DinoV3+LoRA+DPT · p. 20
• Training runs for 150 epochs with AdamW, learning rate 5 × 10−5, weight decay 0.01, 1000 warmup steps, and mixed precision; evaluation uses 768 × 1376px.Augmentations include multi-scale sizes, random resized crops, rotations up to ±20◦, and color jitter.Source: DinoV3+LoRA+DPT · p. 20

• Video Mask2Former uses a COCO instance-segmentation-initialized backbone, one fish thing class, 10 transformer decoder layers, and a multi-scale deformable-attention pixel decoder.The pixel decoder has 6 encoder layers and hidden dimension 256.Source: Mask2Former · p. 20
• The loss combines classification, mask, and dice terms weighted 2.0, 5.0, and 5.0, respectively, with a no-object weight of 0.1.Source: Mask2Former · p. 20
• The three variants use an image anchor frame, frames [−1, 0], or frames [−2, −1, 0]; all frames pass through the network, but only the anchor frame is supervised and scored.Models are tuned on CoralscapesV2 train+val for 15000 iterations.Source: Mask2Former · p. 20

C Supplemental Results

• The test-set visualization presents ground-truth segmentation labels alongside predictions from SegFormer (MiT-b5) and DinoV3 ViT-L+LoRA+DPT trained on V1 and V2.Source: Figure 11 · p. 21
• Extended results compare single-pass prediction at 768×1376px with strided 1024×1024px windows, and compare 39-class training with fine-grained 95-class training mapped to legacy classes for evaluation.Source: Table 4 · p. 22
• Training on Coralscapes V2 improves every model’s performance on both semantic-segmentation metrics and datasets for the 39-label class set.Source: Figure 12 · p. 22