Source-linked AI summary
Frame-to-Panorama Localization and Context-Aware Sampling for Scene-Specific Ship Detection in a Smart Marina Testbed
Ignat Romanov, Andreas Hadjipieris, Neofytos Dimitriou
TL;DR
Historical PTZ video often lacks reliable camera-view metadata, while view direction and zoom affect visual content and complicate curation across the operational field of view. The pipeline localizes frames on a reference panorama, enriches them with weather and solar-state context, and uses diversity sampling plus horizon-focused visual clustering. Sampling reduced data volume by 99.5% to 220 images; sequence-grouped five-fold outer cross-validation yielded 94.78% ± 0.51% AP50 and 75.10% ± 1.73% AP50–95.
Problem
Historical PTZ video often lacks reliable camera-view metadata, while view direction and zoom affect visual content and complicate curation across the operational field of view.
Source: §I · pp. 1–2Method
The pipeline localizes frames on a reference panorama, enriches them with weather and solar-state context, and uses diversity sampling plus horizon-focused visual clustering.
Source: §I · p. 2, A. Camera-view recovery · p. 3Results
Sampling reduced data volume by 99.5% to 220 images; sequence-grouped five-fold outer cross-validation yielded 94.78% ± 0.51% AP50 and 75.10% ± 1.73% AP50–95.
Source: §I · p. 2Takeaways & Limitations
The selected images captured different marina views and environmental conditions, supporting scene-specific detector adaptation with substantially reduced annotation effort.
Source: B. Detection performance · p. 6The trained model sometimes misses larger unseen vessels, vessels under extreme sun glare, and highly zoomed, boundary-truncated ships.
Source: B. Detection performance · p. 6
Abstract
from arXiv · showhide
Smart maritime infrastructures provide continuous access to heterogeneous sensing streams, enabling repeated experimentation, digital-twin development, and AI-based maritime services. However, sensing hardware alone is not sufficient for scene-specific model development: historical video streams must also be spatially indexed, contextualized, and reduced to informative subsets for annotation. This paper presents a frame-to-panorama localization and context-aware sampling pipeline for ship detection in historical PTZ maritime video lacking reliable pan, tilt, and zoom metadata. The main contribution is an end-to-end data-curation approach that recovers camera-view information from historical PTZ video and combines it with environmental context and visual diversity to construct compact, scene-specific training sets. Specifically, frames are localized on a reference panorama using SuperPoint and LightGlue, enriched with weather and solar-state metadata, and selected through diversity sampling to preserve variation across camera view and environmental conditions. A second context-aware stage targets under-represented distant-vessel cases near the horizon using tile-level visual embeddings and Gaussian Mixture Model clustering. Applied within the CMMI MDigi-I Smart Marina testbed, the proposed pipeline reduces 40,718 candidate frames to 220 images for annotation, corresponding to a 99.5% reduction. A YOLO26-m detector fine-tuned on this subset achieves a mean AP50 of 94.78% $\pm$ 0.51% and a mean AP50-95 of 75.10% $\pm$ 1.73% under sequence-grouped five-fold cross-validation. These results demonstrate that highly redundant infrastructure video streams can be transformed into compact, spatially and contextually diverse training sets for scene-specific detector adaptation while substantially reducing annotation effort.
I. INTRODUCTION
The paper turns redundant smart-marina PTZ video into compact, spatially and contextually diverse data for scene-specific ship detection. It addresses the challenge of curating historical footage without reliable camera-view metadata through panorama localization and context-aware sampling.
I. INTRODUCTION
- The CMMI MDigi-I Smart Marina combines sensing assets and computational resources, providing a testbed for scene-specific perception amid varying views, illumination, weather, glare, and vessel scales.The camera is installed approximately 100 m above the water.Source: §I · p. 1
- Historical PTZ footage lacks reliable view metadata, making it difficult to curate data that covers the operational field of view; repeated poses and similar content also make exhaustive annotation inefficient.Smart-marina infrastructure offers continuous data under real environmental and operational conditions for developing and validating computer-vision methods.Source: §I · pp. 1–2
- Frames are localized on a manually constructed panorama using SuperPoint, LightGlue [8], and a homography, then enriched with weather and solar context for diversity sampling.A second stage uses tile-level visual embeddings and Gaussian Mixture Model clustering to retain frames with vessel-relevant open-water context near the horizon.Source: §I · p. 2
- The sampling methodology reduces data volume by 99.5%, producing a manually annotated subset of 220 images; the detector achieves 94.78% ± 0.51% AP50 and 75.10% ± 1.73% AP50–95.Evaluation uses sequence-grouped five-fold outer cross-validation.Source: §I · p. 2
II. METHODS
The study uses daylight frames from an elevated PTZ camera at Ayia Napa Marina, collected over several months and extracted at one frame per minute.
II. METHODS
- The source footage came from a PTZ CCTV camera approximately 100 m above Ayia Napa Marina, Cyprus, with recordings from 1 August 2025 to 10 January 2026.Recordings were saved automatically when scene activity was detected or when an operator manually controlled the camera.Source: §II · p. 3
- One frame per minute was extracted, and the resulting frames lacked reliable pan, tilt, zoom, weather, and lighting metadata.Source: §II · p. 3
- The analysis used daylight frames, which comprised 40,718 of the original 64,000 frames, with daylight determined from solar telemetry.Source: §II · p. 3
A. Camera-view recovery
Camera views are localized in a manually stitched panorama by matching local image features and projecting each frame’s center and coverage into panorama coordinates.
A. Camera-view recovery
- SuperPoint extracts keypoints and descriptors from each frame and panorama, LightGlue [8] matches them, and USAC-MAGSAC [2] estimates a homography.The projected frame center and coverage provide panorama coordinates and an approximate view-scale measure.Source: A. Camera-view recovery · p. 3
- Feature-based localization tolerates minor panorama seams and local distortions when enough consistent correspondences remain in the overlapping region.Sensitivity rises for highly zoomed frames overlapping mainly with a seam or distorted area, where too few matches may remain for reliable estimation.Source: A. Camera-view recovery · p. 3
- Localization also records LightGlue confidence, geometric inlier count, and projected coverage area; coordinate variation mainly affects frames near spatial-bin boundaries.The passage states that such variation is unlikely to substantially alter subsequent multi-feature diversity selection.Source: A. Camera-view recovery · p. 3
B. Metadata enrichment and diversity sampling
Frames receive weather and solar-state context, then are sampled within horizontal panorama bins to reduce redundancy while preserving view and environmental diversity.
B. Metadata enrichment and diversity sampling
- The nearest hourly weather observation is assigned when its timestamp is within one hour of the frame, alongside a solar-state label.Source: B. Metadata enrichment and diversity… · p. 3
- Solar state is labeled day, night, or transition relative to sunrise and sunset, with a 30-minute buffer around each event defining transition periods.Source: B. Metadata enrichment and diversity… · p. 3
- Kennard-Stone sampling [6], selects Z-score-normalized representatives within horizontal panorama bins using weather, vertical coordinate, and projected coverage; the pool falls from 40,718 to 2,402 images (94.1% reduction).Bins with fewer than the per-bin limit retain all images.Source: B. Metadata enrichment and diversity… · p. 3
C. Context-aware sampling
Context-aware filtering targets distant vessels near the horizon by analyzing cropped maritime regions at tile level. GMM clustering retains frames with vessel-relevant context while filtering visually similar structures.
C. Context-aware sampling
- Figure 3 contrasts vessel-relevant tiles with visually similar but non-relevant tiles in the GMM clusters.The non-relevant examples include waves, masts, and breakwater regions.Source: Figure 3 · p. 3
- A lightweight YOLO11-n detector identifies the horizon-facing maritime region, which is cropped from each frame for context-aware filtering.This stage prioritizes frames containing small vessels in open water outside the marina.Source: C. Context-aware sampling · p. 3
- Each crop is divided into overlapping 192 × 192 pixel tiles, and an ATSS-SwinL-DyHead model supplies boat predictions and visual embeddings.Tiles have 10% overlap; edge tiles are retained if they are at least 64 pixels, and only tiles predicted as boat proceed to clustering.Source: C. Context-aware sampling · p. 3
- 2,402 frames were reduced to 220 by retaining frames with at least one tile assigned to a vessel-relevant GMM cluster.The clusters separate vessel-relevant tiles from visually similar non-relevant structures such as waves, buoys, masts, and breakwater regions.Source: C. Context-aware sampling · p. 4
D. Annotation and model training
Manually curated segmentation annotations provide the object-detection labels, and the resulting dataset is used to train and evaluate YOLO26-m with sequence-grouped five-fold cross-validation.
D. Annotation and model training
- The annotation protocol labels visible vessel structures while excluding thin or ambiguous elements such as masts and railings.The manually curated segmentation annotation is used to create object-detection labels.Source: Figure 4 · p. 4
- The dataset contains 220 images and 4,017 vessel boxes; 208 images have at least one labeled vessel, while 12 have none.The standard YOLO26-m architecture is used without architectural modifications.Source: D. Annotation and model training · p. 4
- Five-fold outer cross-validation groups images into 182 acquisition sequences to keep temporally related frames in the same fold.Each fold evaluates 44 images; training uses 158 images and internal validation uses 18, so each image is evaluated once by a model that did not train on it.Source: D. Annotation and model training · p. 4
- Each fold starts from the same pretrained YOLO26-m checkpoint and trains for up to 500 epochs with early stopping.Training uses 1280 × 1280 pixel inputs, batch size 16, cosine learning-rate scheduling, automatic mixed precision, and patience of 50 epochs; seeds are fixed at 42.Source: D. Annotation and model training · p. 4
A. Frame-to-panorama localization
Frame-to-panorama localization is evaluated as spatial-bin classification against SIFT-FLANN. SuperPoint-LightGlue achieves higher global accuracy, with failure patterns differing under zoom and illumination conditions.
A. Frame-to-panorama localization
- The daytime evaluation partitions the horizontal panorama coordinate into 20 equal-width bins and samples up to 100 images per bin.Frames are manually assigned ground-truth bins by cross-referencing them with the reference panorama.Source: A. Frame-to-panorama localization · p. 4
- Across all metrics, SuperPoint+LightGlue significantly outperforms traditional SIFT+FLANN, with 87.36% global accuracy versus the baseline’s 12.34%; the deep-learning pipeline predominantly fails on highly zoomed-in frames.Source: A. Frame-to-panorama localization · p. 4
- SuperPoint-LightGlue failures occur predominantly on highly zoomed-in frames, where limited field of view leaves too few keypoints for robust homography estimation.SIFT-FLANN often matches under the same conditions, but those matches are typically inaccurate.Source: A. Frame-to-panorama localization · p. 4
- Deep-model failures on zoomed-out frames are rare, while illumination-discrepant examples can favor learned matching over handcrafted descriptors.In the latter cases, SuperPoint-LightGlue registers frames where SIFT-FLANN fails.Source: A. Frame-to-panorama localization · pp. 4–5
- Figure 5 uses green lines for SIFT-FLANN and red lines for SuperPoint-LightGlue, with dots marking recovered panorama locations and outside markers indicating failures.The bottom images show cases where one method fails under challenging visual conditions.Source: Figure 5 · p. 5
B. Detection performance
Sequence-grouped evaluation reports detector performance across five folds, alongside a live-feed example and remaining failure cases.
B. Detection performance
- 94.78% ± 0.51% AP50 and 75.10% ± 1.73% AP50–95 were achieved across five outer folds.Table II summarizes sequence-grouped five-fold outer cross-validation.Source: B. Detection performance · p. 5
- The live PTZ feed example detects boats throughout the marina and a small boat on the horizon outside it.Source: Figure 6 · p. 5, B. Detection performance · p. 5
- The small AP50 standard deviation and 93.92%–95.27% fold-level range indicate stable performance across partitions, not just a fortunate 90%/10% split.The reported two-sided 95% fold-level Student-t interval for mean AP50 was 94.15%–95.41%.Source: B. Detection performance · p. 5
- The selected annotation set contains 220 images spanning different marina areas, camera views, and environmental conditions.The set supports scene-specific detector adaptation with substantially reduced annotation effort.Source: B. Detection performance · pp. 5–6
- The model sometimes misses larger vessel types absent from training, vessels under extreme sun glare, and highly zoomed-in vessels truncated at the frame boundary.The cited zoomed-in cases have vessels occupying at least 50% of the image.Source: B. Detection performance · p. 6
- Future work will automate manual Photoshop-based stitching and explore generalizability across other marina-like environments.The proposed stitching replacement is intended to reduce manual intervention and improve scalability and reproducibility.Source: B. Detection performance · p. 6