PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Processing from Electro-optical Sensors for Object Detection and Tracking in Maritime Environment: A Survey

D. K. Prasad, D. Rajan, L. Rachmawati, E. Rajabaly, C. Quek

arXiv:1611.05842v1cs.CV

TL;DR

Maritime autonomous navigation needs EO video processing to complement ranging sensors, but detection and tracking remain difficult in dynamic and variable sea scenes. This survey taxonomizes maritime and computer-vision approaches, introduces the Singapore Marine Dataset for evaluation, and reports mixed technique-specific results, including limited practical precision and recall for dynamic-background subtraction.

  • Problem

    Autonomous-ship navigation requires maritime object detection and tracking, while EO processing is challenged by camera motion, dynamic backgrounds, low contrast, and environmental variation.

  • Method

    The paper provides an approach-based survey of maritime EO detection and tracking methods and evaluates selected techniques on the Singapore Marine Dataset.

  • Results

    Mean SSIM for consecutive frame pairs was 0.75; horizon and cross-correlation registration improved SSIM by 6-7%, while dynamic-background methods did not provide practically useful precision and recall.

  • Takeaways & Limitations

    The survey identifies computer-vision methods with promise for maritime EO processing and provides a benchmark dataset for quantitative comparison.

Abstract

from arXiv · show

We present a survey on maritime object detection and tracking approaches, which are essential for the development of a navigational system for autonomous ships. The electro-optical (EO) sensor considered here is a video camera that operates in the visible or the infrared spectra, which conventionally complement radar and sonar and have demonstrated effectiveness for situational awareness at sea has demonstrated its effectiveness over the last few years. This paper provides a comprehensive overview of various approaches of video processing for object detection and tracking in the maritime environment. We follow an approach-based taxonomy wherein the advantages and limitations of each approach are compared. The object detection system consists of the following modules: horizon detection, static background subtraction and foreground segmentation. Each of these has been studied extensively in maritime situations and has been shown to be challenging due to the presence of background motion especially due to waves and wakes. The main processes involved in object tracking include video frame registration, dynamic background subtraction, and the object tracking algorithm itself. The challenges for robust tracking arise due to camera motion, dynamic background and low contrast of tracked object, possibly due to environmental degradation. The survey also discusses multisensor approaches and commercial maritime systems that use EO sensors. The survey also highlights methods from computer vision research which hold promise to perform well in maritime EO data processing. Performance of several maritime and computer vision techniques is evaluated on newly proposed Singapore Maritime Dataset.

I. INTRODUCTION

Maritime EO video processing is surveyed as a complement to radar and sonar for autonomous navigation, with emphasis on detection and tracking challenges caused by dynamic scenes and environmental variation. The survey organizes existing approaches and evaluates them using the Singapore Marine Dataset.

  • Radar measurements depend on meteorological conditions and target properties, motivating supplementation with other situational-awareness sensors.
  • EO sensors are video cameras operating in visible or infrared spectra, but their useful range is limited to a few kilometers by atmospheric propagation losses.
  • Maritime EO processing is challenged by water dynamics, viewpoint-dependent object appearance, and changing illumination or weather.The cited challenges include waves, wakes, foams, distance and viewing angle, clouds, sunshine, rain, and glint.
  • The survey presents a taxonomy of maritime EO-processing approaches and organizes the discussion around sensor comparison, datasets, detection, and horizon-related methods.
  • III. OBJECT DETECTION: Object detection processes each video frame independently through horizon detection, background subtraction, and foreground segmentation.The general object-detection pipeline is illustrated in Fig. 2.
  • III. OBJECT DETECTION: Horizon detection remains difficult because maritime images may contain low-contrast targets, weakly defined horizons, or misleading horizontal structures.

1) Projections from edge map:

Projection-based horizon detection extracts edge maps and projects them into Hough or Radon spaces to identify prominent lines, but its reliability depends strongly on horizon contrast and preprocessing. Region-based and hybrid methods provide complementary ways to handle weak or ambiguous horizon evidence, with quantitative evaluation comparing representative approaches on challenging maritime videos.

  • 1) Projections from edge map:: Edge maps are projected into Hough or Radon spaces, where prominent cells identify candidate horizon-line parameters.Hough maps edge pixels into (θ, ρ) curves, while Radon space accumulates line evidence; cells with the largest values determine line parameters.
  • 1) Projections from edge map:: Projection transforms detect clear horizon lines but perform poorly when targets have low contrast or the horizon lacks strong edge features.A skyline can provide rough evidence, while wakes may create competing horizontal lines that are detected as horizons.
  • 2) Region based horizon detection:: Region-based methods smooth local image statistics, model sky and sea distributions, and identify their boundary as the horizon.They may use Gaussian or polynomial functions, discriminant analysis, textures, covariances, or eigenvalues, while assuming suitable statistical representations or learned intensity trends.
  • 3) Hybrid methods:: Hybrid methods evaluate projection-generated candidates using sky–sea region statistics or learned local features, with recent variants combining multi-scale filtering and projection.One hybrid selects the candidate maximizing the Mahalanobis distance between hypothetical sea and sky distributions.
  • 3) Hybrid methods:: Horizon evaluation uses position error |YGT −Yest| and angular error |αGT −αest| across projection-, region-, and hybrid-based approaches.The comparison includes Hough, Radon, MuSMF, ENIW, and FGSL on the Singapore Marine dataset; several methods were implemented because their code was unavailable.

4) Comparison of methods for horizon detection:

Horizon detection methods are compared quantitatively, with projection-based methods performing worst and statistical methods performing better. The survey also situates horizon detection among broader background-subtraction approaches for maritime imagery.

  • Horizon-location error is more severe than angular error across the evaluated methods.
  • Projection-based horizon detectors show the poorest performance, while statistical methods perform better.
  • MuSMF performs best on on-shore videos, whereas FGSL performs best on on-board videos.
  • Maritime background subtraction must address dynamic water in open seas and static structures near ports or harbors.
  • Long-wave infrared reduces water-background dynamicity, giving static background subtraction better performance than in visible imagery.

3) Bayes classifier:

The section presents Bayes- and histogram-based background modeling alongside static subtraction evaluation. On the Singapore Marine dataset, the implemented methods achieved inadequate precision and recall, with resolution and spectral differences limiting suitability.

  • 3) Bayes classifier: A Bayes classifier labels a test pixel using learned likelihoods for background and foreground feature vectors.
  • 3) Bayes classifier: Feature-vector histograms are learned a priori, while supervised learning enforces a target proportion of pixels classified as background.
  • D. Comparison of static background subtraction techniques: Static background subtraction evaluation compares detected-object bounding boxes with annotated ground-truth boxes using intersection over union.
  • D. Comparison of static background subtraction techniques: HistComp and static GMM perform similarly, but neither provides adequate precision and recall on the evaluated videos.
  • D. Comparison of static background subtraction techniques: The methods may be unsuitable for high-resolution maritime imaging because most were tested on low-resolution intensity imagery.

IV. OBJECT TRACKING

Maritime object tracking extends framewise detection with temporal information, dynamic background subtraction, and registration to compensate for camera motion. In the evaluated on-board videos, horizon and cross-correlation registration improved SSIM by 6–7%, while SURF was ineffective.

  • IV. OBJECT TRACKING: Object tracking differs from framewise detection by using temporal information and dynamic background subtraction.
  • B. Registration: Horizon detection supports frame registration to compensate for camera or mounting-base motion caused by vessel turbulence.
  • B. Registration: Yaw shifts remain uncorrected by horizon-based registration, producing horizontal mismatch between consecutive frames.
  • B. Registration: 6–7%: horizon and cross-correlation registration improve SSIM on consecutive on-board frames.
  • B. Registration: SURF feature registration hardly improves SSIM because on-board maritime videos lack reliable stationary features.
  • Dynamic background modeling is more effective than static subtraction when visible-spectrum water motion causes incorrect detections.

1) Relatively stationary pixels:

Relatively stationary pixels can be modeled from temporal intensity distributions, while other approaches use wavelets, probabilistic models, optical flow, or combined techniques. These methods seek to separate changing foreground objects from evolving maritime backgrounds.

  • 1) Relatively stationary pixels: Pixels that remain relatively stationary across recent frames are modeled with unimodal temporal intensity distributions.
  • 1) Relatively stationary pixels: A polynomial fit over a temporal window supports background assignment when the Lp deviation falls below a threshold.
  • 1) Relatively stationary pixels: Wavelet methods suppress clutter or detect objects using low-frequency components and temporal correlation.
  • 1) Relatively stationary pixels: Temporal GMM approaches classify pixels against RGB background models whose thresholds can depend on illumination.
  • 1) Relatively stationary pixels: KDE models dynamic ocean backgrounds with flexible kernels and does not require supervised learning like GMM.
  • 1) Relatively stationary pixels: Optical flow learns motion patterns and warps frames so stationary components can be identified as background, including wakes modeled as background.
  • 1) Relatively stationary pixels: Combined approaches integrate multiple techniques to improve background subtraction, including clustering, region-level updates, and model maintenance.

7) Comparison of techniques for dynamic background subtraction:

Dynamic background subtraction methods generally outperform static approaches, but water dynamics still prevent practically useful precision and recall. Tracking methods span segmentation-based, feature-based, Bayesian, level-set, and Kalman-filter approaches with differing initialization and robustness requirements.

  • Dynamic background subtraction approaches generally outperform static background subtraction, except for the temporal mean approach.
  • Optical-flow subtraction achieves better precision by suppressing tiny water-motion artifacts through frame down-scaling and filtering small foreground regions.
  • Adaptive median subtraction provides the best recall, but no evaluated method offers practically useful precision and recall.
  • Level-set tracking requires the number of foreground objects and their initial contours, making occluded objects difficult to handle.
  • Multi-hypotheses Kalman filtering provides a reported balance between computational load and tracking robustness, but its validity on lower-quality videos is not assured.

6) Motion segmentation using optical flow approach:

Optical-flow motion segmentation tracks maritime objects without explicitly pre-segmenting the foreground by grouping pixels or features with similar motion. Its evaluation uses standard tracking metrics and shows that tracker initialization and water dynamics strongly affect performance.

  • 6) Motion segmentation using optical flow approach:: Motion segmentation incorporates spatial information implicitly by grouping pixels or features with similar motion characteristics, enabling direct foreground tracking.
  • 6) Motion segmentation using optical flow approach:: Optical-flow tracking suppresses wakes and shadows through inconsistent motion maps and separates multiple objects within one foreground blob using modified k-means clustering.
  • 6) Motion segmentation using optical flow approach:: The optical-flow constraint ∇g ·f +gt = 0 estimates pixelwise velocity while minimizing flow error under a locally constant-velocity assumption.
  • 6) Motion segmentation using optical flow approach:: Tracking evaluation reports precision, recall, MOTA, MOTP, and FAR, with matches determined using Hungarian assignment and an IOU threshold of 0.5.
  • 6) Motion segmentation using optical flow approach:: MST, KLT, and DAOT benefit from initial guesses, whereas MOT and LKDoG produce many false positives because of water dynamics; MST performs best among them.

V. COMPUTER VISION APPROACHES BEYOND MARITIME

Computer-vision research offers additional background-suppression and object-tracking models that may transfer to maritime EO data. The survey identifies candidate methods spanning temporal filters, Kalman models, Gaussian mixtures, dynamic textures, and saliency-based approaches.

  • V. COMPUTER VISION APPROACHES BEYOND MARITIME: Maritime EO processing has received less domain-specific algorithm development than general computer vision, motivating evaluation of computer-vision methods with dynamic-water examples.
  • V. COMPUTER VISION APPROACHES BEYOND MARITIME: Object detection in maritime EO data mainly uses background subtraction to obtain and segment foreground regions as objects of interest.
  • V. COMPUTER VISION APPROACHES BEYOND MARITIME: Computer-vision background models include temporal intensity filters, median filtering, low-pass filtering, Wiener filtering, and Kalman-filter estimation.

3) Gaussian Mixture models:

The survey reviews Gaussian-mixture, kernel, optical-flow, texture, saliency, and low-rank approaches for modeling dynamic maritime backgrounds. These methods incorporate temporal variation, spatial texture, motion, or adaptive updates, but some remain computationally expensive or require topology modification.

  • 3) Gaussian Mixture models:: Gaussian mixture models are used in maritime background subtraction and are increasingly combined with other techniques for challenging dynamic environments.
  • 3) Gaussian Mixture models:: Kernel-density background models can use adaptive kernel spreads by changing the number of frames included in dynamic updates.
  • 3) Gaussian Mixture models:: Dynamic-background models may represent each pixel using minimum intensity, maximum intensity, and maximum consecutive-frame intensity difference estimated from a time sequence.
  • 3) Gaussian Mixture models:: Mixtures of dynamic textures showed good ability to handle ocean dynamic texture with synthetic translucent objects and flames.
  • 3) Gaussian Mixture models:: HMMs incorporate temporal continuity but are computationally expensive and difficult to modify dynamically when topology changes.
  • 3) Gaussian Mixture models:: Saliency-based methods model spatial and temporal normality, while robust PCA can use motion saliency to separate stable background and background motion from video.

10) Fuzzy classification of background pixels:

The survey covers fuzzy, subspace, sparse, and computer-vision background models, with comparative evaluation showing strong recall but poor precision for tested CV methods.

  • Fuzzy and learned approaches: Fuzzy methods use adaptive thresholds or neurofuzzy models to classify background pixels and can address illumination changes and shadows.The cited approaches were mainly evaluated in urban and semi-urban settings, but their properties may inform maritime infrared processing.
  • Subspace approaches: Subspace methods compactly represent pixel features before learning and include eigenbackground, PCA, robust PCA, ICA, and related approaches.Matrix decompositions support model learning, compact representation, updating, and feature-overlap analysis.
  • Comparative evaluation: EigHMM achieves the best recall among the compared computer-vision techniques, while all methods show poor precision and many false positives.The comparison includes LBP, LBSP, FuzzGMM, FAdaSOM, and EigHMM; combining LBSP with eigenbackground is suggested as potentially useful.
  • Tracking approaches: Tracking literature includes segmented-object methods such as mixture models and silhouette tracking, alongside approaches that operate without prior segmentation.The survey distinguishes tracking of segmented objects from methods that do not require prior object segmentation.
  • Sparse and low-rank approaches: Sparse reconstruction and low-rank formulations model moving foreground objects as outliers to a background representation.These approaches treat motion as structured deviation from the estimated background model.

2) Temporal persistence and dynamic programming for tracking:

Temporal persistence, dynamic programming, optical flow, and feature tracking model object motion across frames, but practical performance remains constrained by wakes, shadows, initialization, and false positives.

  • Temporal persistence and dynamic programming: Dynamic programming updates GMM parameters for pre-segmented foreground objects, while temporal persistence assumes nearby frames retain similar position and appearance.Temporal persistence treats mobile foreground objects as remaining in spatial vicinity with similar color or intensity across consecutive frames.
  • Learning-based tracking: Boosting-based trackers can learn object motion and handle occlusion by modeling each object independently, although some require manual initialization.The cited approaches include unsupervised, semi-supervised, and machine-learning tracking methods.
  • Motion segmentation: Optical flow groups pixels with similar motion and can incorporate normalized color or dense-flow partitioning to address large motion variations.These methods use spatial motion coherence to support foreground-object detection and segmentation.
  • Feature-based motion segmentation: Feature-tracking methods cluster sparse points by motion, require rigid objects and sufficiently limited noise, and are less computationally intensive than optical flow.They can handle random and large motion variations without pre-learned features, but the noise assumption may not hold consistently.
  • Low-rank motion modeling: Low-rank motion modeling can jointly suppress background, segment foreground, and segment motion, but reported results show ineffective suppression of wakes and shadows.The limitation is specifically reported for the approach modeled with contiguous Markov random fields.
  • Comparative evaluation: KCF performs best across all reported tracking metrics and is fast, while AdaBoost and Medflow also generally outperform the other compared methods.The comparison uses computer-vision trackers implemented through the OpenCV tracker library.
  • Concluding remarks: The survey concludes that dynamic background methods generally handle wakes, clouds, and foams better, while computer-vision techniques may aid maritime vision with suitable advancement.Quantitative evaluation is conducted on the Singapore Marine Dataset, created as a challenging benchmark for maritime EO videos.

APPENDIX

The appendix discusses post-processing, vessel classification, multisensor EO systems, and commercial architectures that combine EO data with radar, AIS, sonar, or other sensors.

  • Post-processing: Post-processing can convert vessel positions and speeds from image measurements into physical units using calibrated camera geometry.The described setup uses high-mounted stationary cameras and maps image pixels to physical points on a flat water surface.
  • Post-processing: The geometric approximation assumes the object lies at the horizon and is valid only for points at far distances.For other spatial points, the angle between the point and camera differs.
  • Vessel classification: Vessels may be classified by shape, size, speed, and visibility, ranging from coarse size categories to exact vessel types.The survey gives examples from swimmers and jet skis through cruise ships and cargo ships.
  • Vessel classification: Shape-library classification compares segmented vessel shapes against stored references, requiring coverage across orientations and spatial resolutions.The library must balance generative robustness with discriminative specificity and may require shape fitting and curvature analysis.
  • Vessel classification: Feature-based classification represents vessels with discriminative descriptors including SIFT, Haar, Fisher vectors, and statistical moments.Its performance depends on selecting suitable features for shape classification.
  • Multisensor systems: Multisensor systems process EO data alongside radar, sonar, gyroscopes, motion sensors, weather sensors, or other modalities for detection and tracking.Examples include stereoscopic EO-sonar reconstruction and visible–infrared image fusion.
  • Commercial systems: Commercial systems use EO sensors within larger architectures: VIPS combines AIS, radar, and EO, while an anti-collision system augments EO and compass data with AIS and radar.VIPS uses AIS and radar to provide candidate locations before the calibrated EO system zooms into those areas.
  • Commercial systems: The anti-collision system compares temporal object characteristics with reference and dangerous-object databases every 30 seconds while maintaining 20 minutes of history.The characteristics include azimuth, size, and after-glow patterns.
Loading 1611.05842v1…