Source-linked AI summary
Color-based Segmentation of Sky/Cloud Images From Ground-based Cameras
Soumyabrata Dev, Yee Hui Lee, Stefan Winkler
TL;DR
Sky/cloud segmentation is difficult because cloud appearance and illumination vary, while prior methods lack systematic channel analysis and often require manually defined parameters. The paper develops a supervised PLS-based framework with probabilistic pixel outputs and releases the SWIMSEG annotated database. Its reported contribution is a learning-based, parameter-free approach evaluated across two databases, with performance influenced by database characteristics and imaging calibration.
Problem
Existing cloud-segmentation methods lack systematic color-channel analysis and commonly rely on manually defined parameters and binary labels.
Method
The paper uses systematic color-channel analysis and PLS regression in an entirely learning-based supervised segmentation framework.
Results
The framework provides probabilistic cloud-pixel classification and generally performs better on SWIMSEG than HYTA, while color-channel behavior is similar across datasets.
Takeaways & Limitations
The approach avoids manually defined thresholds, conditions, and parameters while representing each pixel’s degree of belonging to sky or cloud.
Abstract
from arXiv · showhide
Sky/cloud images captured by ground-based cameras (a.k.a. whole sky imagers) are increasingly used nowadays because of their applications in a number of fields, including climate modeling, weather prediction, renewable energy generation, and satellite communications. Due to the wide variety of cloud types and lighting conditions in such images, accurate and robust segmentation of clouds is challenging. In this paper, we present a supervised segmentation framework for ground-based sky/cloud images based on a systematic analysis of different color spaces and components, using partial least squares (PLS) regression. Unlike other state-of-the-art methods, our proposed approach is entirely learning-based and does not require any manually-defined parameters. In addition, we release the Singapore Whole Sky IMaging SEGmentation Database (SWIMSEG), a large database of annotated sky/cloud images, to the research community.
I. INTRODUCTION
Ground-based whole sky imagers provide high-resolution local cloud observations, but cloud segmentation remains difficult under changing shapes and illumination. The paper addresses gaps in systematic color-channel analysis and manually tuned methods through a learning-based framework, database, and evaluation.
- Ground-based whole sky imagers offer high-resolution data for localized and short-term cloud analysis.
- Cloud segmentation is challenging because clouds are nonrigid, feature-less, poorly defined, continuously changing, and observed under varied lighting.
- Existing methods have not systematically studied color-model and channel selection and commonly rely on manually defined parameters and binary labels.
- The paper proposes a robust, parameter-free supervised framework based on partial least squares regression that assigns pixels a degree of sky/cloud belongingness.
- The paper contributes SWIMSEG, a large public database with segmentation masks, and evaluates color components and the framework across two databases.
II. APPROACH & TOOLS
The approach evaluates color representations statistically before selecting discriminatory channels for supervised segmentation. It combines PCA-based correlation and variance analysis with ROC-based discrimination assessment.
- Principal Component Analysis: PCA evaluates correlations among channels and identifies components that capture the most variance.
- Color channels: The study considers 16 color channels spanning RGB, HSV, YIQ, L∗a∗b∗, red-blue combinations, and chroma.
- Color channels: The analysis includes L∗a∗b∗ for perceptual uniformity and chroma because clouds tend to be achromatic.
- Principal Component Analysis: Each image’s channel vectors are stacked into a matrix and normalized using channel-specific means and standard deviations before covariance analysis.
- Principal Component Analysis: Eigenvalues and eigenvectors quantify the relative contributions of color channels to orthogonal axes, while primary-component loading factors indicate channel suitability.
C. Discrimination Threshold
The framework identifies discriminative color components and uses them as predictors in a probabilistic PLS regression formulation for pixel-level sky/cloud segmentation. Testing produces normalized values that represent class belongingness rather than thresholded binary decisions.
- Discrimination Threshold: ROC curves evaluate true- and false-positive fractions across discrimination thresholds, and area under the curve identifies useful color components.
- Discrimination Threshold: Statistical analysis selects the best-performing color channels from 16 candidates for use as discriminatory features.
- Probabilistic Segmentation: The segmentation task predicts sky or cloud labels from selected color-channel features and manually annotated ground truths using PLS regression.
- Probabilistic Segmentation: PLS decomposes predictors and responses into latent variables while maximizing covariance between their score representations.
- Probabilistic Segmentation: The learned regression coefficient matrix maps test features to segmentation outputs, which are normalized to [0, 1] without thresholding.
IV. SKY/CLOUD IMAGE DATABASES
An annotated dataset of suitable images is identified as a prerequisite for rigorous evaluation of segmentation algorithms.
- Annotated image datasets are a prerequisite for rigorously evaluating segmentation algorithms.
A. HYTA Database
HYTA is presented as the available publicly annotated sky/cloud image database, containing 32 images with binary segmentation masks collected from varied sources and conditions.
- HYTA was the only publicly available sky/cloud dataset with annotated segmentation masks at the time described.
- The database contains 32 distinct images covering different sky/cloud conditions, each paired with a binary ground-truth segmentation mask.
- HYTA combines ground-camera images from Beijing and Conghua with additional images obtained from other cameras and locations.
- Figure 1 presents HYTA sample images alongside their corresponding sky/cloud segmentation ground truth.
B. SWIMSEG Database
SWIMSEG is a 1013-image Singapore database built from WAHRSIS captures, with corrections for illumination, color, and lens distortion and expert-created segmentation masks.
- SWIMSEG contains 1013 images captured by the custom WAHRSIS ground-based sky imager in Singapore.WAHRSIS uses a Canon EOS Rebel T3i camera and a 180-degree circular fish-eye lens.
- The database corrects illumination and color distortions caused by the imaging system before preparing the images for analysis.Vignetting darkens image edges, while weather and capture time alter recorded colors; corrections use integrating-sphere analysis and an 18-patch color checkerboard.
- Ray tracing generates undistorted 600 × 600 images by projecting the captured image onto a hemisphere and sampling it from user-defined viewing directions.Each output image represents a 62° viewing angle, and varying azimuth angles produces several undistorted images.
- Segmentation masks were created in consultation with cloud experts from the Singapore Meteorological Services.
- The 1013 selected patches span October 2013 to July 2015 and vary in sun angles, time of day, cloud coverage, and camera elevation.Camera elevation angles are 36°, 45°, and 60°; Figure 5 characterizes distributions by time, cloud coverage, and distance from the sun.
V. EXPERIMENTAL EVALUATION
The evaluation analyzes color-channel structure and selects discriminatory channels before applying a PLS-based segmentation procedure. PCA and channel-level comparisons support using a single color channel, followed by probabilistic pixel outputs and standard binary-segmentation metrics.
- Color-channel analysis: PCA identifies correlations among 16 color channels and the channels capturing most input variance.The analysis uses PCA to examine channel relationships and guide channel selection.
- Color-channel analysis: 85.6% and 85.4% of total variance is captured by the first two principal components in HYTA and SWIMSEG, respectively.These values refer to the concatenated image distributions in the two databases.
- Color-channel analysis: The two databases show consistent color-channel relationships, with c11 differing most because its red/green axis contains mostly noise for sky-cloud discrimination.Several channel pairs, including c5/c15 and c8/c14, are highly correlated.
- Color-channel analysis: Red/blue-ratio channels c13 and c15 and saturation c5 rank highly, whereas c4 and c11 are less discriminatory.The loading-factor and ROC analyses are largely consistent across HYTA and SWIMSEG, except for c11.
- Segmentation procedure: The evaluation represents each image with an individual color channel and uses PLS regression to compute continuous test outputs for segmentation.Training images estimate the regression coefficient matrix, which is then applied to test images.
- Segmentation procedure: Normalized outputs in [0,1] indicate pixel belongingness to cloud or sky, and performance is reported with Precision, Recall, F-score, and Misclassification rate.The objective evaluation is based on TP, TN, FP, and FN samples.
A. Choice of Color Channels
The study evaluates individual color channels for segmentation using cross-validation on HYTA and SWIMSEG. Both databases show the same performance trend, with c5 and red-blue ratio channels c13 and c15 performing best; c15 is selected for subsequent evaluations.
- Experimental setup: Individual color channels are evaluated as feature vectors, with training and test sizes defined separately for HYTA and SWIMSEG.The experiment isolates the effect of color-channel choice on segmentation.
- Experimental setup: F-scores are computed using 5-fold cross-validation for HYTA and 10-fold cross-validation for SWIMSEG.Each fold alternates between testing and training roles.
- Results: Individual-channel classification performance follows the same trend in both HYTA and SWIMSEG.The comparison is shown in Figure 9.
- Results: Channels c5 and c13/c15 perform best, consistent with the earlier color-channel analysis.The subsequent evaluations use c15 as the feature vector.
B. Effect of Database Characteristics
The study examines whether time of day, cloud coverage, and distance from the sun affect segmentation F-scores. Performance is lower around midday, marginally better with greater cloud coverage, and shows no clear relationship with sun distance.
- Time of day: Average F-score is slightly higher in the morning and late afternoon but drops during midday.The paper attributes the midday drop to greater overexposure from direct sun.
- Cloud coverage: Segmentation performance is marginally better for images with higher cloud coverage.The relationship is evaluated through the distribution of F-scores across cloud percentages.
- Distance from the sun: Distance from the sun has no clear relationship with the obtained F-score values.This comparison is illustrated in Figure 10(c).
C. Benchmarking
The paper benchmarks its learning-based segmentation approach against thresholding, superpixel, and feature-based methods on HYTA and SWIMSEG, while analyzing computational cost and practical behavior across image conditions.
- Compared methods: The benchmark compares the proposed method with four state-of-the-art cloud detectors and SLIC superpixels, plus SVM-based feature classifiers.The evaluated descriptors include LBP, colorHIST, and dSIFT, combined with SVM classifiers.
- Output representation: The proposed method produces probabilistic cloud membership values before thresholding them into binary maps for evaluation against available binary ground truth.Lower thresholds indicate a greater tendency to classify a pixel as cloud, while higher thresholds indicate the same tendency more weakly.
- Computation: The method averages 1.31s per HYTA image and 1.89s per SWIMSEG image, with training times of 21.5s and 1018.6s, respectively.These measurements were obtained on a 64-bit Ubuntu 14.04 LTS workstation with an Intel i5 CPU at 2.67GHz.
- Quantitative evaluation: The proposed approach achieves the best F-scores on both HYTA and SWIMSEG, whereas existing threshold-based methods trade high precision against high recall.Long et al. and Mantelli-Neto et al. obtain high recall but lower precision, while Souza et al. shows the opposite pattern.
- Quantitative evaluation: Threshold-based methods can under- or over-identify cloud pixels because a single manually selected threshold may not fit all sky/cloud image types.The proposed framework instead learns from the database and avoids manually defined thresholds, conditions, and parameters.
- Database behavior: SWIMSEG images generally yield better performance than HYTA images, which the authors associate with SWIMSEG’s camera calibration for color, illumination, and geometry.The behavior of color channels remains similar across the two databases.
- Scope boundary: The database does not support studying weather-condition effects because Singapore’s weather is relatively constant and overcast cases often make segmentation unnecessary.The authors suggest simple preprocessing for completely overcast images.
VIII. CONCLUSIONS
The paper concludes by combining a systematic, probabilistic PLS-based segmentation method with a calibrated, annotated sky/cloud database. It also identifies probabilistic ground truth and HDR imagery as future extensions.
- Contributions: The proposed method systematically analyzes color spaces and components and performs probabilistic segmentation using PLS-based regression.It is described as entirely learning-based and requires no manually defined thresholds, conditions, or parameters.
- Contributions: The paper releases an extensive sky/cloud image database captured by a calibrated ground-based camera and annotated with ground-truth segmentation masks.The database is presented as a research resource accompanying the segmentation framework.
- Future work: Future work includes annotating probabilistic ground-truth segmentation maps and extending the method to High-Dynamic-Range images.The authors also identify cloud-type classification, altitude estimation, and movement estimation as ongoing research directions.