A CNN-RNN Architecture for Multi-Label Weather Recognition

1. Introduction

• Weather recognition is relevant to daily activities and computer-vision applications, and image-based recognition offers an alternative to expensive hardware requiring professional maintenance.The text notes potential applications including image retrieval, restoration, outdoor surveillance, robotic vision, and vehicle assistant driving.Source: §1 · p. 2

1.1. Motivation and Overview

• Single-label treatment is inappropriate because weather boundaries can be ambiguous and one label may omit coexisting conditions.Intermediate sunny-cloudy states make labels difficult to determine, while a haze label alone cannot distinguish images that also appear sunny, overcast, or snowy.Source: §1.1 · pp. 2–3
• The proposed CNN-RNN treats weather recognition as multi-label prediction, using channel-wise attention and convolutional LSTM to model label dependencies step by step.The attention model seeks discriminative weather cues, while the convolutional LSTM preserves spatial information as it predicts labels.Source: §1.1 · pp. 4–5

1.2. Contributions

• The paper constructs two datasets: one with about 8K images across seven weather categories and another built from scratch with 10K images.The first dataset is transformed from an existing transient attribute dataset.Source: §1.1 · p. 5

1.3. Organization

• The paper reviews weather-recognition work, presents its approach and dataset construction, analyzes experiments, and concludes.Source: §1.3 · pp. 5–6

2. Related Work

• The review groups weather-recognition work into hand-crafted-feature approaches and approaches that use CNNs.Source: §2 · p. 6

2.1. Weather Recognition with hand-crafted features

• Hand-crafted weather-recognition work uses varied cues and methods, but its features are often limited to specific perspectives or weather classes.The review includes fog-detection methods and broader weather features; such approaches may not apply across a wider range of applications.Source: §2.1 · pp. 6–7

2.2. Weather Recognition with CNNs

• CNNs have been applied to weather recognition, including fine-tuned AlexNet and combinations of hand-crafted and CNN features.The review also notes weather recognition with CNNs has largely treated the task as single-label, despite simultaneous weather conditions and resulting information loss.Source: §2.2 · pp. 7–8

3. Our Approach

• The approach frames weather recognition as multi-label classification and predicts labels stepwise with a CNN, channel-wise attention, and convolutional LSTM.Source: §3 · pp. 8–9

3.1. The Convolutional LSTM in the CNN-RNN Architecture

• Unlike standard LSTM flattening, convolutional LSTM uses 3D states and convolutional transformations to preserve feature spatial information.Source: §3.1 · p. 10
• The convolutional LSTM focuses on critical regions for weather-label prediction and outperforms common LSTM both with and without spatial attention.Source: §3.1 · p. 10

3.2. Channel-wise Attention Model in the CNN-RNN Architecture

• Channel-wise attention uses global-average-pooled visual features and previous hidden-state information to adaptively recalibrate channels for each weather-label prediction.Source: §3.2 · pp. 10–12

3.3. Inference

• Inference predicts labels in a fixed order based on their co-occurrence relationships, flattening the convolutional LSTM hidden state at each step to produce a label prediction.Source: §3.3 · p. 12

3.4. Training Details

• Training proceeds in two stages: fine-tune VGGNet as a sigmoid multi-label CNN, then remove its fully connected layers, fix its remaining parameters, and train convolutional LSTM and attention from extracted features.Source: §3.4 · p. 13
• The authors report that a previously described strategy cannot improve performance.The supplied excerpt does not identify which strategy it refers to.Source: §3.4 · p. 13
• Training uses 256 × 256 images, random flips, crops and noise, and shuffled mini-batches of 50.The mini-batches are formed before each training epoch.Source: §3.4 · p. 13
• Table 1 lists the shapes of critical components in the proposed CNN-RNN architecture.The excerpt says the shapes of all biases can be inferred, but does not provide the component shapes.Source: Table 1 · p. 14, §3.4 · p. 14

4.1.1. The Transient Attribute Dataset

• The transformed transient attribute dataset selects six weather attributes—sunny, cloudy, fog, snow, moist, and rain—and adds an “other” class for examples where every attribute strength is below 0.5, yielding seven weather classes.Source: §4.1.1 · pp. 14–15

4.1.2. The Multi-Label Weather Classification Dataset

• The new dataset contains 10,000 images across five weather classes, covers common conditions, and uses non-exclusive weather labels.Its varied scenes, scales, and views add diversity; the authors say most existing datasets cover only two or three weather classes.Source: §4.1.2 · pp. 15–16
• Cloudy is the largest class in both datasets, and more than half of samples have multiple weather labels in each; the new dataset is more balanced than the transient dataset.The authors attribute cloudy’s prevalence to its co-occurrence with other weather conditions.Source: §4.1.2 · p. 17

4.2. Co-occurrence Relationships

• Co-occurrence analysis found weather-label dependencies, including rainy/cloudy and snowy/foggy pairs near category boundaries; the method represents such samples with combined labels and predicts labels stepwise.The authors report that alternative label orders performed comparably, while the tested orders achieved the best performance in most cases.Source: §4.2 · p. 18

4.3. Evaluation Metrics and Comparison Approaches

• Average precision (AP) and average recall (AR) are defined as the means of per-class precision and recall, respectively.Per-class precision and recall are computed first.Source: §4.3 · p. 19

4.4. Results on the Transient Attribute Dataset

• CNN-Att-ConvLSTM achieves the best OP, OR, and OF1 and comparable AP, AR, and AF1 on the transient attribute dataset, outperforming the listed baselines overall.The comparison includes multi-label AlexNet and VGGNet, ML-KNN, ML-ARAM, and other CNN-RNN methods.Source: Table 3 · p. 20, §4.4 · p. 21

4.3. Evaluation Metrics and Comparison Approaches

• With no other multi-label weather-recognition approaches available, the study compared multi-label AlexNet and VGGNet, plus CNN-LSTM, CNN-LSTM with spatial attention (CLA), CNN-GRU with spatial attention (CGA), and CNN-ConvLSTM without channel-wise attention.Source: §4.3 · p. 20

4.5. Results on the Multi-label Weather Classification Dataset

• CNN-Att-ConvLSTM performs best on almost all evaluation metrics on the multi-label weather classification dataset.The reported comparison is presented in Table 4.Source: Table 4 · p. 21, §4.5 · p. 21

4.4. Results on the Transient Attribute Dataset

• On the transient attribute dataset, CNN-LSTM with spatial attention performs well, while removing spatial attention causes serious degradation; CGA and CNN-ConvLSTM achieve results similar to CLA.The authors associate this with the importance of key regions and the effectiveness of convolutional LSTM for extracting information from them.Source: §4.4 · p. 21
• Transient-dataset methods perform worse on rainy and other classes, attributed to distant views that obscure rain and the rarity of other-class samples.Other samples can be misclassified as sunny or cloudy.Source: §4.4 · p. 21

4.5. Results on the Multi-label Weather Classification Dataset

• Figure 9 examples show attention highlighting relevant weather cues, but the approach misses labels when annotations are ambiguous or visual cues are inconspicuous.It succeeds on some examples but misses sunny in a sunny-cloudy image and predicts cloudy and foggy rather than cloudy and rainy when wet ground is not obvious.Source: Figure 9 · p. 22, §4.5 · pp. 22–23

• Multiple weather conditions can occur simultaneously, so the paper proposes assigning an image multiple weather categories for a more comprehensive description.Source: §5 · p. 23
• The proposed CNN-RNN extends the CNN with channel-wise attention to extract correlated visual features and uses convolutional LSTM to predict labels step by step while maintaining spatial information.Source: §5 · p. 23
• The authors build two weather-recognition datasets to address the lack of training data.Source: §5 · p. 23
• The experimental results verify the effectiveness of the proposed approach.Source: §5 · p. 23
• Future work will predict weather-class strengths and use other modalities, including humidity and temperature.Source: §5 · p. 23