PaperlayerView exact revision on arXiv

Source-linked AI summary

Segmenting Objects in Day and Night:Edge-Conditioned CNN for Thermal Image Semantic Segmentation

Chenglong Li, Wei Xia, Yan Yan, Bin Luo, Jin Tang

arXiv:1907.10303v1cs.CV

TL;DR

Thermal semantic segmentation remains difficult because visible cameras fail under darkness and adverse conditions, although thermal imaging offers robustness in those settings. The paper proposes EC-CNN with gated edge-prior conditioning and introduces SODA for benchmarking; experiments report improvements from synthetic-data training and the gate mechanism.

  • Problem

    Visible cameras are limited by darkness, illumination variation, haze, smog, and thermal segmentation challenges such as low resolution, contrast, ambiguous boundaries, and sensor noise.

  • Method

    EC-CNN uses a gated feature-wise transform layer to adaptively embed hierarchical edge priors, alongside the SODA thermal segmentation benchmark.

  • Results

    Pixel Acc rises from 63.4% to 72.3% and mIoU from 24.8% to 36.6% when synthetic thermal training replaces random initialization; adding the gate raises Pixel Acc from 82.3% to 83.6% and mIoU from 60.8% to 61.3%.

  • Takeaways & Limitations

    SODA supplies a benchmark for thermal semantic segmentation, while EC-CNN demonstrates the reported effectiveness of adaptive edge guidance and synthetic thermal training.

Abstract

from arXiv · show

Despite much research progress in image semantic segmentation, it remains challenging under adverse environmental conditions caused by imaging limitations of visible spectrum. While thermal infrared cameras have several advantages over cameras for the visible spectrum, such as operating in total darkness, insensitive to illumination variations, robust to shadow effects and strong ability to penetrate haze and smog. These advantages of thermal infrared cameras make the segmentation of semantic objects in day and night. In this paper, we propose a novel network architecture, called edge-conditioned convolutional neural network (EC-CNN), for thermal image semantic segmentation. Particularly, we elaborately design a gated feature-wise transform layer in EC-CNN to adaptively incorporate edge prior knowledge. The whole EC-CNN is end-to-end trained, and can generate high-quality segmentation results with the edge guidance. Meanwhile, we also introduce a new benchmark dataset named "Segment Objects in Day And night"(SODA) for comprehensive evaluations in thermal image semantic segmentation. SODA contains over 7,168 manually annotated and synthetically generated thermal images with 20 semantic region labels and from a broad range of viewpoints and scene complexities. Extensive experiments on SODA demonstrate the effectiveness of the proposed EC-CNN against the state-of-the-art methods.

I. INTRODUCTION

Thermal image semantic segmentation addresses visible-camera limitations in adverse conditions, while EC-CNN and SODA provide a baseline, dataset, and benchmark for the task.

  • Thermal cameras operate in darkness and resist illumination variation, shadows, haze, and smog, motivating thermal semantic segmentation.
  • SODA contains more than 2,000 manually annotated and 5,000 synthetically generated thermal images with 20 semantic labels and varied viewpoints and scene complexities.The dataset is intended as a standard benchmark for thermal semantic segmentation.
  • The study benchmarks EC-CNN and eight state-of-the-art segmentation methods using mean IoU, pixel accuracy, and runtime on SODA.The experiments also analyze results to identify research directions for thermal image segmentation.
  • The paper identifies thermal image semantic segmentation as an emerging task and presents EC-CNN as its proposed baseline algorithm.
  • EC-CNN adaptively embeds edge priors through a gated feature-wise transform layer to improve thermal segmentation.The gate is designed to reduce the effects of noisy edge information.

II. RELATED WORK

Related work covers foundational and contemporary semantic segmentation architectures, edge-aware methods, and the paper’s edge-conditioned design for thermal imagery.

  • Image Semantic Segmentation: Fully Convolutional Networks established end-to-end convolutional semantic segmentation with fully resolved output maps.
  • Image Semantic Segmentation: Encoder-decoder, pyramid, CRF, spatial-pyramid, and atrous-convolution methods extend segmentation by combining context with recovered object detail.
  • Prior edge-aware methods use edge-preserving filtering, CRF pairwise features, or boundary detection to improve semantic boundary localization.
  • Unlike prior work, EC-CNN conditions thermal semantic segmentation on object edges to guide the segmentation process.

B. Conditioned Networks

The paper combines conditional feature transformations with adversarial image-to-image translation to support thermal image semantic segmentation and synthetic training-data generation.

  • Conditional feature normalization uses learned conditions for feature-wise affine transformations, extending across tasks including image super-resolution and semantic segmentation.
  • FiLM conditions feature maps using learned scaling and shifting, while spatially aware variants are more suitable for dense prediction.
  • The paper uses adversarial image-to-image translation to create synthetic thermal training data from RGB semantic-scene imagery.
  • EC-CNN is introduced to use edge priors for thermal image semantic segmentation.

A. EC-CNN Architecture

EC-CNN couples an EdgeNet with a DeepLabv3-based segmentation network, embedding hierarchical edge features through GFT layers to guide thermal-image boundaries.

  • EC-CNN has two streams: an EdgeNet for edge detection and a semantic segmentation network.
  • Hierarchical edge features are used to guide segmentation because thermal images can have low resolution, low contrast, ambiguous boundaries, and sensor noise.
  • GFT layers embed EdgeNet feature maps into DeepLabv3 so edge priors guide thermal semantic segmentation.
  • The architecture adapts edge-map resolution and channel count before combining edge information with segmentation features.
  • Figure 4 compares DeepLabv3, ungated EC-CNN, and gated EC-CNN outputs to illustrate the proposed components.

B. GFT Layer

The GFT layer extends feature-wise conditioning to spatial edge priors and adds gating so noisy edge information can be incorporated adaptively.

  • FiLM applies condition-dependent channel scaling and shifting, but cannot represent spatially varying conditions because each channel receives one parameter pair.
  • SFT maps conditioning inputs to spatially shaped scaling and shifting tensors, making spatial information available for feature transformation.
  • GFT combines feature modulation, information control, and feature-wise transformation to embed edge priors into segmentation features.
  • GFT adds an information-control gate because EdgeNet can produce noisy edges that would otherwise affect segmentation performance.
  • The GFT structure is presented alongside FiLM and SFT as the paper’s edge-conditioned feature-transform design.

C. Loss Function

EC-CNN trains semantic segmentation with pixelwise cross-entropy using predicted label probabilities and ground-truth labels.

  • The loss is pixelwise cross-entropy between EC-CNN label-assignment probabilities and ground-truth labels.

D. Network Training

The paper constructs SODA by combining manually annotated real thermal images with synthetically generated thermal images. Synthetic thermal data are produced by translating labeled RGB imagery while retaining its semantic labels.

  • SODA combines 2,168 manually annotated thermal images with 5,000 synthetically generated thermal images.The real images were captured in real scenes, while synthetic data address annotation cost and noise concerns.
  • The real subset covers 20 semantic region labels across indoor and outdoor categories and varied viewpoints, occlusions, blur, and resolutions.Examples include person, building, tree, road, car, chair, and sky.
  • The real images were annotated with labelme by 12 experts and subjected to periodic and second-round quality checks.The selection process retained 2,168 usable images from more than 4,000 submissions.
  • Synthetic thermal images are generated by translating labeled Cityscapes RGB images with pix2pixHD, allowing their original semantic labels to transfer directly.Cityscapes was selected for its relevance to the real thermal scenes.

C. Training and Testing Sets

The training setup uses translated thermal data alongside the manually annotated SODA split, with specified initialization, optimization, and implementation settings. The supplied passages also identify the benchmark dataset tables and training configuration.

  • The manually annotated SODA subset is split into 1,168 training images and 1,000 testing images, while all 5,000 translated thermal images join training.The source Cityscapes data comprise 2,975 training, 500 validation, and 1,525 test images before translation.
  • The paper reports dataset comparisons and model-initialization results in Tables I and III, including the best outcome from combining ImageNet pretraining with a generated thermal model.The supplied table captions identify these scopes but do not provide their cell values.
  • The implementation uses PyTorch on an NVIDIA GeForce GTX TITAN XP GPU with 12GB memory.
  • Thermal-image generation uses pix2pixHD trained from scratch for 20 epochs with batch size 1 and learning rate 0.0002.Weights are randomly initialized from a Gaussian distribution with mean 0 and standard deviation 0.02.
  • The segmentation branch uses a poly learning-rate policy with base learning rate 0.001, power 0.9, 100 epochs, batch size 14, and 480×480 inputs.Momentum is 0.9 and weight decay is 0.0001.

B. Evaluation Metric

Evaluation uses pixel accuracy and mean IoU, with background included as a category in the VOC-like mIoU calculation. EC-CNN is compared with eight state-of-the-art segmentation methods on SODA and achieves the strongest reported benchmark result.

  • The evaluation metrics are pixel accuracy and mean IoU, with the background counted as one category in the VOC-like mIoU evaluation.
  • The benchmark compares EC-CNN with U-Net, ERFNet, FCN-16s, FCN-32s, Deeplabv2, DUC, GCN, and DeepLabv3.ResNet-101 is used as the backbone, and GFT layers are applied in conv3 x.
  • 61.9% mean IoU is achieved by EC-CNN, exceeding DeepLabv3 by 4.8% and GCN by 10.86 percentage points on SODA.DeepLabv3 and GCN are identified as the second- and third-best methods, respectively.
  • Qualitative results show more precise EC-CNN predictions despite low resolution, low contrast, ambiguous thermal-crossover boundaries, and sensor noise.The comparison is visualized in Fig. 10.

D. Ablation Study

The ablations examine synthetic-data pretraining, GFT placement, gating, and inference speed. Results support combining synthetic thermal data with ImageNet pretraining, inserting GFT in shallower blocks, retaining the gate, and obtaining accuracy gains with modest speed impact.

  • Quality of thermal image generation: pix2pixHD generates better thermal images than pix2pix in the qualitative synthetic-data comparison.The paper uses this observation to support integrating generated thermal images into training.
  • Impact of synthetic thermal data: Pixel Acc rises from 63.4% to 72.3% and mIoU from 24.8% to 36.6% when synthetic thermal pretraining replaces random initialization.ImageNet plus synthetic thermal pretraining performs best among the tested initialization strategies.
  • Impact of GFT layer: GFT placement in shallower convolutional blocks produces better results, consistent with edge priors being low-level features.The paper reports this pattern for both ResNet-50 and ResNet-101.
  • Impact of gate mechanism: Adding the gate improves Pixel Acc from 82.3% to 83.6% and mIoU from 60.8% to 61.3% over SFT.The authors associate this improvement with reducing noise from EdgeNet while selecting useful edge information.
  • Comparison of inference speed: EC-CNN with ResNet-50 requires about 0.13–0.17 seconds per image, indicating modest inference-speed impact relative to the baseline.Similar time-consumption observations are reported for ResNet-101.

VI. CONCLUDING REMARK

The paper introduces SODA for thermal image semantic segmentation and proposes EC-CNN with explicit edge priors. Evaluation supports edge guidance for difficult thermal regions, while the authors identify limited annotation scale and complementary RGB information as important future considerations.

  • Dataset: SODA provides a new thermal semantic segmentation benchmark with 2,168 richly annotated images, 20 semantic labels, and additional Cityscapes-based synthetic thermal images.The dataset supports extensive experimental analysis of leading approaches and the proposed method.
  • Method: EC-CNN explicitly incorporates edge priors, and experimental results demonstrate its effectiveness for thermal image semantic segmentation.
  • Findings: Edge priors are especially advantageous for low-resolution, low-contrast, boundary-ambiguous regions and thermal-sensor imaging noise.
  • Limitations: SODA is a first dataset for this task, but its well-annotated data remain insufficient for developing deep learning methods.
  • Future directions: RGB and thermal information are complementary: thermal sensing helps in poor lighting and bad weather, whereas RGB better separates subjects under thermal crossover.
Loading 1907.10303v1…