PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Smoke Segmentation

Feiniu Yuan, Lin Zhang, Xue Xia, Boyang Wan, Qinghua Huang, Xuelong Li

arXiv:1809.00774v1cs.CVeess.IV

TL;DR

Smoke’s fuzzy appearance makes accurate segmentation and manual annotation difficult. The paper proposes a two-path FCN and synthetic smoke images; comparisons report better performance across synthetic and realistic datasets, while smoke variability and blurry edges remain challenges.

  • Problem

    Smoke’s fuzzy edges and translucent appearance make accurate segmentation and manual annotation difficult.

    Source: §I · p. 1, §III · p. 3

  • Method

    A deep coarse-context FCN and a shallower fine-detail FCN are fused for end-to-end smoke segmentation, with synthetic images generated by composing smoke and background images.

    Source: §I · p. 2, §III · p. 3

  • Results

    Comparisons across three synthetic datasets and a realistic dataset report better performance; on DS01, adding path 2 raises mIoU from 64.64% to 71.04%.

    Source: D. Ablation analysis · p. 10, §VI · p. 11

  • Takeaways & Limitations

    Synthetic image composition addresses the difficulty and time required to manually annotate smoke images.

    Source: §III · p. 3

    Smoke feature variation, blurry edges, and visual overlap with fog and clouds remain obstacles to perfect segmentation effects.

    Source: F. Limitations of our method · p. 11

Abstract

from arXiv · show

Inspired by the recent success of fully convolutional networks (FCN) in semantic segmentation, we propose a deep smoke segmentation network to infer high quality segmentation masks from blurry smoke images. To overcome large variations in texture, color and shape of smoke appearance, we divide the proposed network into a coarse path and a fine path. The first path is an encoder-decoder FCN with skip structures, which extracts global context information of smoke and accordingly generates a coarse segmentation mask. To retain fine spatial details of smoke, the second path is also designed as an encoder-decoder FCN with skip structures, but it is shallower than the first path network. Finally, we propose a very small network containing only add, convolution and activation layers to fuse the results of the two paths. Thus, we can easily train the proposed network end to end for simultaneous optimization of network parameters. To avoid the difficulty in manually labelling fuzzy smoke objects, we propose a method to generate synthetic smoke images. According to results of our deep segmentation method, we can easily and accurately perform smoke detection from videos. Experiments on three synthetic smoke datasets and a realistic smoke dataset show that our method achieves much better performance than state-of-the-art segmentation algorithms based on FCNs. Test results of our method on videos are also appealing.

I. INTRODUCTION

Smoke segmentation requires detailed masks despite smoke’s variable appearance and fuzzy boundaries. The paper addresses this with coarse and fine FCN paths, a small fusion network, and synthetic training images.

I. INTRODUCTION

  • Unlike whole-image smoke recognition, smoke segmentation must locate smoke accurately and produce masks with detailed edges.The paper describes segmentation as more difficult because smoke appearance varies substantially and many existing approaches rely on video.Source: §I · p. 1
  • A deep encoder-decoder FCN captures global context for coarse smoke localization, while a shallower encoder-decoder captures local detail for fine segmentation.Skip structures help address scale variation by fusing features rather than feeding differently sized images.Source: §I · pp. 1–2
  • A small add-convolution-activation network fuses both paths’ outputs and enables end-to-end simultaneous optimization.Skip-connected feature maps carry multi-scale coarse and fine information for the fusion network.Source: §I · p. 2
  • Synthetic smoke-image generation is proposed to address the lack of unified segmentation datasets and the difficulty of manually labeling realistic smoke.The paper identifies manual annotation as difficult and time-consuming for realistic images.Source: §I · p. 2

II. RELATED WORK

Related work covers FCN-based object segmentation architectures and earlier smoke-segmentation methods, then positions this paper’s two-path network as a new combination for smoke segmentation.

A. Fully convolutional network for segmentation

  • FCNs use skip structures to recover spatial detail; SegNet instead uses unpooling indices, while U-Net combines skip connections with a symmetric decoder.The cited FCN work also initializes from pretrained VGG16 parameters, whereas SegNet uses stagewise training.Source: A. Fully convolutional network for… · p. 2
  • Other FCN variants combine multi-stage feature fusion, two-stream prediction, or graphical models such as Deeplab’s CRF and ASPP.The paper also describes ResNet-based boundary-refinement and motion-and-appearance approaches.Source: A. Fully convolutional network for… · pp. 2–3
  • The paper proposes combining two paths, encoder-decoder structures, and skip layers in one end-to-end FCN for smoke segmentation.The authors describe this as the first such combination for smoke segmentation.Source: A. Fully convolutional network for… · p. 3

B. Smoke segmentation

  • Earlier smoke-segmentation work mainly used color, rough sets, Gaussian mixtures, motion, or sparse-representation and matting methods; the paper reports no prior deep-learning smoke segmentation literature.The proposed approach combines two-path encoder-decoder FCNs, skip structures, and decoding structures.Source: B. Smoke segmentation · p. 3

III. DATASET GENERATION FOR SMOKE BINARY SEGMENTATION

The paper addresses the shortage of annotated smoke-segmentation data by generating synthetic smoke images and masks. It blends computer-generated smoke with backgrounds and varies smoke concentration to diversify samples.

  • Manually segmenting fuzzy, translucent smoke is difficult and time-consuming, motivating synthetic training data.Source: §III · p. 3
  • The authors generated 8,162 computer-graphics pure smoke images and used linear color composition to create training and three test datasets, each with 1,000 images.Source: §III · p. 3
  • Each pure smoke image has RGB color and alpha channels; blending it with a background RGB image produces a synthesized smoke image.Source: §III · p. 3
  • Training samples pair randomly selected Places365-Standard backgrounds with pure smoke images, while scaling alpha values controls smoke concentration.Source: §III · p. 3
  • Thresholding a pure smoke image’s alpha channel supplies the binary smoke mask used as the synthesized image’s ground truth.Source: §III · p. 4

IV. TWO-PATH FULLY CONVOLUTIONAL NETWORKS

The network combines a deep path for global smoke context with a shallow refinement path for local detail. Its first path uses a VGG16-based encoder-decoder FCN with skip connections to produce a coarse smoke map.

IV. TWO-PATH FULLY CONVOLUTIONAL NETWORKS

  • The two-path FCN uses a deep path for global segmentation prediction and a refinement path that produces a more detailed prediction.Source: §IV · p. 5

A. A deep network for global context information

B. A shallow network for local fine information

The second path restores smoke details lost during coarse global prediction. It uses a shallower VGG16-based encoder-decoder with skip connections to produce a detailed segmentation map.

C. A fusion network of two paths

The fusion network combines the coarse prediction and detailed spatial information to produce the final smoke map. The evaluation uses three synthetic test datasets and a real-image dataset, with real images assessed qualitatively because ground truths are unavailable.

C. A fusion network of two paths

  • The fusion network adds the two paths’ outputs and applies a 1×1 convolution with sigmoid activation to produce the final prediction map.Source: C. A fusion network of two paths · p. 6
  • Unlike the cited VGG16 design, this method uses the last three blocks, replaces deconvolution and summation with upsampling and concatenation, and adds a shallow skip-connected encoder-decoder for finer segmentation.Source: C. A fusion network of two paths · p. 6

A. Image datasets

  • Test images were held out from training and use backgrounds different from the training samples; real smoke images were evaluated only qualitatively because ground truths were unavailable.Source: A. Image datasets · p. 6
  • The three synthetic test datasets each contain 1,000 composited 256×256 images, with backgrounds drawn respectively from StreetScenes, Pascal VOC, and a Baidu people-segmentation dataset.Source: A. Image datasets · p. 6

B. Implementation details

The network is trained end to end using VGG16-pretrained encoder weights, and evaluation compares it with deep-learning segmentation baselines. Qualitative results report sharper smoke edges and more accurate locations for the proposed method.

B. Implementation details

  • Encoder convolutional layers use ImageNet-pretrained VGG16 weights, while decoder parameters are randomly initialized and training uses SGD.Training uses a fixed learning rate of 0.001, momentum of 0.9, and weight decay of 1e-5.Source: B. Implementation details · p. 6
  • The fusion network and skip structures allow simultaneous end-to-end optimization, unlike the staged training strategy described for.The comparison method trains each path separately before fine-tuning the whole network.Source: B. Implementation details · p. 7
  • Synthetic predictions are binarized at 0.5 for comparison, while real-image predictions are shown without post-processing because ground truths are unavailable..Source: B. Implementation details · p. 7

C. Performance comparisons

  • The comparison uses FCN, SegNet, TBFCN, and static-map detection, trained on the same smoke training data.Only TBFCN's first-path results and the static segmentation results of the video method are compared.Source: C. Performance comparisons · p. 7
  • On synthetic and real smoke images, the proposed method produces sharper edges and more accurate smoke locations than the compared methods.Real images have no corresponding ground truths.Source: C. Performance comparisons · p. 7
  • The quantitative evaluation covers three synthetic test datasets and uses mIoU and Mse to assess segmentation.The passages describe these as widely used measures.Source: C. Performance comparisons · p. 7

PR GT n PR GT

Across three synthetic test datasets, the proposed method achieves the highest mIoU and lowest mMse among the compared methods. The section also attributes performance differences to skip structures and pretrained weights.

Metric definition

  • Mse is defined as the average per-pixel squared difference between a test image's prediction and ground truth.The image height and width and pixel coordinates are used in the definition.Source: PR GT n PR GT · p. 7

Quantitative results

Performance interpretation

  • The authors identify skip architectures and VGG16-pretrained weights as factors that can greatly improve prediction performance.They report SegNet as the lowest-performing method and cite its lack of the same skip layers and pretrained initialization as reasons.Source: PR GT n PR GT · p. 8

D. Ablation analysis

Ablations find that both the second refinement path and skip structures contribute to segmentation performance. The proposed fusion operations also outperform a deconvolution-and-add variant on all three synthetic datasets.

Ablation design

  • The ablations remove path 2, path 2's skip structures, or path 2 together with path 1's skip structures.These variants are used to assess the network components' importance.Source: D. Ablation analysis · p. 8

Ablation results

  • Variants with skip layers in both paths perform better than variants without skip layers.The passage identifies Variant 2 (-R) and the complete method as the skip-layer cases, and Variants 1 (-R-Cs) and 3 (-Rs) as cases without skip layers.Source: D. Ablation analysis · p. 9
  • On DS01, adding path 2 raises mIoU from 64.64% to 71.04%, while using skip architectures in both paths yields about 4 to 5 percentage higher mIoU than omitting skip layers.The reported skip-layer pattern also occurs on DS02 and DS03; Tables 7–9 list the ablation results.Source: Table 7 · p. 9, Table 8 · p. 9, Table 9 · p. 9

Qualitative ablations

  • The complete method's real-image results are reported as significantly better than the coarse results of Variant 2 (-R).The qualitative comparison is shown in Fig. 7.Source: Figure 7 · p. 9, D. Ablation analysis · p. 9

Fusion comparison

E. Test on videos

Video tests report accurate frame-level segmentation for black smoke but inaccurate results on a poor-quality white-smoke video. The segmentation-based detector identifies smoke at the first frame of both smoke videos and reports no false alarms on either non-smoke video.

Smoke-video segmentation

Video detection

  • For whole-image recognition, the method counts pixels classified as smoke and labels an image as smoke when that count exceeds a threshold..Source: E. Test on videos · pp. 10–11
  • The method detects smoke at the first frame of both smoke videos and raises no false alarms on the two non-smoke videos.No post-processing techniques are used to further reduce false alarms; Table 11 lists results against LBP_LBPV and Toreyin's method.Source: Table 11 · p. 11, E. Test on videos · p. 11

F. Limitations of our method

The method outperforms FCN-based segmentation algorithms across three synthetic datasets and one realistic dataset, and also outperforms existing methods on videos. The authors identify feature variation, blurry edges, smoke-like objects, and test speed as remaining challenges.

F. Limitations of our method

  • Smoke feature variation, blurry edges, and visual resemblance to fog and clouds remain challenges, while test speed needs further improvement.The authors identify real-time performance as an important application requirement.Source: F. Limitations of our method · p. 11

VI. CONCLUSIONS

  • The method performs much better than state-of-the-art FCN-based segmentation algorithms on three synthetic and one realistic dataset, and outperforms existing methods on videos.Ablations also find the second refinement network and skip structures important for segmentation accuracy.Source: §VI · p. 11
  • The coarse and fine encoder-decoder paths combine global context and localization features, with the second path shallower, smaller, and simpler than the first.The method also uses image composition to generate synthetic smoke images because fuzzy smoke is difficult to annotate manually.Source: §VI · p. 11
Loading 1809.00774v1…