Source-linked AI summary
Learning Statistical Texture for Semantic Segmentation
Lanyun Zhu, Deyi Ji, Shiping Zhu, Weihao Gan, Wei Wu, Junjie Yan
TL;DR
Semantic segmentation lacks an explicit mechanism for exploiting global statistical texture beyond local structural and high-level contextual features. STLNet introduces QCO with TEM and PTFEM to learn statistical texture, and reports state-of-the-art performance across Cityscapes, PASCAL Context, and ADE20K.
Problem
Existing semantic segmentation methods emphasize high-level context and do not explicitly extract and utilize the global statistical texture information in low-level features.
Method
STLNet uses QCO to describe texture statistically, then TEM enhances texture details and PTFEM extracts statistical texture features across multiple scales.
Results
STLNet achieves state-of-the-art results on the Cityscapes, PASCAL Context, and ADE20K semantic segmentation benchmarks.
Takeaways & Limitations
The method provides a practical, plug-and-play way to incorporate statistical texture into semantic segmentation.
Abstract
from arXiv · showhide
Existing semantic segmentation works mainly focus on learning the contextual information in high-level semantic features with CNNs. In order to maintain a precise boundary, low-level texture features are directly skip-connected into the deeper layers. Nevertheless, texture features are not only about local structure, but also include global statistical knowledge of the input image. In this paper, we fully take advantages of the low-level texture features and propose a novel Statistical Texture Learning Network (STLNet) for semantic segmentation. For the first time, STLNet analyzes the distribution of low level information and efficiently utilizes them for the task. Specifically, a novel Quantization and Counting Operator (QCO) is designed to describe the texture information in a statistical manner. Based on QCO, two modules are introduced: (1) Texture Enhance Module (TEM), to capture texture-related information and enhance the texture details; (2) Pyramid Texture Feature Extraction Module (PTFEM), to effectively extract the statistical texture features from multiple scales. Through extensive experiments, we show that the proposed STLNet achieves state-of-the-art performance on three semantic segmentation benchmarks: Cityscapes, PASCAL Context and ADE20K.
1. Introduction
Semantic segmentation models capture high-level context and often skip-connect low-level features for boundaries, but they lack an explicit mechanism for global statistical texture. STLNet addresses this gap with QCO, TEM, and PTFEM to learn and use statistical texture information.
- Existing segmentation approaches emphasize high-level contextual features and use low-level skip connections, but do not explicitly extract and use statistical texture.
- STLNet introduces statistical texture learning for semantic segmentation, jointly learning low-level and high-level features end to end.
- QCO quantizes continuous texture information into multiple intensity levels for statistical description in deep neural networks.
- TEM enhances statistical texture details, while PTFEM extracts texture-related features from multiple scales.
- STLNet is presented as a plug-and-play method achieving state-of-the-art results on Cityscapes, PASCAL Context, and ADE20K.
2. Related Works
Related work positions STLNet within semantic segmentation and feature encoding research. Its QCO is described as a self-adaptive feature encoding method, while the network combines a ResNet101- and ASPP-based architecture with texture modules.
- Modern semantic segmentation methods largely build on fully convolutional networks and extend them with mechanisms for multi-scale or contextual feature extraction.
- The STLNet overview uses ResNet101 and ASPP for high-level context, then processes low-level features with TEM and PTFEM before prediction.
- QCO can be viewed as a feature encoding method that differs from prior approaches through self-adaptive quantization.
3. Method
STLNet combines a semantic base network with a texture branch that statistically analyzes low-level features. Its QCO, TEM, and PTFEM components quantize and count texture information, enhance texture details, and extract multi-scale texture features.
- Overall Structure: STLNet uses a dilated ResNet101 with ASPP as its base network and a separate branch for texture extraction.The texture branch processes features from backbone layers 1 and 2, downsamples them, concatenates them, and applies TEM followed by PTFEM.
- Quantization and Counting Operator: QCO statistically describes texture by quantizing feature responses into N levels, counting their distribution, and encoding average features.The operator uses global average features and spatial responses, with smoother quantization encoding than argmax or one-hot binarization to avoid gradient vanishing during backpropagation.
- Quantization and Counting Operator: The 1-d QCO outputs a quantization encoding map and statistical feature that represent feature distributions across spatial positions.The encoding map records pixel-level quantization assignments, while the statistical feature combines normalized counts with the global average feature.
- Quantization and Counting Operator: The 2-d QCO captures adjacent-pixel texture by representing the co-occurrence of every pair of quantization levels.It multiplies encodings for neighboring pixels, producing nonzero entries only for their corresponding quantization-level pair, then counts these co-occurrences and encodes average features.
- Texture Enhancement Module: TEM enhances low-quality low-level texture features by using 1-d QCO statistics to reconstruct quantization levels through learned information propagation.The module treats each quantization level's statistical feature as a graph node, learns an adjacency matrix, and assigns reconstructed levels back to pixels using the encoding map.
- Pyramid Texture Feature Extraction Module: PTFEM extracts texture-related features from multiple scales using 2-d QCO co-occurrence statistics, inspired by gray level co-occurrence matrices.The module uses feature maps with rich texture details and represents spatial relationships through co-occurrence statistics over quantization levels.
4. Experiments
Experiments evaluate STLNet through component ablations, quantization-level studies, efficiency analysis, visualizations, and comparisons across three semantic-segmentation benchmarks.
- 80.9% mIoU is obtained with SLF, TEM, and PTFEM, improving the 76.4% baseline by 4.5%.
- 81.5% mIoU is achieved with ResNet101 + ASPP + SLF + TEM + PTFEM, showing the modules improve a stronger segmentation network.
- 128 quantization levels produce 81.5% mIoU, while too few levels perform poorly and denser quantization slightly reduces performance.
- TEM visibly clarifies texture details in building and bicycle feature maps, while the proposed modules add little computational cost.
- STLNet achieves state-of-the-art performance on Cityscapes, PASCAL Context, and ADE20K.
5. Conclusion
STLNet learns statistical texture features for semantic segmentation through QCO, TEM, and PTFEM, achieving strong performance across multiple datasets.
- STLNet combines QCO, TEM, and PTFEM to learn, enhance, and extract statistical texture features for semantic segmentation.QCO describes statistical texture representations; TEM enhances details, while PTFEM exploits texture-related low-level information.