PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Transformer-Based Multi-Level Feature Fusion for Multi-Label Sewer Defect Classification

Xu Fang, Zhuoran Wang, Qing Li, Shengyu Zhang, Guanzhi Deng, Jianbiao He, Qingquan Li

arXiv:2609.11375v1cs.CV

TL;DR

Automated sewer defect classification must handle multiple co-occurring defects while balancing accuracy and computational cost. The paper develops a hierarchical multi-level-fusion Transformer and two lightweight variants, reporting leading Sewer-ML performance, strong Sewer-Capsule accuracy, and improved reduced-data transfer with pretraining. Operational generalization and hardware efficiency remain to be validated.

  • Problem

    Automated sewer defect classification must balance accuracy and computational complexity in large-scale multi-label inspection scenarios.

  • Method

    The paper develops Sewer-Transformer-ML, a hierarchical vision Transformer with multi-level feature fusion, plus MobileNetV3-based pure-CNN and CNN–Transformer lightweight variants.

  • Results

    Across Sewer-ML and Sewer-Capsule evaluations, the models achieved leading benchmark performance, 96.43% capsule accuracy, and consistent pretraining gains with reduced training data.

  • Takeaways & Limitations

    The framework supports automated large-scale CCTV analysis, lightweight model design, and adaptation to emerging robotic inspection platforms.

  • Takeaways & Limitations

    Results may not generalize across cities, pipe materials, defect standards, imaging conditions, and robotic systems, and operational hardware efficiency remains insufficiently validated.

Abstract

from arXiv · show

Automated classification of sewer defects is essential for infrastructure condition assessment and maintenance decision-making, but existing deep learning methods struggle to balance classification accuracy and computational complexity in large-scale multi-label scenarios. This study develops Sewer-Transformer-ML, a hierarchical vision Transformer with multi-level feature fusion, together with two lightweight architectures, Sewer-MobileNet-ML and Sewer-Mobile-TransNet, for resource-constrained inspection scenarios. On the Sewer-ML test set, Sewer-Transformer-ML-Base achieved an $F2_{\text{CIW}}$ of 65.68% and an $F1_{\text{Normal}}$ of 92.68%, ranking first on the public leaderboard and exceeding the second-ranked method by 7.6 percentage points in $F2_{\text{CIW}}$. Sewer-MobileNet-ML achieved an $F2_{\text{CIW}}$ of 65.73% with only 17 M parameters, representing an approximately 95% parameter reduction relative to the base model. Under the standard Sewer-Capsule data split, Sewer-Mobile-TransNet achieved 96.43% classification accuracy. When the training set was reduced to 1,177 images, pretraining on Sewer-ML consistently improved model performance. Ablation experiments further showed that direct concatenation was more effective for Transformer features, whereas attention-based fusion better supported multiscale CNN features. These findings provide a computational basis for automated sewer inspection, lightweight model design, and adaptation across civil infrastructure inspection platforms.

PRACTICAL APPLICATIONS

The proposed models target labor-intensive sewer inspection by automatically classifying normal conditions and multiple defect types. Their reported results support lightweight deployment and transfer to emerging inspection platforms, although operational hardware validation remains necessary.

  • PRACTICAL APPLICATIONS: The models can automatically screen sewer inspection images, distinguish normal conditions from defect types, and assign multiple labels for co-occurring defects.These classifications can help prioritize images or pipe segments for further examination.
  • PRACTICAL APPLICATIONS: Knowledge learned from large CCTV datasets improved adaptation to sewer-capsule images, reducing labeled-data requirements for a new inspection device.The paper presents this as evidence of cross-domain transferability.
  • PRACTICAL APPLICATIONS: Inference speed, energy consumption, and long-term reliability on operational field hardware remain unvalidated.Further cross-region and hardware testing is needed to establish deployment reliability.
  • PRACTICAL APPLICATIONS: 65.73% F2_CIW was achieved by Sewer-MobileNet-ML with approximately 95% parameter reduction, supporting a performance–complexity trade-off for field inspection.The model contains approximately 17M parameters and is described as maintaining state-of-the-art accuracy.
  • PRACTICAL APPLICATIONS: 96.43% classification accuracy was achieved by Sewer-Mobile-TransNet under the standard Sewer-Capsule split, while pretraining consistently improved performance with 1,177 training images.The experiments examine transfer from large CCTV datasets to smaller capsule data.

Deep Learning Methods for Pipeline Defect Detection

Deep learning methods have progressed from CNN-based sewer defect classification toward Transformer and hybrid architectures, while multi-label formulation treats each defect class independently. The proposed framework combines hierarchical Transformer or lightweight CNN–Transformer designs with category-weighted loss for imbalanced defects.

  • Deep Learning Methods for Pipeline Defect Detection: Sewer-ML established a large-scale multi-label benchmark with class-importance-weighted evaluation for comparing sewer defect classifiers.Earlier systems included binary subnetworks, shallow CNNs, and hierarchical screening–prediction pipelines.
  • Deep Learning Methods for Pipeline Defect Detection: Existing studies provide limited systematic evidence on fusing multi-level Transformer features and multiscale CNN features for large-scale multi-label sewer classification.Recent work includes stronger CNNs, vision Transformers, and hybrid CNN–Transformer architectures.
  • Deep Learning Methods for Pipeline Defect Detection: The proposed framework uses a Swin Transformer backbone, multi-level feature fusion, lightweight CNN and hybrid variants, and Category Importance Weighting loss to address class imbalance.The variants are Sewer-MobileNet-ML and Sewer-Mobile-TransNet.
  • Multi-Label Classification Formulation: Multi-label classification treats each defect class as an independent binary task, using element-wise Sigmoid outputs and Binary Cross-Entropy loss.The output vector has one component per defect class.
  • Multi-Label Classification Formulation: The loss uses class-specific weighting factors to address dataset imbalance, with C denoting the number of defect classes and y the binary ground-truth vector.The formulation applies Sigmoid activation before Binary Cross-Entropy evaluation.

Sewer-Transformer-ML: Multi-level Vision Transformer

Sewer-Transformer-ML is a hierarchical Swin Transformer that extracts and fuses multi-level representations for complex sewer defects. Its architecture uses patch-based stages, resolution changes, local and shifted-window attention, and feature-fusion units before classification.

  • Sewer-Transformer-ML: Multi-level Vision Transformer: Sewer-Transformer-ML uses a Swin Transformer backbone without convolutional backbone operations, extracting defect features through self-attention and fusing multi-level semantic information.The model is available in Tiny, Small, Base, and Large versions.
  • Sewer-Transformer-ML: Multi-level Vision Transformer: The hierarchical network partitions images into 4 × 4 patches, linearly embeds them, and processes them through successive feature-extraction stages.The four versions use embedding dimensions of 96, 96, 128, and 192.
  • Sewer-Transformer-ML: Multi-level Vision Transformer: Patch Merge progressively reduces feature-map resolution as depth increases, while Swin Transformer Blocks model patterns within local windows.The resulting representations have multiple spatial scales similar to CNN hierarchies.
  • Sewer-Transformer-ML: Multi-level Vision Transformer: Alternating window and shifted-window self-attention limits computation locally while enabling information exchange across adjacent windows.This design supports modeling defect patterns at different spatial scales.
  • Multi-level Vision Transformer Feature Fusion: After multi-level features are resized to a common scale and concatenated, fusion units recalibrate important features before fully connected classification.The study compares separated attention units and channel-spatial attention for feature fusion.

Lightweight Variants for Edge Deployment

The lightweight variants use MobileNetV3 components to reduce deployment cost, with Sewer-MobileNet-ML providing a pure-CNN baseline and Sewer-Mobile-TransNet adding shifted-window self-attention for multiscale fusion.

  • Lightweight Variants for Edge Deployment: MobileNetV3-based variants use inverted residual bottlenecks with depthwise separable convolution, channel expansion and projection, and squeeze-and-excitation.These components are adopted without structural modification.
  • Lightweight Variants for Edge Deployment: Sewer-MobileNet-ML replaces the MobileNetV3-Large classifier with a sigmoid multi-label head and contains approximately 17M parameters.It serves as the pure-CNN lightweight baseline for evaluating accuracy–complexity trade-offs.
  • Lightweight Variants for Edge Deployment: Sewer-Mobile-TransNet combines a MobileNetV3 backbone with Swin Transformer Blocks based on shifted-window multi-head self-attention.Cascaded bottlenecks extract multiscale CNN features before attention-based fusion.
  • Lightweight Variants for Edge Deployment: Sewer-Mobile-TransNet aligns multiscale feature maps, concatenates them, and uses shifted-window attention to screen and semantically correlate their spatial relationships.This fusion module is intended to enhance feature representation for pipeline defects.

Dataset Introduction

The study uses the large multi-label Sewer-ML CCTV dataset to evaluate defect classification and pretrain models, then tests transfer to the single-label Sewer-Capsule dataset. Performance is reported with standard metrics and the class-importance-weighted F2CIW metric.

  • Sewer-ML Dataset: Sewer-ML contains over 1.3 million CCTV images covering 17 defect types and normal images, with CIW factors reflecting operational consequences.The images were collected from 75,618 inspection videos and annotated by inspection professionals.
  • Sewer-ML Dataset: Sewer-ML is split by independent video segments into training, validation, and test sets, preventing duplicate images across subsets.The test labels are withheld and evaluation requires submission through the official website.
  • Sewer-Capsule Dataset: The Sewer-Capsule dataset is a single-label benchmark collected with capsule inspection equipment, containing five defect categories and normal images.Its training and validation sets contain 2,353 and 1,177 images, respectively.
  • Evaluation Metrics: The study evaluates accuracy, precision, recall, F1_Normal, mAP, and CIW-weighted F2CIW, which combines classwise F2 scores using class-importance weights.F1_Normal measures recognition of normal pipe images, while mAP summarizes performance across defect classes.

COMPREHENSIVE EXPERIMENTS ON SEWER-ML DATASET

Comprehensive Sewer-ML experiments compare hierarchical Transformer variants and a lightweight MobileNet model against existing methods. The base Transformer leads the benchmark, while the lightweight model achieves comparable performance with far fewer parameters and shorter training requirements.

  • Benchmark Comparison: 65.68% F2CIW and 92.68% F1_Normal placed Sewer-Transformer-ML-Base first on the public Sewer-ML leaderboard.Its F2CIW exceeded the second-best method by 7.6 percentage points and the previous best published result by more than 11 percentage points.
  • Training Behavior: Sewer-MobileNet-ML stabilized after approximately 50 epochs, whereas Transformer variants stabilized after about 150 epochs.The lightweight model was trained for 200 epochs and the Transformer variants for 300 epochs.
  • Training Cost: The lightweight model required substantially less training time than the Transformer variants for 200-epoch training runs.Training included validation every two epochs and used four A100 GPUs.

Multi-level Vision Transformer Feature Fusion Experiment

Feature-fusion experiments compare direct stacking and attention-based strategies for Transformer and CNN representations. Direct concatenation benefits multi-level Transformer features, while the lightweight CNN-based design combines multiscale features with sliding-window multi-head self-attention.

  • Multi-level Vision Transformer Feature Fusion Experiment: The Transformer fusion ablation compared a no-fusion Tiny baseline, direct stacking, separated attention, and CBAM strategies using loss, F1_Normal, F2CIW, and mAP curves.Validation metrics were evaluated every two epochs during training.
  • Multi-level Vision Transformer Feature Fusion Experiment: Directly stacking multi-level Transformer features improved performance, whereas additional attention fusion strategies did not further enhance the Transformer model.The analysis attributes this pattern to self-attention already screening and fusing features at each Transformer stage, making extra attention potentially redundant.
  • Sewer-Mobile-TransNet Experiment: The lightweight experiment evaluated CAT and CBAM multiscale fusion strategies alongside loss, F1_Normal, F2CIW, and mAP curves.Sewer-MobileNet-ML was trained for 105 epochs, while Sewer-Mobile-TransNet was trained for 100 epochs with a partially fixed pretrained backbone.
  • Sewer-Mobile-TransNet Experiment: Sewer-Mobile-TransNet achieved the best performance across all evaluation metrics among the lightweight architectures.The model used a multiscale fusion strategy based on sliding-window multi-head self-attention and remained competitive with full-scale architectures after approximately 95% parameter reduction.

SEWER-CAPSULE DATASET EXPERIMENTS

Transfer-learning experiments assess whether models pretrained on Sewer-ML generalize to the smaller single-label Sewer-Capsule dataset. Sewer-Mobile-TransNet achieves the highest reported validation accuracy among the compared architectures.

  • Transfer-Learning Setup: The transfer-learning setup uses Sewer-ML pretraining to improve robustness, adaptability, and performance when labeled Sewer-Capsule data are limited.Sewer-Capsule is a single-label dataset collected with pipeline capsule equipment.
  • Transfer-Learning Setup: The Sewer-Capsule training and validation sets contain 2,353 and 1,177 images, respectively, with evaluation performed over 40 training epochs.Validation accuracy was evaluated every two epochs.
  • Transfer-Learning Results: 96.43% accuracy made Sewer-Mobile-TransNet the strongest model on Sewer-Capsule, exceeding Sewer-Transformer-ML-Tiny at 87.76% and ResNet-50 at 87.51%.The result was obtained on the validation set containing 1,177 images.

Transfer Learning Experiment

Transfer experiments evaluated lightweight and feature-fusion designs across sewer inspection datasets, showing strong benchmark performance and cross-platform transfer under reduced labeled-data conditions.

  • Benchmark performance: 65.68% F2_CIW ranked Sewer-Transformer-ML-Base first, exceeding the second-ranked method by 7.6 percentage points on Sewer-ML.The comparison was reported for the public Sewer Defect Classification Challenge leaderboard.
  • Lightweight architectures: 65.73% F2_CIW was achieved by Sewer-MobileNet-ML with approximately 95% fewer parameters than Sewer-Transformer-ML-Base.The lightweight model used about one twentieth of the base model's parameters while remaining close in core accuracy metrics.
  • Feature fusion: Direct concatenation improved Transformer multi-level feature fusion, whereas attention-based fusion improved CNN multi-scale feature fusion.Transformer features had already undergone self-attention-based information interaction, while CNN features benefited from multi-head attention.
  • Cross-platform evaluation: 96.43% classification accuracy was achieved by Sewer-Mobile-TransNet under the standard Sewer-Capsule split.The hybrid architecture performed best across the reported lightweight evaluation metrics.
  • Transfer learning: Pretraining on Sewer-ML consistently improved performance when the Sewer-Capsule training set was reduced to 1,177 images.The reduced-data experiment used 1,177 training images and 2,353 validation images.

Limitations and Future Work

The study reports strong benchmark and transfer results, but its evidence remains bounded by dataset coverage, unavailable test labels, and incomplete hardware-level deployment evaluation.

  • Limitations: Generalization remains uncertain across cities, pipe materials, defect standards, imaging conditions, and robotic systems because evaluation focused mainly on Sewer-ML and the small Sewer-Capsule dataset.The authors identify cross-region and cross-device evaluation as future work.
  • Limitations: Inference latency, memory consumption, energy use, and long-term reliability were not systematically evaluated on operational hardware.The reported results therefore demonstrate edge-deployment potential rather than completed field deployment.
  • Main results: 65.68% F2_CIW and 92.68% F1_Normal placed Sewer-Transformer-ML-Base first on Sewer-ML, while Sewer-MobileNet-ML reached 65.73% F2_CIW with 17 M parameters.The lightweight result used approximately 95% fewer parameters than the 333 M-parameter base model.
  • Feature-fusion insight: Direct concatenation favored Transformer features, whereas attention-based fusion provided greater benefits for multiscale CNN features.The distinction is presented as a transferable design insight for civil infrastructure image analysis.
  • Transferability: 96.43% accuracy was achieved on Sewer-Capsule, and Sewer-ML pretraining improved performance with only 1,177 training images.The results support adaptation across inspection platforms but do not establish field deployment reliability.

APPENDIX I. EXPERIMENTAL TRAINING SETTINGS SUMMARY

The appendix summarizes training configurations for the Transformer, MobileNet, Mobile-TransNet, and baseline experiments, including optimizers, hardware, batch sizes, and learning-rate scaling.

  • Transformer models: Sewer-Transformer-ML models used AdamW on four Nvidia A100 GPUs with weight decay 0.05 and batch-size-scaled learning rates.Per-GPU batch sizes were 320 for Tiny, 196 for Small, and 144 for Base.
  • MobileNet model: Sewer-MobileNet-ML used RMSprop with momentum 0.9, weight decay 1 × 10^-5, four Nvidia A100 GPUs, and per-GPU batch size 512.Its learning rate followed the same scaling rule as the Transformer models.
  • Experiment correspondence: The main-text feature-fusion and Sewer-Mobile-TransNet experiments reused the corresponding Transformer and MobileNet training configurations.This appendix states that the feature-fusion experiments followed the model settings described here.
  • Single-GPU settings: Sewer-Mobile-TransNet used RMSprop-based MobileNet settings with batch size 64 on one GPU, while Transformer-ML Tiny used AdamW-based settings with batch size 64 on one GPU.The baseline ResNet-50 used a separate SGD with Nesterov momentum configuration.
Loading 2609.11375v1…