MSCFNet: A Lightweight Network With Multi-Scale Context Fusion for Real-Time Semantic Segmentation

• Real-time segmentation faces high calculation and storage demands, while prior lightweight work often reduces inference time at the expense of accuracy.Source: §I · p. 1
• The EAR core uses dilated factorized depth-wise separable convolution to extract feature information over large receptive fields efficiently.Source: §I · pp. 1–2
• Short- and long-range spatial and channel attention connections support local-context interaction, while adaptive fusion combines attentive features across scales.Source: §I · p. 2
• With 1.15M parameters, MSCFNet achieves 71.9% mIoU on Cityscapes and 69.3% on CamVid without other data-augmentation skills.Source: §I · p. 2

A. Factorization convolution

• Factorization replaces a two-dimensional convolution with paired one-dimensional convolutions; depth-wise separable convolution operates on input-channel and filter groups individually.These decompositions can reduce computational burden.Source: A. Factorization convolution · p. 2

B. Dilation convolution

• Dilation enlarges receptive fields without adding parameters; DeepLab,, [25] uses dilation rates in spatial pyramids, while EDANet combines dilation with dense connections.Source: B. Dilation convolution · p. 2

C. Lightweight segmentation networks

• ENet [16], ESPNet, ICNet, BiseNet, and LEDNet use distinct efficiency strategies; Fig. 2 compares Bottleneck [16], Non-bt-1D [18], SS-nbt, and EAR residual layouts.Source: C. Lightweight segmentation networks · p. 2, Figure 2 · p. 3

D. Attention mechanism

• Some pixel-level attention methods use matrix multiplication unsuitable for lightweight applications; MSCFNet applies spatial and channel attention according to network stage.The paper describes SENet’s channel correlation and CBAM’s separate spatial and channel attention maps as related mechanisms.Source: D. Attention mechanism · p. 3

III. METHODOLOGY

• The architecture comprises an initial block, three input injection modules, two downsampling blocks, two EAR blocks, and two context branches.The methodology introduces the EAR, efficient attention, and context fusion modules before describing the whole network.Source: §III · p. 3

A. EAR Module

• EAR halves input channels at a 3 × 3 bottleneck, then pairs factorized depth-wise local features with dilated long-range features using different rates to avoid gridding artifacts.The factorized branch splits a 3 × 3 depth-wise convolution into 3 × 1 and 1 × 3 convolutions with fewer parameters.Source: A. EAR Module · pp. 3–4
• EAR branches interact between their factorized convolutions, receive channel attention, and are fused before point-wise convolution and channel shuffling with the input.The paper uses channel shuffling to facilitate channel information exchange and sharing after depth-wise convolution.Source: A. EAR Module · p. 4

B. Efficient Attention

• Attention helps lightweight networks integrate information and emphasize object features when their few layers limit deep feature extraction.The paper includes both spatial and channel attention modules.Source: B. Efficient Attention · p. 4
• Spatial attention forms weight maps from average- and max-pooled features, then multiplies the maps by the input features.The spatial attention modules are placed between the three injection modules and the main branch.Source: B. Efficient Attention · p. 4
• Channel attention is derived from ECANet and is placed after feature combinations and within the context branches.The network also places channel-attention modules in the encoder and decoder processes.Source: B. Efficient Attention · p. 5
• Across network stages, spatial attention indicates where to highlight and channel attention indicates what features to emphasize, facilitating local and contextual interaction.The paper reports that these attention mechanisms were validated as effective in its experiments.Source: B. Efficient Attention · p. 5

C. Context Fusion

• Stage-by-stage finer-scale predictions may leave contextual cues unable to complement one another.The authors contrast this with the benefits of fusing hierarchical information.Source: C. Context Fusion · p. 6
• MSCFNet integrates multi-scale contextual information through context fusion before final classification.The fused features are denoted Fcf.Source: C. Context Fusion · p. 6
• The context fusion module combines attentive contextual information from different network stages and can alleviate limitations caused by loss of pixel spatial statistics.Source: C. Context Fusion · p. 6

D. Network Architecture

• The encoder uses three progressive downsampling steps to reach 1/8 resolution, balancing deeper context collection with retention of image details.The initial feature extraction uses three 3 × 3 convolutions, with stride 2 on the first; downsampling blocks are followed by EAR blocks.Source: D. Network Architecture · p. 6
• The first and second EAR blocks contain 5 and 10 modules, respectively, and inter-block concatenation combines high- and low-level features.The dilation rates are {1, 1, 2, 2, 5} in the first block and {1, 2, 5, 7, 9, 2, 5, 7, 9, 17} in the second, chosen to mitigate gridding artifacts and enlarge the receptive field.Source: D. Network Architecture · pp. 6–7
• Spatial attention in shortcut connections and channel attention after feature concatenations support feature reuse and contextual semantic integration.Long-range shortcuts also address the mismatch between shallow features' limited semantics and deep features' limited boundary and detail information.Source: D. Network Architecture · p. 7
• The asymmetric decoder uses ×4 upsampling followed by ×2 deconvolution to restore the input size and produce segmentation predictions.The authors describe this two-step structure as combining calculation simplicity with recovery of decoded information.Source: D. Network Architecture · p. 7

Datasets

• Cityscapes has 2,975 training, 500 validation, and 1,525 testing samples; CamVid has 367, 101, and 233, respectively.The datasets contain 19 and 11 semantic categories, respectively.Source: A. Implementation Protocol · p. 7
• Inputs are resized to 1024 × 512 for Cityscapes and 480 × 360 for CamVid training.Source: A. Implementation Protocol · p. 7

Parameter Settings

• Cityscapes training uses SGD with batch size 4, momentum 0.9, weight decay 1×10−4, and an initial learning rate of 4.5 × 10−2.The learning rate follows a poly policy with power 0.9.Source: A. Implementation Protocol · p. 7
• CamVid training uses Adam with batch size 8, momentum 0.9, weight decay 2×10−4, and an initial learning rate of 1 × 10−3.It also uses the poly learning-rate policy.Source: A. Implementation Protocol · p. 7

Ablation Study

• The ablations use the CamVid training and validation sets and evaluate on its testing set.Source: B. Ablation Study · p. 8

Context Fusion

• Removing context fusion lowers segmentation results by more than 1%, and feature addition outperforms feature concatenation.The study also examines the effects of using different feature levels.Source: B. Ablation Study · p. 8

Attention Module

• Channel attention SE improves accuracy by 0.1%, while ECA performs better with fewer parameters.Source: B. Ablation Study · p. 8
• Adding spatial attention to injection branches raises segmentation accuracy from 69.16% to 69.30%.Using channel and spatial attention together yields better segmentation performance with a negligible parameter increase.Source: B. Ablation Study · p. 8

Dilation Rate

• Reducing EAR modules to 3 in the first block and 6 in the second produces accuracy 1.03% below the final version; the tested power-of-two dilation rates also underperform its settings.Source: B. Ablation Study · p. 8

Evaluation Results

• The qualitative comparisons in Fig. 7 show MSCFNet alongside DABNet, CGNet, SegNet, and ENet on CamVid testing images.The paper states that its qualitative comparisons verify MSCFNet’s effectiveness.Source: Figure 7 · p. 9, C. Evaluation Results · p. 9
• With 1.15M model size, MSCFNet achieves 71.9% Cityscapes testing class mIoU, 88.4% category mIoU, and 69.3% CamVid testing class mIoU.Source: C. Evaluation Results · p. 9
• On Cityscapes, MSCFNet uses 15% of ICNet’s parameters yet runs faster and scores better; DABNet is 0.4M smaller but has 1.8% lower accuracy.Source: C. Evaluation Results · p. 9

V. CONCLUSIONS

• The EAR module uses factorization depthwise and dilation convolutions to capture object features with fewer parameters and computational budgets across receptive fields.Source: §V · p. 9
• Attention branches at different network stages extract multi-scale context, whose features are combined to enhance feature expression and local-context interaction.Source: §V · p. 9
• Experiments on CamVid and Cityscapes show a better efficiency-accuracy balance than several comparative approaches.Source: §V · p. 9