PaperlayerView exact revision on arXiv

Source-linked AI summary

HDVS: semi-supervised semantic segmentation via heterogeneous dual-branch voting supervision

Yongqi Shan, Yunzhi Zhuge, Huchuan Lu

arXiv:2610.37236v1cs.CV

TL;DR

Dense pixel-wise labels are costly, motivating semi-supervised semantic segmentation, while co-training can rely on pseudo-labels whose errors may accumulate. HDVS combines heterogeneous branches, voting, and feature communication; it reports 0.7∼2.9 mIoU-point improvements across PASCAL VOC 2012 and Cityscapes.

  • Problem

    Dense pixel-wise annotation is costly, and co-training pseudo-label errors may accumulate and yield suboptimal performance.

    Source: §1 · pp. 1–2

  • Method

    HDVS uses heterogeneous dual branches, confidence-aware voting to fuse their predictions into per-pixel pseudo-labels, and feature communication to preserve branch diversity.

    Source: §1 · p. 3

  • Results

    HDVS reports 0.7∼2.9 mean intersection over union (mIoU) point improvements across PASCAL VOC 2012 and Cityscapes.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that HDVS improves over state-of-the-art approaches on both PASCAL VOC 2012 and Cityscapes.

    Source: §5 · p. 13

    Under the 1/16 classical partition protocol on PASCAL VOC 2012, divergent branch predictions in challenging regions can yield less satisfactory segmentation.

    Source: §4.5 · p. 12

Abstract

from arXiv · show

Semi-supervised semantic segmentation, which aims to utilize large volumes of unlabeled images to achieve accurate segmentation results with fewer human annotations, has attracted increasing attention. Prior methods were primarily based on the assumption that the pseudo-labels generated by each branch were sufficiently reliable to supervise the remaining branches. However, there is a concern that the errors in pseudo-labels could accumulate during co-training, potentially resulting in suboptimal performance. To this end, we present a novel framework called heterogeneous dual-branch voting supervision (HDVS), which is designed to enhance the reliability of pseudo-labels and mitigate the issues arising from pseudo-labeling. Specifically, based on the cross-supervision framework, we introduce a voting mechanism that correlates pseudo-labels from heterogeneous branches to produce pseudo-labels with enhanced reliability. Concurrently, we employ a feature communication module to introduce perturbations at the feature level in each branch to maximize prediction diversity across the dual branches while maintaining convergence. Comprehensive evaluations of the proposed HDVS on two benchmark datasets (PASCAL VOC 2012 and Cityscapes) demonstrate its superiority to the state-of-the-art approaches.

1 Introduction

Dense pixel-wise annotation is costly, motivating semi-supervised segmentation with limited labels and many unlabeled images. HDVS combines heterogeneous branches, confidence-aware voting, and feature communication; experiments report gains across two benchmarks.

1 Introduction

  • The introduction frames costly dense pixel-wise annotation as motivation for using a few labeled and many unlabeled images in semantic segmentation.Source: §1 · p. 1
  • The paper attributes iterative improvement to diverse complementary branch predictions being fused into more reliable pseudo-labels that strengthen subsequent supervision.The authors describe this as a cycle of prediction refinement, pseudo-label enhancement, and model improvement.Source: §1 · p. 2
  • Prior approaches emphasize either reliability through a single teacher–student network or diversity through co-training, while swapping branch argmax labels without explicit confidence fusion risks confirmation bias.Source: §1 · pp. 2–3
  • HDVS combines heterogeneous dual branches with confidence-based voting and feature communication to produce more reliable pseudo-labels while preserving branch diversity.Voting fuses branch outputs per pixel; bidirectional feature communication is described as preserving global–local complementarity and inter-branch diversity.Source: §1 · p. 3
  • HDVS improves mean intersection over union (mIoU) by 0.7∼2.9 points across PASCAL VOC 2012 and Cityscapes.The paper describes these as performance improvements across the two public benchmarks.Source: §1 · p. 3

2 Related Work

Related work describes two broad SSL principles and their use in semi-supervised semantic segmentation: entropy minimization with pseudo-labeling, and consistency across perturbations or views.

2.1 Semi-Supervised Learning

  • SSL work uses entropy minimization to encourage confident predictions and pseudo-labeling, alongside consistency regularization across perturbed versions of an input.Source: §2.1 · p. 3

2.2 Semi-supervised Semantic Segmentation

  • Semantic-segmentation methods apply consistency through varied views and augmentations, CPS cross-trains separate networks, and teacher–student approaches use EMA-updated teachers to generate pseudo-labels.Related approaches also assess pseudo-label reliability, correct labels, or augment less-confident samples; HDVS instead combines heterogeneous branches, feature communication, and voting.Source: §2.2 · pp. 3–4

3 Method

HDVS pairs heterogeneous CNN and Transformer branches, using confidence-based voting for unlabeled pseudo-labels and feature communication for cross-branch features.

3.1 Preliminary

  • CPS-style co-training uses two encoder-decoder branches on labeled and unlabeled images, with each branch’s pixel-wise one-hot pseudo-label supervising the other.The supervised and unsupervised losses are combined with a weighting coefficient.Source: §3.1 · p. 4

3.2 Heterogeneous Dual-Branch Network

  • The heterogeneous network uses Transformer and CNN models, producing predictions from both original and feature-communication branches; labeled-image losses use manual ground truth.The Transformer is branch 0 and the CNN is branch 1.Source: §3.2 · pp. 4–5

3.3 Voting-based Pseudo-label Generation

  • The voting module compares each branch’s top-class confidence using a uniform random draw to select pseudo-labels, then entropy filtering removes the most uncertain 20% of pixels.The resulting filtered pseudo-labels supervise both branches.Source: §3.3 · p. 5, §3.2 · p. 5

3.4 Feature Communication Module

  • FCM communicates complementary encoder features across heterogeneous branches using aligned multi-level features, scaling coefficients, and dropout to add controlled perturbations.The module integrates features from different encoders and introduces controlled perturbations to enrich their interactions.Source: §3.4 · p. 6

3.5 Overall Pipeline of HDVS

  • For unlabeled images, HDVS weakly augments inputs, votes and filters labels, then strongly augments inputs for cross-supervision of original and communicated-feature predictions.The overall objective combines supervised and weighted unsupervised losses.Source: §3.5 · p. 7

4 Experiment

This section evaluates HDVS on VOC and Cityscapes, then examines its component effects, feature representations, and failure conditions.

4.1 Experiment Setup

  • Experiments use VOC Train/Aug partitions and Cityscapes labeled-data partitions, reporting validation mIoU with center crops on VOC and sliding windows on Cityscapes.Source: §4.1 · p. 7

4.2 Comparison with State-of-the-art Methods

  • On VOC, HDVS improves over diverse co-training by 1.5% (ResNet-50) and 1.0% (ResNet-101); on Cityscapes, it exceeds prior SOTA by 0.6% and 0.4% mIoU at 1/8 and 1/2.The VOC figures are average accuracy improvements; the Cityscapes gains are reported under the 1/8 and 1/2 partition protocols.Source: §4.2 · p. 8, Table 1 · p. 9, Table 2 · p. 9

4.4 Ablation Studies

  • Ablations favor heterogeneous branches: voting adds gains, including 9.4% at 183 labels, while FCM helps heterogeneous but can over-align homogeneous branches and outperforms heavier communication alternatives efficiently.FCM's communication scheme improves on the heterogeneous baseline while using fewer parameters than Concat+MLP and cross-attention.Source: §4.4 · pp. 8–11

4.3 Feature Visualizations

  • On Cityscapes 1/2 features, t-SNE visualizations show HDVS separates categories more clearly than CPS, especially eight classes whose CPS features overlap.Source: §4.3 · p. 8, Figure 6 · p. 11

4.5 Failure Case Analysis

  • Under VOC 1/16 supervision, divergent predictions in difficult regions can impair segmentation; early noisy pseudo-labels and FCM perturbations may transiently destabilize co-training.Source: §4.5 · p. 12

5 Conclusion

HDVS combines a heterogeneous dual-network structure and feature communication with collaborative voting to refine labels. It achieves significant performance improvements over state-of-the-art approaches on PASCAL VOC 2012 and Cityscapes.

5 Conclusion

  • HDVS combines heterogeneous dual networks, feature communication, and collaborative voting to strengthen co-confident labels and weaken noisy ones.Source: §5 · p. 13
  • HDVS achieves significant performance improvement against state-of-the-art approaches on both PASCAL VOC 2012 and Cityscapes.Source: §5 · p. 13
  • The authors state that source code and trained model weights will be made publicly available upon acceptance.Source: §5 · p. 13

Related papers

Loading 2610.37236v1…