Source-linked AI summary
Significance-aware Information Bottleneck for Domain Adaptive Semantic Segmentation
Yawei Luo, Ping Liu, Tao Guan, Junqing Yu, Yi Yang
TL;DR
Semantic segmentation domain adaptation is hindered by complex latent representations and unstable adversarial alignment. SIBAN introduces a significance-aware information bottleneck for feature purification before adaptation, and reports leading feature-space results that match state-of-the-art output-space methods. The method also addresses stronger constraints on rare classes through channel-wise significance.
Problem
Feature-space adversarial adaptation often fails in semantic segmentation because latent representations contain nuisance factors and make stable alignment difficult.
Method
SIBAN inserts a significance-aware information bottleneck before adversarial adaptation to remove task-independent information while balancing constraints across semantic channels.
Results
SIBAN achieves leading feature-space adaptation results and performs on par with state-of-the-art input/output-space adaptations across two simulated-to-real tasks.
Takeaways & Limitations
The results place feature-space and output-space unsupervised domain adaptation methods on the same starting line for semantic segmentation.
Takeaways & Limitations
Standard information bottlenecks can disproportionately constrain infrequent-class features, and the theoretical analysis assumes independent task-dependent and task-independent feature parts.
Abstract
from arXiv · showhide
For unsupervised domain adaptation problems, the strategy of aligning the two domains in latent feature space through adversarial learning has achieved much progress in image classification, but usually fails in semantic segmentation tasks in which the latent representations are overcomplex. In this work, we equip the adversarial network with a "significance-aware information bottleneck (SIB)", to address the above problem. The new network structure, called SIBAN, enables a significance-aware feature purification before the adversarial adaptation, which eases the feature alignment and stabilizes the adversarial training course. In two domain adaptation tasks, i.e., GTA5 -> Cityscapes and SYNTHIA -> Cityscapes, we validate that the proposed method can yield leading results compared with other feature-space alternatives. Moreover, SIBAN can even match the state-of-the-art output-space methods in segmentation accuracy, while the latter are often considered to be better choices for domain adaptive segmentation task.
1. Introduction
Domain shift and overly complex latent features make feature-space adaptation difficult for semantic segmentation. SIBAN purifies features with a significance-aware information bottleneck before adversarial adaptation, achieving leading feature-space results and matching state-of-the-art input/output-space methods.
- Motivation: Domain shift causes models trained on simulated images to perform poorly on realistic images despite abundant simulated labels.The domains have different data distributions, creating a central challenge for semantic segmentation.
- Approach: SIBAN applies an information bottleneck before adversarial adaptation to filter nuisance factors while retaining pure semantic information.The bottleneck explicitly constrains mutual information so the representation preserves task-dependent factors and removes task-independent ones.
- Motivation: High-dimensional segmentation features encode nuisance visual factors that can mislead domain alignment and destabilize adversarial training.These representations may include appearance, shape, and context beyond task-dependent semantic information.
- Findings: SIBAN combines information bottleneck and adversarial learning, stabilizes adversarial training, and achieves leading feature-space adaptation results.The paper reports performance on par with state-of-the-art input/output-space adaptations.
- Approach: The significance-aware layer accounts for channel-wise semantic significance and balances information constraints across classes.This design targets unbalanced pixel frequencies and helps maintain performance on rare classes.
2. Related Work
Domain adaptation research primarily addresses distribution shift, with latent feature-space methods showing limited success for segmentation and information bottleneck theory offering an explicit information-control principle.
- Domain adaptation: Domain adaptation bounds emphasize source-domain loss, domain divergence, and shared error between source and target domains.Recent work commonly focuses on reducing domain divergence because the other terms are treated as comparatively well studied or sufficiently low.
- Feature-space adaptation: Latent feature-space adaptation has generally succeeded in classification but often fails to transfer effectively to semantic segmentation.Segmentation features are more complex and can encode task-independent factors that hinder alignment.
- Information bottleneck: Information bottleneck theory constrains mutual information between input X and latent representation Z while encouraging Z to predict labels Y.The constraint can be expressed through an upper bound on the KL divergence between the joint distribution and the product of marginals.
3. Method
SIBAN combines feature-space adversarial adaptation with an information bottleneck that purifies latent representations before alignment. Its significance-aware weighting preserves important channels, especially for infrequent classes, while supporting stable adversarial training.
- Problem setting: SIBAN formulates unsupervised adaptation with labeled source images and unlabeled target images, learning a segmentation model G composed of feature extractor F and classifier C.The latent representation is z = F(x), and predictions are produced by C ◦ F(x).
- Adversarial adaptation: The baseline trains F and C with supervised source task loss while adversarially training F against a discriminator D to make source and target features domain-invariant.The discriminator distinguishes features from the two domains, while the feature extractor aims to confuse it.
- Information-constrained adaptation: SIBAN adds a variational information bottleneck to F, constraining latent information to filter task-independent nuisance factors and facilitate feature alignment and adversarial training.The prior marginal r(z) is modeled as a standard Gaussian, and the bottleneck penalizes dependence between inputs and representations through KL divergence.
- Information-constrained adaptation: The bottleneck multipliers βS and βT are adaptively updated by dual gradient descent so information constraints can be strengthened or relaxed around the target threshold Ic.Training alternates optimization of F, C, D, βS, and βT until convergence.
- Significance-aware information bottleneck: SIB detects channel-wise significance Vsig and weights the information constraint by 1 − Vsig, imposing less compression on more significant channels.This significance-aware design addresses the stronger standard-IB constraint suffered by infrequent classes and is applied to source and target latent features before classification or discrimination.
- Network architecture: The purified source representation is used for supervised segmentation and adversarial losses, while the purified target representation uses adversarial and information-constraint losses without target annotations.The SIB is attached to the output of F’s last convolutional layer; D is a fully convolutional binary classifier.
- Theoretical insight: The method is mathematically linked to domain-adaptation theory by pushing task-independent source and target factors toward the same standard Gaussian distribution, reducing their H∆H-distance toward zero.The paper states that this optimizes an upper bound related to target error.
4. Experiments
Experiments evaluate SIBAN on synthetic-to-real adaptation using GTA5 or SYNTHIA as source and Cityscapes as target. SIBAN improves feature-space adaptation, stabilizes adversarial training, protects uncommon classes, and reaches performance comparable to output-space methods.
- Experimental Setup: SIBAN is evaluated on GTA5 →Cityscapes and SYNTHIA →Cityscapes using Cityscapes as the target domain.GTA5 provides 24,966 automatically annotated images, while SYNTHIA provides 9,400 synthetic images compatible with Cityscapes classes.
- Experimental Setup: The experiments use DeepLab-v2 with ResNet-101 and VGG-16-based FCN8s backbones, with a feature-space adversarial method as baseline.SIBAN's significance-aware layer uses a 1 × 1 convolution, ReLU, and Sigmoid to produce channel-wise significance weights.
- Comparative Studies: +16.3% on VGG-16 and +6.0% on ResNet-101: SIBAN improves over source-only segmentation on GTA5 →Cityscapes.SIBAN also improves mIoU by over +2.5% compared with FCNs, AdaptSeg, and CyCADA.
- Sensitivity to Constraint: SIBAN outperforms IBAN across all tested information constraints, with an appropriate I_c range of 200–400.Too small an I_c removes essential information, while too large an I_c introduces noise and degrades SIBAN toward the baseline.
- Training Stability: SIBAN combines good training stability with outstanding segmentation performance, whereas standard IB improves stability at the cost of less informative features.Without IB, LossD quickly drops because the discriminator overpowers the generator; IB and SIB constrain discriminator performance.
- A-distance: SIBAN achieves superior domain-adaptation performance with a relatively small A-distance, indicating a better trade-off between feature purification and domain alignment.IBAN has a slightly smaller A-distance, but the paper attributes this to discarding necessary information.
- Ablation Studies: Appending a significance-aware layer improves standard IB by 1.5%, adaptive βS/βT adds 0.4%, and extra discriminators add 0.8% or 2.3%.The additional discriminators operate on significance tensors and segmentation maps, respectively.
5. Conclusion
The conclusion presents SIBAN as a feature-space domain-adaptive semantic-segmentation method that purifies features before adversarial adaptation. Across two simulated-to-real tasks, it achieves leading feature-space results and matches state-of-the-art output-space accuracy.
- Conclusion: SIBAN performs significance-aware feature purification before adversarial adaptation to ease feature alignment and stabilize adversarial training.The method is proposed for domain-adaptive semantic segmentation.
- Conclusion: Across two challenging simulated-to-real tasks, SIBAN yields leading results among feature-space methods and matches state-of-the-art output-space methods in segmentation accuracy.The conclusion frames feature-space and output-space UDA methods as reaching the same starting line for semantic segmentation.
Hyper-parameter Analysis
The hyper-parameter analysis examines how the information constraint I_c controls feature purification and adaptive βS/βT during training. A relatively large I_c between 200 and 300 can constrain the information to a specific threshold.
- Hyper-parameter Analysis: For I_c values of 100, 200, and 300, the analysis reports KL-divergence curves and adaptive βS/βT values over training.These diagnostics assess information constraint behavior in both source and target domains.
- Hyper-parameter Analysis: Choosing I_c between 200 and 300 makes the information constraint readily enforceable to a specific threshold.The passage describes this range as relatively large.
Feature Distribution Visualization
The feature-distribution visualization uses t-SNE to compare source and target latent features at global and class levels. SIBAN shows good alignment in both views, unlike the nonadaptive model and the partially mismatched IBAN features.
- Visualization Method: The visualization maps paired source and target latent features from high-dimensional space into 2-D using t-SNE.The first row labels maps by domain to evaluate marginal, or global, alignment; class-level alignment is also examined.
- Comparison: The nonadaptive model produces poor global and class-level alignment, limiting direct application of a source-trained classifier to target samples.The passage attributes this to limited generalization ability on target samples.
- Comparison: IBAN aligns marginal domain distributions but mismatches some features from different semantic classes.This indicates that global alignment does not ensure class-level alignment in the visualization.
- SIBAN: SIBAN achieves good global and local feature alignment between domains, helping explain its leading feature-space adaptation results.The visualization includes both domain-level and class-level views.
More Qualitative Results
The paper presents qualitative domain-adaptive segmentation results and visualizes learned source- and target-domain representations with t-SNE across three models.
- Figure 9 compares qualitative segmentation outputs from the baseline method, IBAN, and SIBAN against the ground-truth label map.
- Figure 8 visualizes learned representations zS and zT for the Non-adapted model, IBAN, and SIBAN using t-SNE.
- The first t-SNE row labels points by domain, with red for the source domain and blue for the target domain.
- The second t-SNE row labels points by different classes.