Source-linked AI summary
Vision-Language-Guided Pseudo-Labels for Unsupervised Domain Adaptation in Semantic Segmentation for Waste Sorting
Udo Schlegel, Shubhangi, Gabriel Dax, Sai Rahul Kaminwar, Florian Karl, Thomas Seidl
TL;DR
Dense segmentation labels are costly to obtain under domain shift, especially in changing industrial environments. The paper addresses this with a cross-modal pseudo-labeling pipeline that uses SAM proposals, EVA-CLIP region–text matching, filtering, and optional BLIP verification to supervise target-domain self-training. Across synthetic-to-real driving and lab-to-factory waste sorting, the pipeline outperforms source-only training, with pseudo-label quality governing downstream success.
Problem
Dense pixel-wise labels are expensive and often impractical to collect for new operating conditions, including lab-to-factory industrial waste sorting.
Method
The pipeline chains SAM for class-agnostic region proposals, EVA-CLIP for region–text semantic labeling, confidence filtering, and optional BLIP verification to generate external target pseudo-labels.
Results
The pipeline consistently improves over source-only training on synthetic-to-real driving and lab-to-factory industrial waste-sorting benchmarks.
Takeaways & Limitations
Pseudo-label quality rather than quantity governs self-training success, making fidelity improvements through verification, cross-modal checks, and calibrated filtering practically important.
Takeaways & Limitations
The approach assumes a closed label set, routes open-set items to an unknown bucket, and remains limited on thin structures and other difficult regions.
Abstract
from arXiv · showhide
Obtaining labeled data for semantic segmentation in applied settings (e.g., autonomous driving, industrial waste sorting) is expensive and often infeasible at scale. We present a cross-modal pseudo-labeling pipeline that enables unsupervised domain adaptation without any target-domain annotations. The pipeline is built on two core foundation models: SAM generates class-agnostic region proposals, and EVA-CLIP assigns semantic labels based on region-text similarity, with confidence filtering ensuring that only reliable pseudo-labels are used for self-training a segmentation model. As an optional extension, BLIP provides language-grounded verification for ambiguous regions, thereby improving pseudo-label quality without altering the overall pipeline. Evaluated on two domain shifts, synthetic-to-real autonomous driving and, with a primary focus, lab-to-factory industrial waste sorting, the pipeline consistently improves over source-only baselines. Our results demonstrate that pseudo-label quality, not quantity, is a decisive factor in self-training under domain shift, and that cross-modal language grounding offers a practical path to reliable automatic annotation in deployment-critical applications.
1 Introduction
Dense target-domain segmentation labels are costly and difficult to collect under changing conditions and domain shift. The paper proposes an annotation-free cross-modal pipeline that uses foundation models to generate and verify pseudo-labels for target-domain self-training.
- Motivation: Dense pixel-wise annotations are expensive and impractical when target conditions vary across autonomous-driving and industrial-inspection deployments.Industrial waste sorting adds complex backgrounds, occlusions, dirt, and visually similar fine-grained categories.
- Approach: The pipeline uses SAM for class-agnostic target-region proposals and EVA-CLIP for semantic labels based on region–text similarity.Confidence filtering retains reliable region–label pairs, while optional BLIP captioning verifies ambiguous regions.
- Approach: Target supervision comes from pretrained foundation-model outputs rather than the segmentation model’s own confidence, reducing dependence on potentially miscalibrated predictions under shift.The segmenter is trained jointly on labeled source data and filtered pseudo-labeled target data.
- Results and contribution: The method requires no target-domain annotations and improves over a source-only DeepLabV3 baseline on both synthetic-to-real driving and lab-to-factory waste-sorting benchmarks.The pipeline produces an explicit pseudo-label bank and is designed to remain backbone-agnostic.
2 Related Work
Prior work addresses domain shift through adversarial alignment, self-training, foundation-model supervision, and open-vocabulary segmentation. This paper positions its method as a closed-set UDA pipeline that uses vision–language models upstream to create external pseudo-labels for a task-specific segmenter.
- Unsupervised domain adaptation: UDA segmentation methods primarily align source and target distributions adversarially or iteratively retrain on target pseudo-labels.Self-training methods use confidence thresholding, class balancing, and pseudo-label refinement to mitigate confirmation bias.
- Foundation models: SAM provides domain-agnostic class-agnostic masks, while vision–language models connect visual regions with textual semantics through contrastive image–text pretraining.These capabilities support using foundation models as proposal and semantic-supervision components.
- Pseudo-labeling: Pseudo-labeling under domain shift is vulnerable to confirmation bias because erroneous early labels can propagate and reinforce model errors.The paper motivates auxiliary supervision signals that bypass the segmentation model’s own confidence.
- Open-vocabulary segmentation: Open-vocabulary segmentation commonly combines mask generators with CLIP-like encoders to classify regions using natural-language prompts or class names.Such methods may adapt CLIP or use mask-aware strategies for direct open-vocabulary inference.
- Positioning: Unlike open-vocabulary inference methods, this approach operates in a closed-set UDA regime and uses EVA-CLIP predictions as external pseudo-labels for self-training a task-specific segmenter.The distinction concerns both the label setting and the downstream role of the vision–language model.
3 Our proposed cross-modal pseudo-labeling pipeline
The pipeline converts unlabeled target images into filtered, inspectable pseudo-labels using modular foundation models, then adapts a conventional segmentation network through joint self-training. SAM proposes regions, EVA-CLIP assigns labels and confidence scores, optional BLIP verifies ambiguous regions, and labeled pixels drive target-domain training.
- Pipeline overview: The pipeline has four stages: source training, target mask proposal and extraction, cross-modal pseudo-label generation and verification, and target self-training.These stages transform unlabeled target images into structured supervision without target-domain annotations.
- Stage I: Source-Domain Baseline Segmentation: A DeepLabV3 model with a ResNet-50 backbone is trained on labeled source data using cross-entropy and Dice losses to create the source-only baseline checkpoint.The checkpoint initializes self-training and serves as the quantitative source-only reference; it does not generate target pseudo-labels.
- Stage II: Target-Domain Mask Proposal and Region Extraction: Frozen SAM decomposes each target image into candidate object or part regions, then area filtering, non-maximum suppression, and context-aware masked crops produce usable region patches.The processing removes very small or highly overlapping proposals while retaining local objects and surrounding context.
- Stage III: Cross-Modal Pseudo-Labeling and Verification: EVA-CLIP compares each region embedding with text embeddings for a fixed shared label set, selecting the best-matching class and its cosine-similarity confidence.Class prompts use the form “a photo of a {class name}”, with optional synonyms.
- Stage III: Cross-Modal Pseudo-Labeling and Verification: Confidence filtering retains reliable region–label pairs, while optional BLIP caption verification can restore low-confidence regions when its label is sufficiently confident or agrees with the original prediction.Accepted regions are fused into a sparse pixel-wise pseudo-label map; uncovered pixels remain unlabeled and are ignored in the target loss.
- Stage IV: Target-Domain Self-Training: Joint self-training combines labeled source samples with pseudo-labeled target samples, computing target loss only on labeled pixels and weighting it with λ.The adapted DeepLabV3 model is evaluated on held-out target images against the source-only baseline and alternative adaptation strategies.
4 Experiments
Experiments cover synthetic-to-real driving and lab-to-factory waste sorting, with primary emphasis on the latter. Across settings, adaptation quality depends on pseudo-label reliability, while BLIP verification and AdaBN can provide additional gains.
- Evaluation settings: The pipeline is evaluated on GTA5-to-Cityscapes and LabWaste-to-RealWaste, with the industrial waste transfer serving as the primary deployment-motivated setting.The waste benchmark transfers from lab-captured images to conveyor-belt images from an operational facility.
- Pseudo-label quality: 0.542 → 0.934 → 0.960 EVA-CLIP Top-1 label accuracy on GTA5 improves from frozen to LoRA to Full FT source training.Stronger verifier adaptation increases the fraction of region-label pairs that remain confident and correct after filtering and fusion.
- Pseudo-label quality: 29.5% mIoU and 61.6% pixel accuracy result from adding BLIP to CLIP on GTA5→Cityscapes, versus 18.5% mIoU and 46.3% pixel accuracy for CLIP alone.BLIP also raises accepted-label coverage from 6.3% to 10.2%, recovering supervision for ambiguous regions.
- Target-domain segmentation: 26.3% mIoU is achieved on GTA5→Cityscapes with Full FT and BLIP, compared with 20.0% source-only and 20.0% with LoRA.Full FT alone reaches 25.0% mIoU, while BLIP provides a moderate additional gain in this setting.
- Target-domain segmentation: 18.0% mIoU is achieved by Full FT with BLIP on LabWaste→RealWaste, a +133.8% relative improvement over the 7.7% source-only baseline.Full FT without BLIP reaches 14.1% mIoU (+83% relative), while LoRA reaches 9.3% (+21%).
- Initialization and adaptation: 27.4% mIoU from AdaBN alone on LabWaste→RealWaste shows that target-domain normalization substantially reduces representational mismatch before pseudo-label training.Combining AdaBN with Full FT pseudo-labels reaches 30.2% mIoU, indicating additive gains from target-aligned initialization and higher-quality pseudo-labels.
5 Discussion
The discussion identifies pseudo-label reliability as the central determinant of self-training gains and highlights deployment constraints, structured errors, and scope limitations. It also outlines future evaluations aimed at separating backbone capacity from cross-modal supervision benefits.
- More reliable pseudo-labels yield larger target-domain gains, whereas increasing pseudo-label amount without improving accuracy offers limited benefit.The pipeline uses a standard DeepLabV3-R50 backbone to isolate pseudo-labeling effects from architectural advances.
- Verifier-level accuracy can diverge from segmentation gains when structured errors affect boundaries, small objects, or systematically confused classes.Spatially correlated boundary leakage and class-biased residual errors can be reinforced during self-training.
- Industrial deployment faces changing acquisition conditions, occlusion, contamination, and evolving object appearance, making controlled pseudo-label reliability important.The setting includes shifts in lighting, belt texture, and site-specific artifacts.
- Proposal redundancy and overconfident incorrect assignments require top-K selection, overlap suppression, class-based thresholds, or consistency checks.These controls address both supervision efficiency and confidently wrong labels under domain shift.
- The approach assumes a closed-label set, remains limited on thin structures and small objects, and should be compared cautiously with stronger UDA systems.The authors position the method as measuring how far an off-the-shelf segmenter can be pushed with foundation-model supervision.
- Future work will test stronger backbones, hybrid internal and external refinement, and larger multi-site and temporal industrial evaluations.Planned comparisons include SegFormer, ViT-based architectures, DAFormer, and PLSR.
6 Conclusion
The paper presents a modular, target-label-free pseudo-labeling pipeline for unsupervised domain adaptation in semantic segmentation. Evaluations on driving and waste-sorting domain shifts show improvements over source-only training, with pseudo-label quality governing self-training success.
- The pipeline provides a modular alternative for unsupervised domain adaptation in semantic segmentation without target-domain labels.
- SAM generates class-agnostic masks, EVA-CLIP assigns semantic labels, and confidence filtering selects supervision for self-training.Optional BLIP verification refines ambiguous regions without changing the self-training loop.
- The pipeline supports explicit, inspectable target pseudo-labels and straightforward integration into existing segmentation workflows.The conclusion describes foundation models as interchangeable with stronger models in future deployments.
- The method outperforms source-only training on GTA5-to-Cityscapes and LabWaste-to-RealWaste domain shifts.The evaluation covers synthetic-to-real driving and lab-to-factory industrial waste sorting.
- Pseudo-label quality, rather than quantity, governs self-training success when verifier errors are structured and reinforced during optimization.