PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging the Gap between Labeled and Unlabeled Data via Unified Flow with Feature Memory Bank

Shanwen Wang, Xin Sun, Danfeng Hong, Junyu Dong, Patrick Le Callet

arXiv:2608.16681v1cs.CVcs.AI

TL;DR

Independent training lets labeled data dominate remote-sensing semi-supervised segmentation and degrades pseudo-label quality; UFFM unifies pseudo-labeling, supervision, and class-feature memory. It achieves the highest mean mIoU across almost all categories on the evaluated remote-sensing datasets.

  • Problem

    Independent processing of labeled and unlabeled data causes labeled samples to dominate optimization, degrading pseudo-label quality and producing inconsistent semantic representations.

  • Method

    UFFM combines VFM and domain-teacher pseudo-labeling with unified supervision and a class-specific feature memory bank that aligns labeled and unlabeled features.

  • Results

    UFFM achieves the highest mean mIoU across almost all categories on the evaluated remote-sensing datasets.

  • Takeaways & Limitations

    UFFM bridges optimization and feature-representation gaps between labeled and unlabeled data in remote-sensing semi-supervised segmentation.

  • Takeaways & Limitations

    The paper’s future work is to develop a general-purpose semi-supervised remote-sensing foundation model supporting diverse downstream vision tasks.

Abstract

from arXiv · show

Although semi-supervised semantic segmentation ($\text{S}^4$) utilizes abundant unlabeled data to reduce manual labeling burdens, independent training of labeled and unlabeled data causes the former to dominate, which severely degrades pseudo-label quality. To address this challenges, we propose a novel remote sensing (RS) $\text{S}^4$ method via unified flow with feature memory bank (UFFM). Specifically, UFFM comprises two key innovations: unified flow (UF) and feature memory bank (FMB). The UF is a new training flow that generates less biased pseudo-labels by combining an external visual foundation model (VFM) with an RS domain teacher, and jointly optimizes labeled and pseudo-labeled data under a unified training objective. The FMB is a novel memory module for $\text{S}^4$ that dynamically updates class-specific features during training and reduces the feature discrepancy between labeled and unlabeled data through class-feature alignment. To verify the effectiveness of our model, we conduct extensive experiments on RS datasets. The experimental results show the superiority of our method over SOTA $\text{S}^4$ methods. Moreover, the results demonstrate the effectiveness of our contributions in bridging the optimization and feature representation gap between labeled and unlabeled data. Our code is released at \href{https://github.com/wangshanwen001/RS-UFFM}{https://github.com/wangshanwen001/RS-UFFM}.

I. INTRODUCTION · II. RELATED WORK · A. Semi-supervised Semantic Segmentation

Remote sensing semantic segmentation is hindered by costly pixel-level annotation, motivating semi-supervised learning that exploits unlabeled data but faces optimization and pseudo-labeling gaps. UFFM addresses these issues through unified flow and a feature memory bank, while related work highlights both established S4 strategies and the difficulty of transferring them to remote sensing.

  • I. INTRODUCTION: Manual labeling of extensive remote sensing datasets is time-consuming and labor-intensive, motivating S4 methods that combine limited labeled samples with abundant unlabeled data.
  • I. INTRODUCTION: UFFM proposes unified flow that integrates labeled and unlabeled data through collaborative pseudo-labeling, data fusion, and a shared training objective.Its unified flow fuses external visual foundation model knowledge with domain-teacher expertise to generate less biased pseudo-labels.
  • I. INTRODUCTION: Traditional S4 training can let explicitly annotated data dominate, degrading pseudo-label quality and widening the gap between labeled and pseudo-labeled data.AllSpark improves unlabeled accuracy but can degrade labeled performance because reconstructing labeled samples from unlabeled features introduces noise into discriminative labeled features.
  • I. INTRODUCTION: UFFM also introduces a feature memory bank that maintains long-term category feature memory and aligns class features to reduce representation discrepancies between labeled and unlabeled data.The method reports superior performance across both labeled and unlabeled data compared with existing S4 baselines.
  • II. RELATED WORK: Related work surveys S4 methods and recent advances specifically targeting the remote sensing domain.
  • A. Semi-supervised Semantic Segmentation: Semantic segmentation supports land cover classification, urban modeling, and environmental monitoring, but supervised approaches require costly pixel-level annotations, encouraging S4 research.
  • A. Semi-supervised Semantic Segmentation: Classical computer-vision S4 approaches use consistency regularization, pseudo-labeling, teacher–student frameworks, and adversarial learning; FixMatch and UniMatch exemplify these directions.UniMatch v2 replaces traditional ResNet backbones with vision foundation models such as DINOv2, while AllSpark reconstructs labeled features from unlabeled representations.
  • A. Semi-supervised Semantic Segmentation: Although general S4 methods perform well on natural-image segmentation, remote sensing introduces scale variation, visual similarity, complex backgrounds, and intricate textures that hinder direct transfer.

B. Remote Sensing Semi-supervised Semantic Segmentation · III. METHODS · A. Main Framework

Remote sensing semi-supervised segmentation has adopted domain-specific refinement, contrastive learning, and foundation-model teachers, but optimization imbalance still lets labeled data dominate and harms pseudo-label quality. UFFM addresses this by jointly training labeled and unlabeled data through unified flow and feature memory mechanisms within a teacher–student framework.

  • B. Remote Sensing Semi-supervised Semantic Segmentation: RS S4 methods address multi-scale variation through contextual label refinement or uncertainty consistency and reduce inter-class similarity using contrastive learning with cross-teacher–student attention.
  • B. Remote Sensing Semi-supervised Semantic Segmentation: Foundation-model approaches improve RS S4 by using VLMs for pseudo-label quality or multiple VFMs with distillation-and-fusion as teachers.
  • B. Remote Sensing Semi-supervised Semantic Segmentation: Existing RS S4 methods often overlook optimization imbalance, allowing the supervised labeled pathway to dominate and produce low-quality unlabeled-data pseudo-labels.
  • A. Main Framework: VFM-generated class-agnostic masks are mapped to semantic classes by the teacher, while the FMB performs feature-memory updating and class-feature alignment.
  • III. METHODS: The methods section presents UFFM’s main framework, then introduces the unified-flow and feature-memory-bank principles.
  • A. Main Framework: UFFM integrates labeled and unlabeled samples instead of treating them as separate training sets, using a specially designed weighting strategy for joint optimization.
  • A. Main Framework: UFFM unifies supervised and unsupervised learning in a teacher–student framework, applying weak and strong augmentations to unlabeled data for pseudo-label generation and robust feature learning.
  • A. Main Framework: UFFM introduces unified flow and FMB to resolve labeled-data optimization dominance and its resulting degradation of pseudo-label quality.

B. Unified Flow

Unified Flow jointly processes labeled and unlabeled data by combining a visual foundation model with an in-domain teacher to produce less biased pseudo-labels. It filters unreliable predictions, fuses the resulting supervision with ground truth, and computes a unified training loss.

  • Unified Flow: UF replaces independent labeled–unlabeled processing with a joint pseudo-labeling and data-fusion strategy.The strategy combines labeled and unlabeled supervision within a unified training flow.
  • Unified Flow: Cross-domain collaboration between a VFM and an in-domain teacher mitigates biases inherited from labeled data when generating pseudo-labels.The VFM contributes external knowledge, while the teacher supplies domain-specific predictions.
  • Unified Flow: VFM regions are retained only when class consistency and mean confidence satisfy both reliability thresholds; otherwise, teacher predictions or discarded pixels are used.For unreliable VFM regions, UF reverts to standard teacher pseudo-labels, while unreliable teacher pixels are discarded.
  • Unified Flow: UF separately weights pseudo-labels and ground-truth labels, then combines them to compute the loss under unified supervision.This weighted fusion completes the UF pipeline.

C. Feature Memory Bank

The Feature Memory Bank (FMB) updates class-specific feature prototypes using labeled and pseudo-labeled data, aligning their representations and deriving prototype-based loss weights for unified training.

  • Feature Memory Bank: FMB reduces feature discrepancy by aligning class-specific representations across labeled and unlabeled samples that share the same category.Its workflow comprises memory-bank updating and loss computation stages.
  • Memory bank updating: FMB applies the same prototype-update mechanism to labeled and unlabeled data, using ground-truth labels or UF pseudo-labels to assign pixels to classes.For each class, it aggregates the corresponding batch pixel features into a mean feature vector before prototype updating.
  • Memory bank updating: An EMA mechanism updates normalized class prototypes while preserving shared characteristics between labeled and unlabeled data.The EMA uses decay parameter α and training step t.

IV. EXPERIMENTS

The experiments evaluate UFFM on benchmark remote sensing datasets through comparisons with state-of-the-art methods and ablation studies of its proposed modules.

  • UFFM is evaluated on benchmark remote sensing datasets.
  • Comparative experiments assess UFFM against several state-of-the-art methods.
  • Comprehensive ablation studies validate the effectiveness of each proposed module.

A. Datasets · DeepGlobe:

The study uses established high-resolution remote sensing benchmarks for semantic segmentation, detailing their spatial scales, image contents, categories, and patch-based preparation. DeepGlobe is extensively described, while ISPRS Potsdam provides a second benchmark with finer spatial resolution.

  • DeepGlobe:: DeepGlobe is widely used for remote sensing image semantic segmentation and land cover mapping.
  • DeepGlobe:: DeepGlobe has 0.5-meter spatial resolution and includes 803 satellite images sized 2448 × 2448 pixels.
  • DeepGlobe:: DeepGlobe covers seven categories: Urban land, Agriculture land, Rangeland, Forest land, Water, Barren land, and Unknown.
  • DeepGlobe:: Cropping DeepGlobe images into 512 × 512 patches produces 20,075 images, split into 12,045 training, 4,015 validation, and 4,015 test patches.
  • DeepGlobe:: ISPRS Potsdam is a widely used benchmark for semantic segmentation on high-resolution remote sensing images.
  • DeepGlobe:: Potsdam has 0.05-meter spatial resolution and contains 38 satellite images, each measuring 6000 × 6000 pixels.
  • DeepGlobe:: Potsdam covers six categories: impervious surfaces, buildings, low vegetation, trees, cars, and background.
  • DeepGlobe:: Potsdam images are cropped into 512 × 512-pixel patches, yielding 5,472 im.

ISPRS-Potsdam: · B. Data Augmentation and Experiment Settings · C. Evaluation metrics

The experiments use differentiated augmentation for labeled and unlabeled images, evaluate UFFM across multiple supervision regimes, and measure performance with foreground-class mIoU under a standard protocol.

  • B. Data Augmentation and Experiment Settings: Labeled images receive geometric transformations and Gaussian blurring, including scaling, horizontal and vertical flipping, and aspect-ratio warping.The augmentation pipeline combines several geometric operations with Gaussian blur.
  • B. Data Augmentation and Experiment Settings: Unlabeled images use geometric transformations for weak augmentation, while strong augmentation adds photometric transformations, Gaussian blur, and CutMix.Weak and strong augmentation therefore differ in both transformation type and intensity.
  • B. Data Augmentation and Experiment Settings: Experiments run for 50 epochs on one NVIDIA RTX H100 GPU with CUDA v11.7, using SAM 3 as VFM and DINOv2-small backbones for student and teacher.These settings define the computational environment and the principal UFFM model components.
  • B. Data Augmentation and Experiment Settings: Training evaluates 1%, 5%, and 10% labeled partitions with remaining unlabeled samples on the ISPRS Potsdam and DeepGlobe datasets.The experiments compare performance across three supervision regimes.
  • C. Evaluation metrics: Following prior remote-sensing semi-supervised segmentation protocols, mean Intersection-over-Union (mIoU) is the primary performance metric.The metric is used to assess model performance consistently with previous methods.
  • C. Evaluation metrics: Unknown and background pixels are excluded, with mIoU averaged over six foreground classes for DeepGlobe and five for ISPRS-Potsdam.The class averaging differs by dataset while retaining the same primary metric.
  • C. Evaluation metrics: For each class k, mIoU uses TP_k, FP_k, and FN_k, representing true positives, false positives, and false negatives, across K total classes.The notation specifies the confusion components and class count used in the metric calculation.

D. Quantitative Results compared to SOTA · E. Visualization Results compared to SOTA · F. Ablation Study

UFFM is evaluated against SOTA methods on DeepGlobe and ISPRS-Potsdam using 1%, 5%, and 10% labeled-data ratios, with quantitative and visual results favoring UFFM. Ablation experiments further examine how its components bridge optimization and feature-representation gaps between labeled and unlabeled data.

  • D. Quantitative Results compared to SOTA: Experiments compare UFFM with ten SOTA methods on DeepGlobe and ISPRS-Potsdam under 1%, 5%, and 10% labeled-data ratios.Comparative networks follow the default configurations reported for their respective methods.
  • D. Quantitative Results compared to SOTA: Traditional S4 methods such as FixMatch and UniMatch perform poorly because they overlook remote-sensing domain-specific challenges.The comparison distinguishes general semi-supervised approaches from methods designed specifically for remote sensing.
  • D. Quantitative Results compared to SOTA: UFFM achieves the highest mean mIoU across almost all categories on the evaluated remote-sensing datasets.The authors attribute this advantage to addressing pseudo-label degradation caused by labeled-data-dominated training.
  • E. Visualization Results compared to SOTA: On ISPRS-Potsdam, existing methods frequently misclassify Low vegetation and Tree in complex regions where class boundaries overlap or intermingle.These errors are highlighted in black dashed boxes in the qualitative comparison.
  • E. Visualization Results compared to SOTA: On DeepGlobe, baselines confuse water and rangeland with Agriculture, or falsely predict Barren land in scenes where it is absent.The visualization identifies these errors particularly for FixMatch, UniMatch, AllSpark, and DWL.
  • E. Visualization Results compared to SOTA: UFFM produces the most accurate and robust DeepGlobe segmentations, preserving fine-grained regions and structural boundary details better than competing methods.Even SemiEarth, which performs relatively better, retains large-scale regional errors.
  • F. Ablation Study: The ablation study validates UFFM’s component design and analyzes how the method bridges optimization and feature-representation gaps between labeled and unlabeled data.The subsection provides detailed ablation experiments and an in-depth explanation of these effects.

1) Ablation of Components: · 2) Hyperparameters Analysis of UFFM: · 3) Backbone Investigation:

UFFM’s ablations show that unified flow and feature memory bank each improve semi-supervised segmentation, while hyperparameter and backbone studies identify robust operating choices and architecture-dependent dataset effects.

  • 1) Ablation of Components:: Ablation experiments compare Baseline, UF, and UF + FMB under 1% labeled data with DINOv2-small and SAM 3.The Baseline omits both proposed components.
  • 1) Ablation of Components:: UF improves mIoU over the baseline by 3.59% on DeepGlobe and 4.23% on Potsdam.
  • 1) Ablation of Components:: FMB adds further mIoU gains of 0.61% on DeepGlobe and 1.41% on Potsdam.Together, the ablations support the effectiveness of UF and FMB for RS S4.
  • 2) Hyperparameters Analysis of UFFM:: mIoU peaks at τconsistency ≈0.6 as the consistency threshold first increases performance and then causes it to decline.Very low thresholds retain substantially disagreeing predictions, whereas very high thresholds revert most pixels to the original teacher.
  • 2) Hyperparameters Analysis of UFFM:: Across τconf values from 0.75 to 0.95, mIoU follows an inverted U-shape, increasing before decreasing.Lower thresholds admit many low-quality pseudo-labels, introducing noise and hindering training.
  • 3) Backbone Investigation:: mIoU on the RS dataset consistently improves as DINOv2 and DINOv3 backbone size increases for both student and teacher models.The results indicate that UFFM accommodates backbones across different parameter scales.
  • 3) Backbone Investigation:: DINOv3 consistently outperforms DINOv2 on DeepGlobe but performs worse on Potsdam, likely because its larger patch size discards fine-grained information.The explanation is especially relevant to Potsdam’s 0.05m spatial resolution.
  • 3) Backbone Investigation:: DINOv2-small is selected as UFFM’s default backbone because it provides highly stable training and is adequate for most cases.

4) Ablation Study of UF: · 5) t-SNE Visualizations of Learned Representations: · V. CONCLUSION

The ablation study supports UF’s two-stage pseudo-labeling rationale, while t-SNE visualizations show progressively improved class separation with UF and FMB. UFFM addresses labeled–unlabeled optimization and representation gaps in RS semantic segmentation but remains limited to that task.

  • 4) Ablation Study of UF:: UF decouples pseudo-label generation by first producing anonymous labels with a VFM and then assigning semantic categories using an RS-domain teacher.This design is compared with direct pseudo-label generation by the RS teacher or VFM.
  • 4) Ablation Study of UF:: The standalone RS teacher achieves the lowest performance, while VFM-only pseudo-labels marginally improve results because the VFM lacks domain-specific RS knowledge.The passage states that the improvement over the RS teacher is slight despite the VFM’s large-scale pre-training.
  • 5) t-SNE Visualizations of Learned Representations:: The t-SNE evaluation compares learned representations for the Baseline, Baseline with UF, and the complete model combining UF and FMB.Visualizations are generated for three configurations across the DeepGlobe and Potsdam datasets.
  • 5) t-SNE Visualizations of Learned Representations:: In the baseline, different classes overlap substantially and same-class features are loosely clustered, indicating limited class discrimination.The within-class looseness is highlighted by a black dashed ellipse in the visualizations.
  • V. CONCLUSION: UFFM addresses labeled-sample dominance by introducing Unified Flow and Feature Memory Bank within a unified teacher–student framework.UF synergizes Vision Foundation Models with domain-specific teacher models to mitigate the stated labeled–unlabeled training problem.
  • V. CONCLUSION: UFFM’s applicability is currently confined to semantic segmentation, limiting its use across diverse remote-sensing downstream tasks.The paper identifies aerial object detection and land-use change detection as examples of other tasks.
  • V. CONCLUSION: Future work will develop a general-purpose semi-supervised foundation model for RS imagery that learns transferable representations from limited labeled data.The proposed direction is intended to accommodate diverse downstream vision tasks.
  • V. CONCLUSION: UFFM is presented as the first RS S4 framework addressing optimization and feature representation gaps between labeled and unlabeled data.The conclusion identifies this gap as the framework’s central target.
Loading 2608.16681v1…