Spectral Invariant Fusion and Refinement for Domain Generalized Semantic Segmentation

Motivation

• DGSS methods struggle to balance domain robustness with the fine-grained spatial cues needed for accurate segmentation, while direct fusion of pretrained VLM and VFM priors faces misalignment and domain-specific appearance entanglement.DGSS has no target-domain information during training, unlike domain adaptation.Source: §I · p. 1

Method

• SIFR uses frozen CLIP semantic priors and DINOv2 structural representations in Spectral-Guided Invariant Fusion, which suppresses domain-variant appearance responses while enhancing stable cues.SGIF combines token-level cross-attention with frequency-domain refinement to support complementary interaction between the pretrained representations.Source: §I · pp. 1–2

Results

• 71.30 mIoU on Cityscapes, 62.54 on BDD100K, and 71.96 on Mapillary in synthetic-to-real, with gains up to +1.32 points over previous methods.SIFR also reports 67.60 and 78.48 mIoU on BDD100K and Mapillary, respectively, in real-to-real evaluation; Fig. 1 reports best mIoU on representative real-to-real and adverse-weather benchmarks.Source: Figure 1 · p. 2, §I · p. 2

Method

• Spectral Decoder Refinement uses domain-stable normalization and frequency-guided enhancement to stabilize multi-scale decoder features and preserve informative components for precise segmentation under domain shift.Together, SGIF and SDR apply spectral modeling at fusion and refinement stages.Source: §I · p. 2

A. Domain Generalized Semantic Segmentation

• DGSS approaches diversify source data or learn domain-invariant representations, with architectural designs also used to strengthen cross-domain modeling and dense prediction.Foundation-model approaches include synthetic data generation, model adaptation, and semantic guidance.Source: A. Domain Generalized Semantic… · p. 2
• How to use heterogeneous pretrained representations under domain shift, especially when their cues differ in domain sensitivity, remains an open question.Source: A. Domain Generalized Semantic… · p. 3

B. Vision Foundation Models

• VFMs offer transferable representations and dense-prediction cues, but existing work gives limited attention to disentangling domain-sensitive appearance statistics from transferable structure.Spectral modeling is presented as a complementary mechanism for exposing and suppressing domain-variant responses while preserving structure-aware cues.Source: B. Vision Foundation Models · p. 3

C. Frequency Modeling Under Domain Shift

• SIFR differs from prior frequency-aware methods by regulating heterogeneous VLM–VFM interaction and multi-scale decoder refinement, rather than adapting only a single pretrained visual stream.Prior work explored input-level frequency perturbation and feature-level filtering or modulation.Source: C. Frequency Modeling Under Domain Shift · pp. 3–4

A. Overview

• Frozen VLM and VFM encoders provide semantic and structural priors; SGIF fuses them before the pixel decoder and SDR refines its multi-scale features before text-guided mask decoding.SGIF suppresses domain-sensitive interference during fusion, while SDR mitigates residual bias from feature aggregation and dense decoding.Source: A. Overview · p. 4, Figure 3 · p. 5

B. Spectral-Guided Invariant Fusion

• SGIF combines token-level cross-attention over concatenated features with a parallel branch that adaptively reweights each stream’s frequency responses.The spectral responses are inverse-transformed, converted back to tokens, and combined with the cross-attention result through residual addition and convolution.Source: B. Spectral-Guided Invariant Fusion · p. 4
• The fusion strategy combines complementary semantic interactions and domain-robust structural cues to yield more discriminative and robust intermediate features under domain shift.The SDR caption describes its subsequent decoder refinement as low-frequency-statistics normalization followed by adaptive enhancement of segmentation-relevant frequencies.Source: B. Spectral-Guided Invariant Fusion · p. 5, Figure 3 · p. 5

C. Spectral Decoder Refinement

• SDR separates suppression of residual domain bias from preservation of segmentation-relevant spectral information through a suppress-then-select design.DS-Norm stabilizes feature statistics; FGE then adaptively emphasizes informative frequencies for robust mask prediction.Source: C. Spectral Decoder Refinement · p. 5
• For each of four pixel-decoder scales, SDR extracts a low-frequency component and computes channel-wise mean and standard deviation from it.The low-frequency region is controlled by filter ratio ρ; these statistics normalize the original full-spectrum decoder feature.Source: C. Spectral Decoder Refinement · p. 5
• FGE generates a frequency mask from pooled channel-wise average, maximum, and minimum magnitude responses, preserving spatial frequency resolution.A lightweight convolution and sigmoid produce the mask, which reweights the normalized spectral representation before inverse Fourier transformation.Source: C. Spectral Decoder Refinement · p. 5
• SDR suppresses residual domain-sensitive statistical deviations while emphasizing structurally informative spectral responses, providing more robust multi-scale features for mask prediction under domain shift.Source: C. Spectral Decoder Refinement · p. 5

D. Training Objective

• The training objective combines standard segmentation loss with a pixel-level vision-language alignment objective.The alignment loss enforces semantic consistency between pixel-level visual representations and textual class embeddings.Source: D. Training Objective · p. 5
• Binary cross-entropy and Dice losses supervise mask prediction, while classification loss supervises query prediction.The pixel-text alignment loss uses class prompts with learnable context tokens and a frozen CLIP text encoder.Source: D. Training Objective · p. 5
• Normalized pixel-level visual features are compared with normalized class text embeddings using inner-product similarity scaled by a fixed temperature of 0.07.Dense similarity scores are supervised with pixel-wise cross-entropy against ground-truth labels; ignored pixels are excluded.Source: D. Training Objective · pp. 5–6

A. Experiment Settings

• Evaluation covers five driving-scene benchmarks across synthetic-to-real, real-to-real, and real-to-adverse-weather settings, using mIoU.The benchmarks include GTA5, Cityscapes, BDD100K, Mapillary, and ACDC; ACDC spans fog, rain, snow, and nighttime.Source: A. Experiment Settings · p. 6
• CLIP visual and text encoders and the DINOv2 backbone are frozen; the remaining trainable framework components are jointly optimized end to end.Training uses a single stage without stage-wise pretraining or alternating optimization.Source: A. Experiment Settings · pp. 6–7
• Training runs for 30,000 iterations with AdamW at a base learning rate of 1e-4, and inference uses 512 × 512 sliding windows with stride 341 × 341.Images are resized to 512 × 512, with batch size 2 per GPU.Source: A. Experiment Settings · p. 7

B. Comparison with State-of-the-Art Methods

• The comparison includes foundation-model-based and conventional DGSS methods across three transfer settings.It emphasizes recent foundation-model methods, including CLOUDS, VLTSeg, REIN, SET, MFuser, DepthForge, and tqdm.Source: B. Comparison with State-of-the-Art… · p. 7
• SIFR achieves the best performance on both real-to-real targets, improving over MFuser by +2.02 mIoU on C →B and +0.38 on C →M.It also surpasses DepthForge by +1.41 on C →B and +2.55 on C →M; Tab. I covers Cityscapes →BDD100K + Mapillary.Source: B. Comparison with State-of-the-Art… · p. 7, Table I · p. 7
• SIFR obtains the best performance on all three synthetic-to-real targets, improving over MFuser by +0.06, +1.46, and +0.82 mIoU on Cityscapes, BDD100K, and Mapillary.Tab. II covers GTA5 →Cityscapes + BDD100K + Mapillary.Source: B. Comparison with State-of-the-Art… · p. 7
• On Cityscapes →ACDC, SIFR achieves the best overall performance, with 75.3 mIoU on Snow, 57.9 on Night, and 83.7 on Fog.Compared with FADA, gains are +1.8, +0.5, and +3.5 mIoU on Snow, Night, and Fog, respectively; Tab. III covers this adverse-weather setting.Source: B. Comparison with State-of-the-Art… · p. 7
• Across five synthetic-to-real and real-to-real settings, SIFR predictions are closer to ground truth, especially for small objects, thin structures, and ambiguous boundaries.The reported examples include more complete vehicle and roadside structures, cleaner layouts, fewer category confusions, and clearer low-light scene boundaries.Source: B. Comparison with State-of-the-Art… · p. 7

C. Ablation Analysis

• SIFR feature responses are clearer and more concentrated than MFuser’s on distant objects, thin structures, and semantic boundaries.The visual analysis also reports better-preserved contours and suppressed irrelevant activations in ambiguous areas.Source: Figure 5 · p. 8
• DS-Norm achieves the best average mIoU at 73.04, while combining min, average, and max pooling also reaches 73.04 average mIoU.The reported explanation is that low-frequency statistics suppress residual domain-sensitive deviations and the pooling operators capture complementary frequency statistics.Source: C. Ablation Analysis · p. 8
• SGIF and SDR each improve cross-domain performance over the baseline, and using both together yields further gains.SGIF strengthens fused representations with token interaction and frequency-guided invariant cues; SDR removes residual domain-sensitive bias during decoding.Source: C. Ablation Analysis · p. 8
• Average mIoU is 72.49, 72.38, 73.04, and 72.04 with Lq set to 32, 48, 64, and 128, respectively.Lq = 64 is the best reported setting; the passage suggests excessive tokens may introduce redundant cross-prior interactions.Source: C. Ablation Analysis · p. 8
• Average mIoU is 73.04, 72.75, and 72.94 for low-pass filter ratios 0.25, 0.50, and 0.75, respectively.The maximum difference is 0.29 mIoU, and ρ = 0.25 is adopted for its best overall performance.Source: C. Ablation Analysis · pp. 8–9

Global Feature Space Analysis

• On unseen BDD100K and Mapillary, SIFR produces more compact intra-class clusters and clearer inter-class separation than the baseline.The t-SNE visualization uses decoder embeddings from the final mask-feature projection before mask prediction.Source: D. Representation Analysis · p. 9, Figure 6 · p. 10
• On BDD100K, intra-class distance falls 4.4342→3.9008 and DBI 2.4203→1.7189, while inter-/intra-class ratio rises 1.5037→2.1936 and CH 1103.83→2217.43.The reported criteria indicate more compact within-class distributions and stronger between-class separability.Source: Table VIII · p. 10, D. Representation Analysis · p. 10

Failure Case Analysis

• SIFR still confuses roadside vegetation with terrain and drivable road with sidewalk, and some small or thin structure boundaries remain inaccurate.The vegetation errors involve mixed natural elements with similar appearances; road errors occur where pavement looks similar and curb boundaries are unclear.Source: D. Representation Analysis · pp. 10–11, Figure 7 · p. 10

• SIFR is a frequency-aware dual-stream framework that leverages complementary frozen CLIP and DINOv2 priors for domain generalized semantic segmentation.Source: §V · p. 11
• Spectral-Guided Invariant Fusion reduces domain-sensitive interference, while Spectral Decoder Refinement stabilizes multi-scale decoding with low-frequency statistics-based normalization and adaptive spectral enhancement.Source: §V · p. 11
• Across synthetic-to-real, real-to-real, and real-to-adverse-weather benchmarks, SIFR consistently improves cross-domain segmentation.The paper presents frequency-aware modeling as an effective way to exploit heterogeneous foundation priors for DGSS.Source: §V · p. 11