PaperlayerView exact revision on arXiv

Source-linked AI summary

Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation

Zhixiang Wei, Lin Chen, Yi Jin, Xiaoxiao Ma, Tianle Liu, Pengyang Ling, Ben Wang, Huaian Chen, Jinjin Zheng

arXiv:2312.04265v5cs.CV

TL;DR

DGSS requires models to generalize across unseen domains, while adapting large Vision Foundation Models is difficult with limited training data. The paper evaluates VFMs and introduces Rein, a parameter-efficient token-based refinement method for adapting frozen backbones. Across varied settings, Rein improves DGSS generalization and outperforms existing methods, while future work remains for real-data, adverse-weather, and specialized-road settings.

  • Problem

    Adapting large Vision Foundation Models for DGSS is challenging because common DGSS datasets are much smaller than pre-training datasets, while the effectiveness of VFMs in DGSS was previously underexplored.

  • Method

    Rein freezes the VFM backbone and uses low-rank learnable tokens linked to distinct instances to refine feature maps across backbone layers with fewer trainable parameters.

  • Results

    Rein significantly outperforms state-of-the-art methods across diverse DGSS settings and improves VFM generalization with efficient training.

  • Takeaways & Limitations

    Rein can be integrated as a plug-and-play adapter for existing VFMs, while frozen VFMs demonstrate substantial potential for DGSS.

  • Takeaways & Limitations

    Further work is needed to extend the approach to models trained on real datasets, adverse weather, special road types, and additional settings.

Abstract

from arXiv · show

In this paper, we first assess and harness various Vision Foundation Models (VFMs) in the context of Domain Generalized Semantic Segmentation (DGSS). Driven by the motivation that Leveraging Stronger pre-trained models and Fewer trainable parameters for Superior generalizability, we introduce a robust fine-tuning approach, namely Rein, to parameter-efficiently harness VFMs for DGSS. Built upon a set of trainable tokens, each linked to distinct instances, Rein precisely refines and forwards the feature maps from each layer to the next layer within the backbone. This process produces diverse refinements for different categories within a single image. With fewer trainable parameters, Rein efficiently fine-tunes VFMs for DGSS tasks, surprisingly surpassing full parameter fine-tuning. Extensive experiments across various settings demonstrate that Rein significantly outperforms state-of-the-art methods. Remarkably, with just an extra 1% of trainable parameters within the frozen backbone, Rein achieves a mIoU of 78.4% on the Cityscapes, without accessing any real urban-scene datasets.Code is available at https://github.com/w1oves/Rein.git.

1. Introduction

The paper asks how Vision Foundation Models perform in Domain Generalized Semantic Segmentation and how to harness them with fewer trainable parameters. It introduces Rein, which uses instance-linked learnable tokens to refine VFM features and reports stronger generalization across DGSS settings.

  • Motivation: DGSS aims to improve prediction accuracy on unseen domains without accessing their data, commonly using augmentation and domain-invariant feature strategies.
  • Stronger: Frozen Vision Foundation Models outperform previous DGSS methods in the reported GTAV → Cityscapes + BDD100K + Mapillary benchmark.
  • Research questions: The paper evaluates whether Vision Foundation Models can serve as stronger backbones for DGSS and how they can be adapted effectively.
  • Fewer: Rein uses fewer trainable parameters to adapt VFMs, addressing the limited scale of common DGSS datasets and the poor generalizability associated with fully fine-tuning large models.
  • Superior: Rein links learnable low-rank tokens to distinct instances, producing attention-like similarity maps for instance-level feature refinement across backbone layers.
  • Evaluation: The proposed method is evaluated against DGSS and parameter-efficient baselines across multiple VFM backbones and generalization settings.

3. Methods

Rein parameter-efficiently adapts frozen Vision Foundation Models for DGSS by refining feature maps across backbone layers with instance-linked learnable tokens. It bridges scene and task gaps while reducing parameters through low-rank tokens and shared MLP weights.

  • Core of Rein: Rein freezes the VFM backbone and trains an efficient module that generates feature modifications Δf_i at each layer.The refined feature maps are forwarded through subsequent backbone layers.
  • Core of Rein: Learnable tokens bridge pre-training and fine-tuning gaps while enabling tailored adjustments for multiple instances in one image.The method targets both scene differences and divergence between pre-training and semantic-segmentation tasks.
  • Core of Rein: Feature-to-token similarities are softmax-normalized to estimate feature changes, while dropping the first token limits unnecessary adjustments.Selecting tokens 2 through m lets each row’s similarity sum vary from 0 to 1.
  • Core of Rein: Instance-level feature modifications produce diverse refinements for different categories within a single image.The refinement is applied after combining the estimated change with the original feature map through an MLP.
  • Details of Rein: Low-rank token factorization and layer-shared MLP weights reduce Rein’s trainable parameter count and inference overhead.Tokens use matrices with rank r much smaller than channel dimension c, and shared weights replace layer-specific MLP parameters.

4. Experiments

Experiments evaluate VFMs and Rein across multiple DGSS settings, backbones, ablations, and efficiency measures. Rein outperforms existing methods while using fewer trainable parameters and reduced storage.

  • Experimental settings: Experiments cover five datasets, three generalization settings, DGSS baselines, and parameter-efficient fine-tuning approaches.The settings include synthetic-to-real and real-to-real generalization, with real-world and synthetic datasets.
  • Comparison with state-of-the-art methods: Rein surpasses previous DGSS and PEFT methods under GTAV → Citys + BDD + Map generalization with DINOv2 and EVA02 backbones.The comparison attributes the gains to the feature extraction capabilities of VFMs and reports superior performance for Rein using the same backbones.
  • Generalization and backbones: Rein generalizes across Citys → BDD + Map and additional real-world settings, while also performing strongly with ConvNeXt, ResNet, and smaller DINOv2 backbones.The experiments include Citys → ACDC, Citys → Cityscapes-C, and Citys → BDD + Map settings.
  • Ablation studies: Rein improves recognition for most of 19 categories and significantly outperforms Full and Freeze on wall, motorcycle, and bicycle.Freeze sometimes performs better than Full on road and sidewalk, consistent with reduced loss of pre-training knowledge during fine-tuning.
  • Ablation studies: m = 100 and m = 150 both reach 64.3% mIoU, leading the authors to select m = 100; r = 16 and r = 32 are optimal with DINOv2.Layer-shared MLP weights and low-rank token sequences reduce trainable parameters while positively influencing performance.
  • Efficiency: 0.01GB of Rein weights plus 0.08GB of head weights can be stored and swapped instead of all model parameters.The reported efficiency analysis considers training speed, GPU memory usage, and model storage.

5. Conclusions

The paper evaluates and harnesses VFMs for DGSS, then introduces Rein as a parameter-efficient fine-tuning approach. Across diverse settings, Rein improves generalization and outperforms state-of-the-art methods.

  • Conclusions: The paper assesses VFMs in DGSS and introduces Rein to parameter-efficiently harness them for the task.Rein is presented as a plug-and-play adapter for existing VFMs.
  • Conclusions: Rein uses fewer trainable parameters while improving VFM generalizability and outperforming state-of-the-art methods by a large margin.Extensive experiments support the reported effectiveness across DGSS settings.
  • Conclusions: Rein can be seamlessly integrated as a plug-and-play adapter for existing VFMs with efficient training.

7. Fewer Trainable Parameters

The paper motivates fewer trainable parameters as a way to address overfitting when fine-tuning large VFMs on small DGSS datasets. Experiments show that Rein reaches the strongest target-domain metrics despite not minimizing training loss.

  • Motivation: Increasing model capacity can reduce training risk while eventually increasing true risk on small fixed datasets through overfitting.
  • Motivation: Unknown target distributions make valid evaluation data unavailable in domain generalization, while DGSS fine-tuning datasets are smaller than ImageNet or LVD-142M.The paper therefore motivates fewer trainable parameters as a strategy to mitigate overfitting.
  • Empirical analysis: As trainable parameters increase from 0.00M to 2.53M to 304.24M, training loss decreases monotonically, but target-domain test metrics peak with Rein at 2.53M parameters.The Full baseline has the lowest training loss but sub-optimal test performance, which the authors identify as overfitting.

8. Value of synthetic data

Synthetic datasets support Rein’s synthetic-to-real transfer and data-efficient semantic segmentation. Training on synthetic UrbanSyn, GTAV, and Synthia yields strong Cityscapes validation performance.

  • Synthetic-to-real transfer: 78.4% mIoU is achieved on the Cityscapes validation set after training on UrbanSyn, GTAV, and Synthia.The result uses only synthetic training datasets for the reported synthetic-to-real evaluation.
  • Data-efficient training: 82.5% mIoU is reached using 1/16 of the Cityscapes training set with the resulting pre-trained weight.The paper presents this as a data-efficient semantic segmentation result.
  • Future improvement: Additional synthetic data and higher-quality diffusion-model images are identified as possible avenues for further improvement.

9. Ablation on decode head

Rein remains effective with alternative decode heads, but Mask2Former delivers stronger performance and motivates its use in the main experiments.

  • Mask2Former shares structures or core concepts with numerous dense prediction methods, extending the relevance of the findings beyond semantic segmentation.The paper specifically mentions instance and panoptic segmentation.
  • Rein surpasses the “Full” and “Freeze” baselines with SemFPN using 2.53 million trainable backbone parameters.The SemFPN decode head contains 1.63 million parameters.
  • 64.3% mIoU with DINOv2-Large using SemFPN is lower than the complete Rein configuration with Mask2Former.A second SemFPN result reports 62.1% mIoU in another setting.
  • Mask2Former produces 11.7% mIoU for ResNet101, supporting the decision to focus the main paper’s experiments on Mask2Former.

10. Ablation on EVA02

The EVA02 ablation identifies a preferred low-rank setting and reports efficiency gains over full fine-tuning, while the comparison table defines the evaluated SemFPN setting.

  • r = 16 yields the optimal results with EVA02 as the backbone.
  • Rein improves training speed and reduces GPU memory usage compared with the “Full” baseline.
  • Table 14 compares Rein with SemFPN under GTAV → Cityscapes + BDD100K + Mapillary, using GTAV for fine-tuning and the three target datasets for testing.
  • Table 15 reports results on GTAV→Citys+BDD+Map, with first-line metrics taken from Wildnet.
  • Table 16 presents Rein’s comparison with other DGSS methods under GTAV + Synthia → Cityscapes + BDD100K + Mapillary generalization.

11. Multi-source generalization.

Under multi-source training with GTAV and Synthia, Rein substantially outperforms existing DGSS methods across Cityscapes, BDD100K, and Mapillary.

  • 65.2% average mIoU is achieved by Rein, compared with 45.9% for existing DGSS methods.The evaluation uses EVA02 and DINOv2 under GTAV + Synthia → Cityscapes + BDD100K + Mapillary.

12. More details about VFMs

The paper details the architectures and pre-training sources of the VFMs used in its experiments, including CLIP, MAE, SAM, EVA02, and DINOv2.

  • CLIP uses a ViT-Large architecture with 16 × 16 patches and 1024-dimensional layer features.
  • CLIP is pre-trained through contrastive learning on publicly available image-caption data assembled from web crawling and existing image datasets.
  • DINOv2-L uses the same patch size, dimensionality, and layer count as EVA02-L, with features from layers 7, 11, 15, and 23 fed into the decode head.
  • DINOv2 is pre-trained self-supervised on the LVD-142M dataset.

13. Algorithm of Proposed Rein

Algorithm 1 describes Rein’s training procedure and specifies the dimensions of its feature maps, layer stack, iteration count, and low-rank hyperparameter.

  • 13. Algorithm of Proposed Rein: Rein is trained for T iterations while enforcing the weight constraints in Eq. (11).The algorithm uses c for feature-map channels, N for backbone layers, and r as a hyperparameter much smaller than c.
  • 13. Algorithm of Proposed Rein: The algorithm defines c as the feature-map channel count and N as the total number of model layers.
  • 13. Algorithm of Proposed Rein: The low-rank dimension r is a hyperparameter chosen considerably smaller than the channel dimension c.

14. Qualitative Results and Future works

Qualitative results show Rein generalizing across multiple urban-scene datasets despite training only on GTAV, while the paper identifies generated samples and broader adverse-condition settings as future directions.

  • Qualitative Results: Rein is evaluated on Cityscapes, BDD100K, and Mapillary after training exclusively on GTAV.The reported evaluation uses no fine-tuning on real-world urban-scene datasets.
  • Qualitative Results: Rein outshines other approaches in accuracy, particularly for traffic signs, bicycles, traffic lights, sidewalks, roads, and trucks.The passage reports high precision for both large objects and smaller targets.
  • Qualitative Results: Rein maintains surprisingly high night-time performance despite not being specifically optimized for night-time segmentation.The reported night performance is almost akin to daytime performance on the BDD100K results.
  • Future works: The paper anticipates using high-quality generated samples to approach models trained with supervision on real datasets.
  • Future works: Future work will examine VFMs for real-dataset semantic segmentation under adverse weather or on special road types, and extend Rein to additional tasks.The listed additional tasks include instance, panoptic, and open-vocabulary segmentation and object detection.
Loading 2312.04265v5…