PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentation

Inkyu Shin, Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, In So Kweon, Kuk-Jin Yoon

arXiv:2204.12667v1cs.CV

TL;DR

Multi-modal 3D semantic segmentation faces test-time domain shift, while separately adapting its modalities can be unstable. MM-TTA combines within-modality pseudo-label generation with cross-modal refinement, and reports favorable performance across several benchmark settings.

  • Problem

    Multi-modal data can shift from the training distribution at test time, and separately adapting 2D and 3D branches can increase discrepancy or destabilize predictions.

    Source: §1 · pp. 1–2

  • Method

    MM-TTA combines slow-fast within-modality pseudo-label generation with consistency-based selection of pseudo labels across 2D and 3D modalities.

    Source: §1 · p. 2

  • Results

    MM-TTA achieves favorable performance across cross-dataset sensor, synthetic-to-real, and day-to-night benchmark settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that MM-TTA provides a simple yet effective framework for producing reliable cross-modal pseudo labels across several benchmark settings.

    Source: §5 · p. 8

    Entropy minimization only encourages sharp output distributions, which may reinforce wrong predictions and may not ensure cross-modal consistency.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Test-time adaptation approaches have recently emerged as a practical solution for handling domain shift without access to the source domain data. In this paper, we propose and explore a new multi-modal extension of test-time adaptation for 3D semantic segmentation. We find that directly applying existing methods usually results in performance instability at test time because multi-modal input is not considered jointly. To design a framework that can take full advantage of multi-modality, where each modality provides regularized self-supervisory signals to other modalities, we propose two complementary modules within and across the modalities. First, Intra-modal Pseudolabel Generation (Intra-PG) is introduced to obtain reliable pseudo labels within each modality by aggregating information from two models that are both pre-trained on source data but updated with target data at different paces. Second, Inter-modal Pseudo-label Refinement (Inter-PR) adaptively selects more reliable pseudo labels from different modalities based on a proposed consistency scheme. Experiments demonstrate that our regularized pseudo labels produce stable self-learning signals in numerous multi-modal test-time adaptation scenarios for 3D semantic segmentation. Visit our project website at https://www.nec-labs.com/~mas/MM-TTA.

1. Introduction

The introduction frames multi-modal 3D semantic segmentation test-time adaptation as a response to domain shift and instability when modalities are adapted separately. MM-TTA combines within-modality pseudo-label generation with cross-modal refinement, and reports favorable performance across benchmark settings.

1. Introduction

  • Test-time adaptation uses source-pretrained model parameters and unlabeled test data for quick adaptation, typically with one epoch, while multi-modal inputs face test-time distribution shift.Unlike usual domain adaptation, this setting does not access source data during adaptation and has a limited adaptation budget.Source: §1 · p. 1
  • Separately applying entropy minimization to 2D and 3D branches can increase cross-modal discrepancy and destabilize adaptation; consistency loss may also worsen predictions when a branch is inaccurate.The introduction motivates jointly designed self-supervisory signals rather than independent branch updates.Source: §1 · p. 2
  • Intra-PG generates within-modality pseudo labels by combining slow- and fast-updated batch-normalization statistics, balancing stability with current test-data statistics.The slow model receives momentum updates from the fast model, whose statistics are updated directly from test data.Source: Figure 1 · p. 1, §1 · p. 2
  • Inter-PR uses prediction consistencies to select reliable pseudo labels from 2D and 3D branches and form cross-modal self-training signals.The two modules collaborate to produce final cross-modal pseudo labels for updating branch batch-normalization parameters.Source: §1 · p. 2
  • MM-TTA achieves favorable performance across cross-dataset sensor, synthetic-to-real, and day-to-night benchmark settings.The experiments also analyze the benefits of Intra-PG and Inter-PR and compare stability with existing methods.Source: §1 · p. 2
  • The paper studies a new test-time adaptation setting for multi-modal 3D semantic segmentation, where 2D image context and 3D point geometry provide complementary information.The task requires geometric and semantic reasoning about the scene.Source: §1 · pp. 1–2

2. Related Work

Related work covers test-time adaptation, 3D segmentation, multi-modal fusion, and unsupervised domain adaptation. MM-TTA differs by addressing multi-modal 3D segmentation at test time, when source data is unavailable for adaptation.

2. Related Work

  • Earlier TTA includes online self-supervised proxy-task training, while TENT avoids requiring a proxy task but can increase confidence in false predictions.The paper positions its multi-modal 3D setting alongside these approaches while developing modules for more reliable self-learning signals.Source: §2 · pp. 2–3
  • TENT uses entropy minimization to update test-time batch-normalization parameters, while S4T regularizes pseudo labels with aligned predictive views for 2D segmentation.S4T's spatial-augmentation design is described as specific to image-level tasks; MM-TTA instead develops intra- and inter-modal modules for multi-modal 3D segmentation.Source: §2 · pp. 2–3
  • LiDAR segmentation methods use spherical projection or raw point clouds, but LiDAR alone lacks 2D contextual information needed to understand complex scene semantics.Multi-modal methods combine RGB images and LiDAR point clouds through separate backbones and fusion techniques.Source: §2 · p. 3
  • Multi-modal segmentation benefits from contextual and geometric information, but modality-specific dataset biases make adaptation to new data harder.The paper addresses this setting through test-time adaptation that incorporates test-data statistics during inference.Source: §2 · p. 3
  • Unlike UDA methods such as xMUDA and DsCML, which can access source data during adaptation, MM-TTA adapts from a source-pretrained model with a limited update budget.xMUDA uses cross-modal consistency learning during training; DsCML adds adversarial learning and dynamic sparse-to-dense cross-modal learning.Source: §2 · p. 3

3. Proposed Method

The method adapts a source-pretrained 2D/3D segmentation model using target test data, combining within-modality pseudo-label generation with cross-modal refinement.

3.1. Preliminaries

  • The adaptation setting uses a source-pretrained multimodal model and target test data without source data, updating only batch-normalization parameters that occupy less than 1% of each branch.The model has separate 2D image and 3D point-cloud branches.Source: §3 · p. 3, §3.1 · p. 3

3.2. Baselines for MM-TTA

  • Existing entropy and consistency baselines can reinforce wrong predictions or fail to capture cross-modal agreement, while pseudo-labeling lacks cross-modal refinement.Entropy minimization encourages sharp outputs and may reinforce errors; consistency can fail when a branch predicts incorrectly.Source: §3.2 · pp. 4–5

3.3. Intra-modal Pseudo-label Generation

  • Intra-PG aggregates fast- and slow-updated models within each modality to produce pseudo-labels, with the slow model providing a stable signal and serving at inference.The fast model updates batch-normalization statistics directly from test data; the slow model moves toward those statistics through momentum.Source: §3.3 · p. 5

3.4. Inter-modal Pseudo-label Refinement

  • Inter-PR uses slow-fast consistency measures to select or weight pseudo-labels across modalities, discarding labels whose maximum consistency falls below a class threshold.Hard Select chooses a modality; Soft Select weights the two modalities using their consistency measures.Source: §3.4 · p. 5

4. Experimental Results

The experiments evaluate MM-TTA across sensor, synthetic-to-real, and day-to-night shifts, then analyze its components and stability. MM-TTA performs favorably against TTA baselines across the three benchmark settings, with ablations and pseudo-label measurements supporting the roles of its modules.

4.1. Datasets and Settings

  • The benchmarks cover sensor differences between A2D2 and SemanticKITTI, synthetic-to-real shifts from Synthia, and day-to-night changes in nuScenes.Synthia-to-SemanticKITTI includes camera-style and LiDAR point-distribution and depth-accuracy gaps; the Synthia point clouds are constructed by sampling pixels from dense depth maps.Source: §4.1 · pp. 5–6

4.2. Implementation Details

  • The evaluation uses 2D, 3D, and ensemble mIoU, compares TTA baselines with UDA references, and applies hyperparameter search to all methods.The UDA references can access source and target data during training, unlike the TTA setting.Source: §4.3 · p. 6

4.3. Main Results

  • MM-TTA performs favorably against TTA baselines in all three benchmark settings; on nuScenes Day-to-Night, its ensemble also surpasses xMUDA approaches using UDA.The 3D branch remains competitive while the 2D branch and ensemble improve significantly; example SemanticKITTI predictions become more complete and accurate than TENT’s during adaptation.Source: §4.3 · pp. 6–7

4.4.1 Analysis on MM-TTA

  • Slow-fast modeling improves Intra-PG variants, while Hard Select and Soft Select outperform the tested Consensus, Merge, and Entropy refinement alternatives.Thresholding pseudo-labels also performs better than the compared alternative; Intra-PG and Inter-PR are coupled in the final model.Source: §4.4.1 · pp. 7–8

4.4.2 Sensitivity Analysis

  • MM-TTA remains stable across learning rates, with higher mean (44.2/44.3) and lower standard deviation (2.45/2.55); θ(k)=0.3 performs best, while λ=1.0 is not optimal.The threshold filters points class-wise, and λ=1.0 retains the source batch-normalization statistics.Source: §4.4.2 · p. 8

4.4.3 Analysis on Pseudo-labeling Accuracy

  • Slow-fast modeling raises pseudo-label accuracy by 2% over the fast-model baseline, and combining both modules yields a 5% gain across adaptation iterations.Accuracy is averaged over categories for valid points collected at six phases from iterations 100 to 600.Source: §4.4.3 · p. 8

5. Conclusions

The conclusion presents MM-TTA as a new test-time adaptation setting for 3D semantic segmentation and summarizes its two-module framework and experimental analysis.

5. Conclusions

  • The paper presents MultiModal Test-Time Adaptation (MM-TTA) as a new problem setting for 3D semantic segmentation.It proposes a self-training framework with Intra-PG and Inter-PR to produce reliable cross-modal pseudo-labels.Source: §5 · p. 8
  • Experiments demonstrate the framework in several benchmark settings.The paper also reports ablation studies and analysis of the proposed modules.Source: §5 · p. 8
  • The conclusion says the paper identifies baselines and their limitations before proposing its framework.Source: §5 · p. 8

A. Dataset Construction

The appendix specifies shared dataset classes, describes construction of the Synthia synthetic-to-real data, and points to the MM-TTA algorithm.

A.1. A2D2, SemanticKITTI and nuScenes

  • A2D2 and SemanticKITTI use 10 shared classes for training and testing; the passage also names the nuScenes dataset setting.The listed classes include car, truck, bike, person, road, parking, sidewalk, building nature, and other-objects.Source: §A.1 · p. 11

A.2. Synthia

  • Synthia is reorganized from 9,000 RGB images with segmentation and depth labels by sampling about 15k pixels into point clouds and merging 23 classes into 10.The resulting categories are shared with SemanticKITTI.Source: §A.2 · p. 11

B. Algorithm for MM-TTA

C. Oracle Test

Oracle tests compare MM-TTA pseudo-label performance with adaptation using real target labels; on nuScenes Day → Night, Soft Select is comparable to Oracle TTA.

C. Oracle Test

  • On nuScenes Day → Night, MM-TTA with Soft Select obtains results comparable to Oracle TTA using real labels.Oracle TTA fine-tunes only 2D/3D batch-normalization parameters for one epoch with real target labels; Oracle Full updates all layers from scratch for 30 epochs.Source: C. Oracle Test · p. 11
  • Figure 6 depicts Synthia dataset construction for point clouds containing 15k points.Source: Figure 6 · p. 11

D. Analysis on Class-wise Adaptation

The analysis uses t-SNE to examine class-wise adaptation and reports better category-level feature alignment for MM-TTA than TENT across iterations.

D. Analysis on Class-wise Adaptation

E. Qualitative Results for Pseudo Labels

The authors report that MM-TTA’s two modules produce pseudo labels that are more refined, accurate, and similar to ground truth than other baselines.

E. Qualitative Results for Pseudo Labels

  • MM-TTA’s two modules produce pseudo labels described as more refined and accurate, and more similar to ground truth, than other baselines.The passage reports this comparison based on the qualitative visualization in Figure 8.Source: E. Qualitative Results for Pseudo Labels · p. 11

F. More Qualitative Results

Across all adaptation scenarios shown, MM-TTA’s Hard Select and Soft Select results are described as more similar to ground truth than TENT and xMUDA.

F. More Qualitative Results

  • Across all adaptation scenarios, MM-TTA (Hard Select and Soft Select) shows segmentation results more similar to ground truth than TENT and xMUDA.The comparison covers three adaptation benchmarks, as specified in Figure 9.Source: F. More Qualitative Results · p. 11

G. Limitations and Discussion

The discussion notes that MM-TTA’s effectiveness may vary across categories. The accompanying procedure describes pseudo-label generation and selection, while a table caption specifies two oracle settings.

G. Limitations and Discussion

  • MM-TTA’s effectiveness may vary across categories, motivating category-aware test-time adaptation as a future direction for improving weaker classes.The stated limitation concerns its approach of selecting or adaptively weighting modalities for general pseudo-label generation.Source: G. Limitations and Discussion · p. 11
  • Table 4 distinguishes Oracle TTA, which finetunes batch-normalization parameters, from Oracle Full, which updates all layers.The supplied caption gives the oracle settings but no result values.Source: Table 4 · p. 12

1 begin

  • The procedure fuses slow- and fast-model outputs, aggregates pseudo labels, measures model consistency, then either selects one modality or weights both.The algorithm lists Hard Select and Soft Select as the alternatives.Source: §1 · p. 12

Related papers

Loading 2204.12667v1…