PaperlayerView exact revision on arXiv

Source-linked AI summary

Mix3D: Out-of-Context Data Augmentation for 3D Scenes

Alexey Nekrasov, Jonas Schult, Or Litany, Bastian Leibe, Francis Engelmann

arXiv:2110.02210v2cs.CV

TL;DR

Large-scale 3D segmentation models can overfit contextual priors, limiting performance on unseen and rare configurations. Mix3D mixes transformed scene pairs to expose objects to novel contexts while preserving local structure, and it consistently improves segmentation across indoor and outdoor datasets, including 78.1 mIoU on ScanNet.

  • Problem

    High-capacity 3D models trained on small, low-variance datasets may overfit contextual priors, causing poor generalization to unseen long-tail examples.

  • Method

    Mix3D randomly transforms and combines pairs of training scenes, exposing objects to novel contexts while concatenating their point clouds and labels.

  • Results

    Mix3D consistently improves point-based and voxel-based models across three indoor and outdoor datasets, reaching 78.1% mIoU on the ScanNet test benchmark challenge.

  • Takeaways & Limitations

    Mix3D balances global context priors with local structure and can be incorporated into existing 3D segmentation training pipelines.

  • Takeaways & Limitations

    Mix3D generates unrealistic scenes, and applying it to approaches combining 3D scans with corresponding 2D images remains an open question.

Abstract

from arXiv · show

We present Mix3D, a data augmentation technique for segmenting large-scale 3D scenes. Since scene context helps reasoning about object semantics, current works focus on models with large capacity and receptive fields that can fully capture the global context of an input 3D scene. However, strong contextual priors can have detrimental implications like mistaking a pedestrian crossing the street for a car. In this work, we focus on the importance of balancing global scene context and local geometry, with the goal of generalizing beyond the contextual priors in the training set. In particular, we propose a "mixing" technique which creates new training samples by combining two augmented scenes. By doing so, object instances are implicitly placed into novel out-of-context environments and therefore making it harder for models to rely on scene context alone, and instead infer semantics from local structure as well. We perform detailed analysis to understand the importance of global context, local structures and the effect of mixing scenes. In experiments, we show that models trained with Mix3D profit from a significant performance boost on indoor (ScanNet, S3DIS) and outdoor datasets (SemanticKITTI). Mix3D can be trivially used with any existing method, e.g., trained with Mix3D, MinkowskiNet outperforms all prior state-of-the-art methods by a significant margin on the ScanNet test benchmark 78.1 mIoU. Code is available at: https://nekrasov.dev/mix3d/

1. Introduction

3D semantic segmentation models benefit from global context but can overfit contextual priors, harming generalization to rare or unusual scenes. Mix3D addresses this by mixing training scenes, improving segmentation across indoor and outdoor benchmarks.

  • Motivation: Global context supports semantic reasoning, but high-capacity 3D models can overfit contextual cues in small, low-variance datasets.Full-scene models capture global context, while limited 3D data variance creates large generalization gaps.
  • Motivation: Overreliance on training-scene context can cause failures on unseen long-tail events, including pedestrians walking in the road.Such out-of-context generalization is especially important in safety-critical autonomous driving settings.
  • Approach: Mix3D mixes transformed pairs of training scenes to expose objects to novel contexts and reduce overfitting to contextual priors.Unlike single-sample augmentation, the method changes scene context while preserving corresponding point and label structure.
  • Contribution: Mix3D balances global context and local geometry, with analyses examining their roles in model generalization and segmentation performance.The method is presented as a data augmentation technique for large-scale 3D scenes and high-capacity models.
  • Results: Mix3D consistently improves point-based and voxel-based models across three large-scale indoor and outdoor datasets.The evaluated benchmarks are ScanNet, S3DIS, and SemanticKITTI.
  • Results: 78.1% mIoU establishes new state-of-the-art semantic segmentation performance on the ScanNet test benchmark challenge.The introduction reports this as a headline result of the work.

2. Related Work

Prior work studies mixed-sample augmentation, context manipulation, and expanded spatial context in 3D models. Mix3D combines scene mixing with explicit preservation of full scene structure to create novel contextual arrangements.

  • Mixed Sample Data Augmentation: MixUp creates virtual samples by interpolating paired inputs and labels, encouraging smoother behavior outside the empirical training distribution.Several extensions modify how samples or labels are mixed.
  • Context Augmentations: Context-augmentation methods reduce contextual dependence by removing context or objects, or by placing new instances at plausible image locations.Prior studies report that semantically plausible placement can improve 2D object detection, whereas arbitrary placement can reduce performance.
  • Context Aggregation of Deep Learning Models: 3D segmentation methods differ in spatial context: chunk-based point methods restrict global information, while sparse voxel models process full scenes.Increasing spatial context has been associated with improvements in subsequent approaches.
  • Mix3D: Mix3D illustrates scene mixing by creating novel object arrangements, such as a classroom table facing an apartment door.The mixed scenes expose objects to the combined context of both source scenes.

3. Method

Mix3D creates out-of-context training examples by transforming and combining pairs of 3D scenes. The unchanged segmentation model learns from mixed inputs while retaining each scene’s labels and full contextual structure.

  • Goal: Mix3D aims to reduce misleading context bias by training models to balance local structures with global scene context.Its stated goal is generalization beyond the contextual priors of training scenes.
  • Mixing: Pairwise mixing exposes each object to combined contexts and yields O(N^2) novel contexts from N original scene contexts.The network encounters object arrangements that would not normally appear in individual training scenes.
  • Data Augmentation: Scenes are augmented through centering, random flips, rotations, translations, subsampling, elastic distortion, scaling, and optional color changes.These transformations are applied before the two scenes are mixed.
  • Mixing: The mixed scene is formed by taking the union of two augmented point clouds, implemented as pairwise batch concatenation.Because point order is preserved, ground-truth labels are concatenated in the same order.
  • Training Pipeline: Mix3D leaves the underlying 3D model unchanged and supervises semantic segmentation with standard cross-entropy loss on mixed predictions and concatenated labels.The approach is designed to integrate into existing training pipelines and was tested with MinkowskiNet and KPConv.
  • Design Rationale: Unlike interpolation-based methods, Mix3D preserves complete context for each mixed sample rather than distorting points or restricting structure to local chunks.This distinction is presented as a key difference from PointMixUp and RSMix.

4. Experiments

Experiments evaluate Mix3D across indoor and outdoor 3D segmentation benchmarks and analyze how context, local geometry, overfitting, and scene overlap affect performance. Mix3D consistently improves models while reducing reliance on training-scene context.

  • Comparison with State-of-the-art Methods: Mix3D improves MinkowskiNet and KPConv across ScanNet, S3DIS, and SemanticKITTI, with gains obtained without changing training hyperparameters.The reported comparisons use the same training time, effective batch size, and other hyperparameters.
  • Comparison with State-of-the-art Methods: 78.1 mIoU establishes a state-of-the-art ScanNet test score for a MinkowskiNet ensemble trained with Mix3D.The ensemble is trained for 120k iterations on the train and validation splits.
  • Motivation and Analysis Experiments: Increasing context generally helps segmentation, but performance plateaus or declines when models overfit to scene context.MinkowskiNet trained on quarter-scene fractions performs on par with models trained on full scenes, and KPConv gains stop increasing after a point.
  • Motivation and Analysis Experiments: Mix3D improves isolated-instance segmentation by +11.5 mIoU, indicating greater use of local geometry when global context is absent.Both compared models are trained on full scenes, while evaluation uses cropped validation instances.
  • Motivation and Analysis Experiments: The largest scene-overlap gain is +2.4 mIoU, showing that physical context overlap is decisive beyond merely concatenating distant scenes.Placing scenes nearby without physical overlap improves performance by at least +1.2 mIoU, whereas distant concatenation shows no significant difference from baseline.
  • Motivation and Analysis Experiments: Mix3D reduces overfitting by narrowing the training-validation loss gap and delaying the rise of validation loss.Learning-curve analysis evaluates MinkowskiNet with and without Mix3D on ScanNet validation.

5. Discussion and Conclusion

Mix3D is presented as a simple augmentation for large-scale 3D scene segmentation that helps models balance global context priors with local structure. The method improves segmentation across indoor and outdoor datasets, while its realism, task coverage, and multimodal extension remain open questions.

  • Mix3D balances global context priors and local structure in large-scale 3D scene segmentation.
  • Mix3D significantly boosts semantic segmentation performance for multiple state-of-the-art 3D models on indoor and outdoor datasets.
  • Mix3D is easy to implement and can be incorporated directly into existing training pipelines.
  • Applying Mix3D to approaches combining 3D scans with corresponding 2D images remains an open research question.
  • Mix3D generates unrealistic scenes, and increased realism could improve performance; object detection and instance segmentation are also open applications.

Abstract

The supplementary material expands the paper's implementation details and analysis. It also reports per-class semantic segmentation scores across the three evaluated datasets.

  • The supplement details network architectures and training and inference procedures.
  • Additional analyses, experiments, and results are provided in Section 4.2.
  • Per-class semantic segmentation scores are reported for ScanNet, S3DIS, and SemanticKITTI.

A. Details on Architectures and Training

The appendix specifies the evaluated MinkowskiNet and KPConv implementations, training protocols, voxel settings, batch-size adjustments, and ScanNet benchmark inference ensemble.

  • MinkowskiNet: MinkowskiNet uses official model definitions, with 2 cm voxels for ScanNet benchmark comparisons and 5 cm voxels for faster analysis experiments.
  • KPConv: KPConv experiments use official code releases or the original authors' PyTorch reimplementation, following the same evaluation procedure.
  • Training details: Mix3D batch sizes are reduced or adjusted to keep the total points per training iteration comparable with baseline methods.
  • Inference settings: The ScanNet benchmark submission uses a four-model MinkowskiNet ensemble trained with Mix3D.

B. Additional Analysis

Additional analyses examine mixing quantity, architecture details, batch composition, dataset size, and alternatives to complete-scene mixing. They identify two-scene mixing and physical context overlap as important design choices, with larger benefits in scarce-data settings.

  • Influence of the number of mixed scenes: Validation performance peaks when two scenes are mixed and decreases noticeably when more scenes are combined.
  • Influence of non-mixed spheres in a batch: +1.0 mIoU is achieved with two or four non-mixed batch entries, compared with +0.4 mIoU when training only on mixed scenes.
  • Consistent improvements wrt. dataset size: +[2.0, 2.8] mIoU absolute gains occur across all tested ScanNet train-set sizes, with larger relative improvements on smaller datasets.
  • Influence of the number of mixed scenes: Mixing eight point clouds makes individual objects hardly visually distinguishable, unlike mixing two point clouds.
  • Comparison to single-instance mixing: Single-instance mixing matches Mix3D's context-bias motivation but requires instance databases, instance annotations or synthetic data, and placement procedures.

C. Per-Class Quantitative Evaluation

Per-class evaluations compare semantic segmentation with and without Mix3D across ScanNet, S3DIS, and SemanticKITTI, including isolated ScanNet instances without scene context. On isolated instances, Mix3D improves performance on 18 of 20 semantic classes.

  • ScanNet: Per-class semantic segmentation IoU is reported on the ScanNet validation set for models trained with and without Mix3D.Table 12 reports mean validation performance over 3 trained models.
  • S3DIS and SemanticKITTI: Per-class semantic segmentation IoU is reported for S3DIS Area 5 and the SemanticKITTI test set.The evaluations cover both indoor and outdoor datasets.
  • Isolated instances: 18 of 20 semantic classes show improved IoU with Mix3D on isolated ScanNet validation instances without scene context.The comparison uses MinkowskiNet with 5 cm voxels and means from 3 trained models.
Loading 2110.02210v2…