PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Performance Discrepancy in Cross-Domain 3D Class-Incremental Learning

Jinge Ma, Gautham Vinod, Bruce Coburn, Jui-Feng Chi, Siddeshwar Raghavan, Fengqing Zhu

arXiv:2609.04860v1cs.CVcs.AI

TL;DR

Heterogeneous 3D point clouds create domain-dependent performance degradation in class-incremental learning. The paper establishes Domain3D-CIL and proposes PolyMem, which captures high-order feature-distribution interactions; experiments show improved performance across domains and a reduced cross-domain gap.

  • Problem

    Heterogeneous 3D point-cloud sources create a performance discrepancy in class-incremental learning, with degradation varying substantially across domains.

  • Method

    The paper establishes Domain3D-CIL and proposes PolyMem, an exemplar-free framework that models high-order interactions in heterogeneous 3D feature distributions.

  • Results

    Performance discrepancy is prevalent across CIL baselines, while PolyMem improves performance on both domains and reduces the cross-domain performance gap.

  • Takeaways & Limitations

    Cross-domain robustness in 3D class-incremental learning requires evaluating both average performance and domain-wise performance discrepancy.

  • Takeaways & Limitations

    The paper does not analyze all causes of performance discrepancy across all baselines.

Abstract

from arXiv · show

3D perception plays a crucial role in real-world applications such as autonomous driving, robotics, and AR/VR. In practical scenarios, 3D perception models need to continually adapt to newly emerging 3D object categories, making class-incremental learning (CIL) particularly important. However, unlike 2D images, 3D point clouds are inherently heterogeneous: objects from the same class may not only come from the clean CAD domain, but also from RGB-D camera scans of varying quality, video reconstructions, or even corrupted observations. We discover that such heterogeneity introduces a new challenge beyond catastrophic forgetting: the degree of performance degradation can vary substantially across domains, a phenomenon we term performance discrepancy. To investigate this problem, we establish the Domain3D-CIL training and evaluation protocol, which contains point cloud categories from heterogeneous domains. We further adapt a wide range of mainstream CIL methods to the 3D modality. The results demonstrate that this performance discrepancy consistently appears across these baselines. To mitigate this issue, we introduce PolyMem, an exemplar-free approach that implicitly models rich high-order statistics of the feature distribution to enhance cross-domain robustness. Experiments demonstrate that our method effectively alleviates the performance discrepancy while improving the model's performance across domains. Code will be made publicly available upon acceptance.

1 Introduction

Heterogeneous 3D data creates performance discrepancy in class-incremental learning, with domain-specific degradation emerging alongside catastrophic forgetting. The paper formalizes this challenge through Domain3D-CIL and proposes PolyMem to mitigate cross-domain discrepancy and within-domain forgetting.

  • Problem: 3D point clouds combine CAD, RGB-D scans, video reconstructions, and corrupted observations, creating substantial intra-class heterogeneity for CIL.Samples from heterogeneous domains can share semantic labels while differing substantially in quality and origin.
  • Problem: As tasks progress, models suffer both catastrophic forgetting and uneven performance degradation across heterogeneous domains.The imbalance in memory retention across domains contributes to the emerging performance discrepancy.
  • Protocol: Domain3D-CIL formalizes this setting by mixing clean CAD, real-world, and corrupted point clouds within shared semantic classes.The protocol is designed to quantify and analyze cross-domain challenges in 3D CIL.
  • Empirical findings: Experiments show that performance discrepancy is prevalent among existing 3D-adapted CIL baselines.The paper transfers widely used CIL baselines to 3D and analyzes mechanisms behind their cross-domain performance differences.
  • Method: PolyMem mitigates cross-domain performance discrepancy while also reducing within-domain catastrophic forgetting.The proposed framework is presented as the paper’s solution to the identified cross-domain robustness problem.

2 Related Works

Prior work advances 3D representation learning and class-incremental learning through improved architectures and continual-learning strategies. However, cross-domain performance discrepancy within the same semantic class remains underexplored in heterogeneous 3D data.

  • 3D Representation Learning and Data Heterogeneity: PointNet and PointNet++ established shared-MLP point-set processing, while Transformer architectures improved 3D representation learning through global context modeling.These works mark the progression from early point-set frameworks to more effective global-context architectures.
  • 3D Representation Learning and Data Heterogeneity: Unified 3D representation learning increasingly addresses heterogeneous sources, including CAD models, RGB-D scans, LiDAR acquisitions, and multi-view reconstructions.Different construction and acquisition processes introduce substantially different domain properties.
  • Class-incremental Learning: Class-incremental learning sequentially introduces new classes without full historical data, using exemplar replay, parameter regularization, or architectural expansion to reduce inter-task interference.The objective is to learn new classes while retaining previously acquired knowledge.
  • Class-incremental Learning: Existing 3D incremental-learning studies mainly target few-shot settings, while prior domain-incremental approaches emphasize shifts across tasks rather than within-class cross-domain performance discrepancy.The work therefore analyzes cross-domain performance discrepancy under the stated 3D class-incremental setting.

3 Preliminaries

This section formalizes replay-free class-incremental learning and introduces Domain3D-CIL, which mixes CAD data with one heterogeneous non-CAD domain per task. It evaluates performance separately across domains while retaining shared semantic labels and exposes realistic quality and long-tail distribution shifts.

  • CIL formulation: Class-incremental learning uses disjoint label sets across tasks, evaluates all observed classes, and suffers catastrophic forgetting because previous-task datasets are unavailable.The replay-free setting trains each stage only on current-task data.
  • Domain3D-CIL protocol: Domain3D-CIL constructs each task by mixing a canonical CAD domain with one additional domain: RGB-D scans, video reconstructions, or corrupted CAD point clouds.The non-CAD subset contains only categories semantically matched to the task’s CAD labels.
  • Domain3D-CIL protocol: Training provides semantic labels without domain labels, whereas evaluation reports performance separately for each domain over all observed classes.This design avoids requiring discrete domain boundaries, which can be ambiguous in real-world 3D data and often unavailable in web-collected data.
  • Experimental datasets: Domain3D-CIL experiments combine ModelNet or ShapeNet with ScanObjectNN, CO3D, OmniObject3D, or ShapeNet-C, yielding four mixed datasets.The datasets are ModelNet + ScanObjectNN, ModelNet + CO3D, ShapeNet + OmniObject3D, and ShapeNet + ShapeNet-C.
  • Experimental datasets: Non-CAD data varies from clear high-quality geometry to scans with missing regions, redundant points, and perturbations, while all mixed datasets exhibit clear long-tail class distributions.OmniObject3D has a much smaller sample scale than ShapeNet because high-quality RGB-D point clouds are scarcer than CAD models.

4 Methodology

The methodology attributes cross-domain performance discrepancy to missing higher-order feature statistics, showing that real point clouds differ from CAD primarily beyond first-order distributions. PolyMem addresses this by using a structured, finite-order polynomial feature mapping with compact TensorSketch approximations and accumulated statistics.

  • Statistical-memory analysis: SimpleCIL preserves only class means, so its first-order statistical memory cannot capture higher-order distributional diversity.The method uses prototypes as classifier weights or nearest-mean decisions.
  • Statistical-memory analysis: CAD and real domains have similar first-order statistics but clear differences in second-order statistics across semantically matched classes.The comparison averages per-dimension statistics over 10 semantically matched classes.
  • Statistical-memory analysis: Missing second-order statistics leaves real point clouds more vulnerable to interference from new-task prototypes, contributing to performance discrepancy.RanPAC alleviates this failure mode by preserving first- and second-order statistics in a lifted feature space.
  • Motivation for PolyMem: RanPAC improves discrepancy by introducing second-order memory, but its random nonlinear lifting lacks a structured, order-wise interpretation.Its expanded feature space implicitly introduces nonlinear feature interactions.
  • PolyMem design: PolyMem makes interaction orders explicit while modeling higher-order statistics implicitly through Q and G computed over polynomially lifted features.This retains memory-efficient accumulated statistics without explicitly storing higher-order tensors.
  • PolyMem design: PolyMem truncates the RBF expansion at order L and uses TensorSketch to compactly approximate each polynomial interaction within fixed dimension M.The resulting lifted features replace RanPAC’s random mapping while preserving the accumulated-statistics ridge-regression procedure.

5 Experiments

Experiments evaluate PTM-based CIL baselines and PolyMem on the heterogeneous Domain3D-CIL protocol using cross-domain accuracy gaps. Results show that CIL disproportionately harms the real or corrupted domain, while higher-order feature interactions improve cross-domain balance.

  • Experimental setup: The evaluation compares ten PTM-based CIL baselines and PolyMem using the same backbone on six public 3D datasets spanning CAD and real or corrupted domains.The domains are denoted D1 and D2, respectively.
  • Evaluation protocol: Performance is measured by Last Accuracy and Cumulative Accuracy on D1, D2, and their non-absolute gap, with smaller gaps indicating better cross-domain balance.All reported results are averaged over three runs.
  • Performance discrepancy: After introducing CIL, performance drops more severely on D2 than D1 relative to joint training, producing larger domain gaps and confirming performance discrepancy.Joint training provides an upper bound without catastrophic forgetting, while its static D1–D2 gap is only around 2–3%.
  • PolyMem analysis: Increasing PolyMem’s feature-interaction order L from 1 to 4 clearly improves performance on both domains and reduces the Last Accuracy gap.The default hyperparameters are L = 4 and M = 10000, with M matched to RanPAC for fair comparison.

6 Conclusion.

The paper formalizes performance discrepancy in cross-domain 3D class-incremental learning, establishes Domain3D-CIL, and proposes PolyMem to reduce discrepancy while improving performance across domains.

  • The study formalizes performance discrepancy in cross-domain 3D CIL and introduces Domain3D-CIL for systematic evaluation on heterogeneous point clouds.
  • Performance discrepancy is prevalent across the evaluated CIL baselines.
  • PolyMem implicitly captures high-order interactions in heterogeneous 3D feature distributions to address performance discrepancy.
  • PolyMem improves performance on both domains while effectively reducing the discrepancy.

7 Appendix · 7.1 Theoretical Background

The appendix reviews standard distributional statistics, polynomial interactions, kernel approximation, and ridge regression underlying PolyMem. It explains how compact mapped features and additive statistics enable exemplar-free class-incremental learning without storing prior samples.

  • 7.1 Theoretical Background: Theoretical Background: The appendix presents standard concepts rather than methodological contributions, covering distributional statistics, polynomial feature interactions, kernel approximation, and closed-form ridge regression.These topics provide the theoretical basis for the PolyMem construction.
  • 7.1.1 Distributional Statistics: Distributional Statistics: Moments summarize distribution shape through expectations of powers, with mean and variance describing central location and dispersion.Higher-order moments capture additional shape properties such as asymmetry and tail behavior.
  • 7.1.2 Polynomial Feature Interactions: Polynomial Feature Interactions: Aggregated polynomial products form raw moments and cross-moments, while centered products yield statistics such as variance and skewness.Single-sample products become statistics only after expectation or empirical aggregation.
  • 7.1.2 Polynomial Feature Interactions: Polynomial Feature Interactions: For D = 1024, full ordered second-, third-, and fourth-order interaction tensors require approximately 4 MiB, 4 GiB, and 4 TiB, respectively.The storage cost increases further when tensors are maintained separately for multiple classes.
  • 7.1.3 RBF Kernel: RBF Kernel: The RBF kernel decomposes into polynomial interactions of all degrees, and PolyMem retains terms only up to finite order L.The order L controls the maximum polynomial degree in the mapped feature space.
  • 7.1.4 Approximation by TensorSketch: Approximation by TensorSketch: TensorSketch approximates explicit degree-n polynomial maps while avoiding their O(D^n) expansion, allowing PolyMem to retain order-wise interactions in fixed-dimensional features.PolyMem applies TensorSketch separately to each polynomial order and concatenates the resulting blocks.
  • 7.1.5 Ridge Regression: Ridge Regression: PolyMem learns a linear classifier in the mapped space by minimizing regularized squared loss, with λ improving numerical stability and controlling weight magnitude.The resulting classifier has the closed-form solution W = (G + λI)−1Q.
  • 7.1.5 Ridge Regression: Ridge Regression: Because G and Q are additive over samples, PolyMem updates them with each task and avoids storing previous-task samples, provided the feature mapping remains fixed.Memory depends on G ∈R^M×M and Q ∈R^M×C rather than the number of training samples.

7.2 Algorithmic Description of PolyMem

PolyMem uses a fixed polynomial feature mapping throughout incremental learning, accumulates task-wise statistics from current-task samples, and estimates the classifier in closed form. This preserves a shared lifted feature space across tasks while updating Q and G incrementally.

  • Task-wise Statistic Accumulation: At each task, PolyMem uses only current-task training samples to update the statistics Q and G.The procedure is designed for the PTM-based CIL setting and performs task-wise statistic accumulation.
  • Closed-form Classifier Estimation: The classifier is obtained through a closed-form ridge regression solution after the mapped features and statistics are computed.For an input sample, the backbone feature F_i is transformed into the polynomial feature F′_i = ϕPoly(F_i).
  • Fixed Feature Mapping: PolyMem initializes ϕPoly(·) once at the first task and keeps it fixed throughout incremental learning.The fixed mapping represents accumulated statistics from all tasks in the same polynomially lifted feature space.

7.3 Dataset Details

The dataset protocol combines CAD, real-scanned, reconstructed, and corrupted point clouds through semantic alignment, standardized preprocessing, and controlled clean/corrupted splits. Four mixed benchmarks support domain-wise evaluation under explicit dataset construction and licensing constraints.

  • Preprocessing and domain labels: All point clouds are XYZ-only, resampled to N=1024 points, centered by their centroid, and isotropically scaled into [−1, 1]3 using unit-cube normalization.Domain tags record source or corruption type for bookkeeping and domain-stratified evaluation but are unavailable to the model during training.
  • Source datasets: The source suite spans synthetic CAD objects, RGB-D indoor scans, crowdsourced-video reconstructions, high-quality real scans, and corrupted ShapeNet counterparts.The datasets include ModelNet40, ShapeNetCore(v2), ScanObjectNN, CO3D, OmniObject3D, and ShapeNet-C.
  • Benchmark sizes: The mixed benchmarks contain train/test totals of 11,844/2,971, 18,663/4,673, 36,231/10,405, and 35,708/20,522, respectively.The first three combine CAD samples with semantically matched real scans or reconstructions; ShapeNet + ShapeNet-C uses equal clean and corrupted training branches.
  • Clean/corrupted split protocol: For ShapeNet + ShapeNet-C, each training object appears in exactly one clean or corrupted branch, while every test object appears once in each branch.Training uses disjoint equal-size clean and corrupted object subsets; evaluation concatenates the full clean test split with a corresponding corrupted subset.
  • Data availability and licensing: The paper does not redistribute official datasets or derived files and provides benchmark-construction information for users who obtain the source datasets under their applicable licenses.Users are responsible for obtaining the source datasets from their official providers.

7.4 PTM-based CIL Experiment Details

This subsection clarifies the evaluation protocol for PTM-based CIL, including variability reporting, cross-domain gap definitions, complementary forgetting curves, and aggregate comparisons. PolyMem achieves strong overall performance while reducing cross-domain discrepancy.

  • Experiment reporting: Results report mean ± standard deviation over the repeated runs used in Tab. 1, split across four mixed benchmarks.The benchmarks are ModelNet + ScanObjectNN, ModelNet + CO3D, ShapeNet + OmniObject3D, and ShapeNet + ShapeNet-C.
  • Cross-domain gap definitions: The raw Last Accuracy gap is the performance difference between the CAD domain and its corresponding real or corrupted domain after incremental learning.Joint-training results show a relatively small static domain gap compared with most CIL-induced gaps.
  • Cross-domain gap definitions: The normalized LA Gap measures how much more the real or corrupted domain degrades than the CAD domain when moving from joint training to CIL.Smaller normalized gaps indicate less additional cross-domain imbalance beyond the joint-training reference; cumulative-accuracy gaps remain raw.
  • Forgetting analysis: Complementary task-wise forgetting curves are provided for ModelNet + CO3D and ShapeNet + ShapeNet-C, supplementing the two benchmarks shown in the main paper.The additional curves appear in Fig. 8a and Fig. 8b.
  • Aggregate comparison: PolyMem achieves a favorable aggregate trade-off across four mixed datasets, maintaining high cross-domain average Last Accuracy while reducing average LA Gap.Fig. 9 aggregates final-task results across both domains and all mixed datasets; ideal performance combines high LA with small GapLA.

7.5 Additional Task-Split Experiments

Additional task-split experiments show that PolyMem’s cross-domain robustness persists beyond the default partition. Across both fewer-task and many-task settings, it maintains higher cross-domain accuracy and reduces performance discrepancy between domains.

  • Quantitative results: PolyMem reduces the cross-domain gap and performs strongly on both domains across the two additional task splits.The experiments compare ModelNet + CO3D with B20Inc4 and ShapeNet + ShapeNet-C with B7Inc2 against PTM-based CIL baselines.
  • Task splits: The additional settings use B20Inc4 with 6 tasks for ModelNet + CO3D and B7Inc2 with 25 tasks for ShapeNet + ShapeNet-C.These task partitions test PolyMem under fewer-task and many-task incremental designs.
  • Incremental performance: PolyMem maintains higher cross-domain average accuracy throughout the incremental process, especially in later tasks where catastrophic forgetting is more severe.Figure 10 averages the CAD-domain and reconstructed or corrupted-domain accuracies at each incremental task.
  • Robustness across partitions: PolyMem’s advantage remains stable under both fewer-task and many-task incremental settings rather than depending on the default task split.The reported curves cover ModelNet + CO3D with B20Inc4 and ShapeNet + ShapeNet-C with B7Inc2.

7.6 Parameter Analysis

The parameter analysis excludes the shared frozen backbone and separates trainable method-specific parameters from stored auxiliary parameters. PolyMem matches RanPAC in additional learnable parameters, while its auxiliary cost is dominated by statistical memory rather than TensorSketch descriptors.

  • Accounting protocol: The analysis excludes the shared frozen backbone and reports method-specific learnable and auxiliary parameter costs under ShapeNet + ShapeNet-C B7Inc2.Learnable parameters include prompts, adapters, and classifier-related parameters; auxiliary parameters are stored non-backbone quantities used in computation.
  • PolyMem overhead: 100.571M is PolyMem’s total auxiliary parameter cost, combining 100.550M with 0.021M.The values are reported in M, where 1M = 10^6 scalar entries.
  • Comparison with RanPAC: PolyMem introduces the same number of additional learnable parameters as RanPAC because both use a frozen backbone and a closed-form classifier in lifted feature space.Their main auxiliary costs instead come from accumulated statistical memory, with PolyMem’s TensorSketch descriptors contributing only 0.021M scalar entries.

7.7 Experiment with Different Backbones … 7.10 Baseline Implementation Details

The supplementary experiments show that PolyMem generalizes across Point-BERT and Point-MAE, while cross-domain performance discrepancy persists in both PTM-based and classic CIL settings. The section also specifies compute resources and baseline implementation protocols.

  • 7.7 Experiment with Different Backbones: PolyMem is evaluated with Point-BERT and Point-MAE on ModelNet + ScanObjectNN and ModelNet + CO3D to test backbone generality.The comparison includes Joint-Training, SimpleCIL, RanPAC, and PolyMem.
  • 7.7 Experiment with Different Backbones: Across both alternative backbones, D2 accuracy is generally lower than D1 accuracy, especially after class-incremental learning.The table defines D1 as CAD and D2 as real-scanned or reconstructed data.
  • 7.7 Experiment with Different Backbones: PolyMem consistently outperforms RanPAC on Point-BERT and Point-MAE, improving non-CAD accuracy and reducing the cross-domain gap.This supports that PolyMem is not tied to the Uni3D-Base backbone.
  • 7.8 Compute Resources: Experiments use compute nodes with 8× NVIDIA A40 GPUs, each providing 46,068 MiB of memory, while each random-seed run uses only one GPU.Multiple GPUs are used only to run different experiments in parallel.
  • 7.9 Classic CIL Paradigm: Classic CIL baselines replace the 2D backbone with PointNet++, train from scratch, and use a replay memory budget of 10 samples per class.These methods are evaluated under a replay-based protocol on point clouds.
  • 7.9 Classic CIL Paradigm: Classic CIL produces more severe discrepancy: most baselines exhibit Last Acc. and Cum. Acc. gaps above 20%.The D2 accuracy is usually much lower than D1, with a substantially larger CIL-induced drop on D2.
  • 7.9 Classic CIL Paradigm: The discrepancy also appears in classic CIL, indicating that it is not specific to PTM-based methods when heterogeneous domains share semantic classes.Classic CIL without pre-training suffers more severe catastrophic forgetting and performance discrepancy.
  • 7.10 Baseline Implementation Details: PTM-based baselines adapt the official LAMDA-PILOT codebase by replacing its 2D encoder with a frozen Uni3D point-cloud encoder and modifying data handling and evaluation.Uni3D’s compatible vision-transformer architecture makes most adaptations minimal.
Loading 2609.04860v1…