PaperlayerView exact revision on arXiv

Source-linked AI summary

Recovering Black-Box Point Cloud Models from Pose Shifts via Derivative-Free Optimization

Seonghwan Park, Min-Gyu Park

arXiv:2610.34850v1cs.CVcs.AIcs.LG

TL;DR

Rotation shifts can degrade frozen point-cloud models, while common recovery approaches require access unavailable through score-only interfaces. CANDO searches input rotations using model confidence and forward queries; across tasks, scales, and architectures it recovers part of the lost performance, including substantial fractions of the oracle gap in segmentation.

  • Problem

    Frozen point-cloud models can degrade under test-time rotations outside their training orientations, while existing approaches may require training, gradients, or mutable model components.

    Source: §1 · p. 1

  • Method

    CANDO uses derivative-free optimization to search input rotations that maximize model confidence, with an identity prior, using only forward queries on a frozen model.

    Source: §1 · p. 1, §3.3 · p. 3

  • Results

    Across tasks, scales, and architectures, CANDO recovers lost performance; for PointNet++ part segmentation it recovers 76% of the oracle gap under azimuth rotations and 51% under arbitrary SO(3) rotations.

    Source: §4.2 · p. 5, §6 · p. 8

  • Takeaways & Limitations

    Recovery depends on an augmentation gap, confidence-accuracy alignment, and global search over a multi-modal objective.

    Source: §1 · p. 2

    Confidence-only recovery can be unreliable for non-pose corruptions that increase confidence without improving correctness, motivating task-specific constraints or external priors in such cases.

    Source: §5.3 · p. 7

Abstract

from arXiv · show

Point cloud models are increasingly deployed through score-only black-box interfaces. Under this access model, test-time rotations outside the training distribution can substantially degrade performance. Although existing mitigations such as equivariant retraining and gradient-based adaptation are effective, they are infeasible without access to model weights or gradients. To address this limitation, we propose CanDO, a test-time canonicalization method that recovers lost performance using only forward queries. Specifically, CanDO searches for an input transformation that maximizes model confidence and uses derivative-free optimization to perform this search without gradients. We further identify three conditions for successful recovery: a training augmentation gap, an accuracy-aligned confidence surrogate, and global search over a multi-modal objective. Across multiple architectures and benchmarks, CanDO recovers 76% of the oracle gap under azimuthal misorientation and 51% under arbitrary SO(3) rotations for object part segmentation. These results show that frozen score-only point cloud models can recover from rotation-induced distribution shifts through post-deployment canonicalization alone.

1. Introduction

Frozen score-only point-cloud models can suffer substantial prediction degradation when test-time rotations differ from their training orientations. CANDO searches candidate input rotations using model confidence and forward queries, with recovery depending on an augmentation gap, confidence–accuracy alignment, and global search.

1. Introduction

  • Rotation-induced orientation mismatch can substantially degrade predictions in standard point-cloud backbones without built-in rotation invariance or equivariance.Rotations preserve semantic content while changing the coordinate representation.Source: §1 · p. 1
  • Prior approaches rely on pre-deployment training, gradients or mutable statistics, or large external models, which are unavailable for frozen score-only adaptation.This motivates asking whether recovery is possible using only forward queries.Source: §1 · p. 1
  • CANDO uses derivative-free optimization to select confidence-maximizing input rotations for frozen score-only models, recovering performance without labels or gradients.The paper also reports that CANDO outperforms prior forward-only methods across multiple tasks, scales, and architectures.Source: §1 · pp. 1–2
  • Recovery depends on an augmentation gap, confidence aligned with accuracy, and global search; these factors also clarify when task-specific constraints or external priors are needed.Rotation search uses a compact three-degree-of-freedom space rather than point coordinates or model parameters.Source: §1 · pp. 1–2

2. Related Work

Related work addresses point-cloud robustness through architecture, canonicalization, test-time adaptation, and derivative-free optimization. The paper distinguishes CANDO by applying score-only optimization to post-deployment canonicalization over low-dimensional rotations.

2. Related Work

  • Point-cloud models span point-based, graph, convolutional, transformer, and rotation-equivariant architectures, yet remain vulnerable to corruptions and distribution shifts [7] [8] [30] [34] [35] [36] [38].Rotations preserve semantic labels while changing the coordinate frame, exposing orientation bias [7] [10].Source: §2 · p. 2
  • Rotation-invariant and equivariant architectures encode robustness, while canonicalization uses heuristics or trained modules; these approaches require design changes, retraining, or an additional component [7] [10] [20] [41] [54].Heuristic canonicalization can face sign, symmetry, and degeneracy ambiguities.Source: §2 · p. 2
  • Test-time adaptation updates parameters, statistics, or auxiliary modules, while input-space methods may rely on external diffusion or foundation-model priors [24] [27] [28] [44] [46].CANDO instead uses only the deployed model’s score outputs.Source: §2 · p. 2
  • Derivative-free optimization uses queried function values and includes evolutionary strategies, Bayesian optimization, and finite-difference methods [42].Prior applications include black-box attacks and pretrained-model tuning [4] [19] [31]; this work applies it to canonicalization over low-dimensional rotation parameters.Source: §2 · p. 2

3. Proposed Method

CANDO frames post-deployment recovery as a compact search over input transformations for a frozen score-only point-cloud model. It scores candidate poses with model confidence, protects the identity pose, and uses CMA-ES to search the resulting non-convex objective.

3. Proposed Method

  • CANDO evaluates candidate input transformations with forward queries, selects a pose using confidence and a do-no-harm prior, and optimizes the objective with an evolution strategy.The selected pose’s cached prediction is inverse-warped to the original frame without an extra query.Source: §3 · p. 3, §3.4 · p. 4

3.1. Problem Formulation

  • The ideal per-input objective minimizes task loss after transforming the input, but black-box deployment provides neither model gradients nor test-time labels to evaluate that loss.The model remains fixed while the transformation is optimized.Source: §3.1 · p. 3

3.2. Parameterizing the Input Transform

  • CANDO represents rotations with a three-parameter Rodrigues axis-angle vector, enabling azimuthal, tilt, or full SO(3) search in a compact transformation space.An isotropic scale parameter can be appended for scaling experiments.Source: §3.2 · p. 3

3.3. Search Objective

  • The search maximizes mean maximum class probability as a confidence surrogate, whose correlation with mIoU across poses typically ranges from 0.5 to 0.65.A magnitude penalty and identity candidate protect already-aligned inputs unless another pose yields a higher confidence score.Source: §3.3 · p. 3

3.4. The CMA Evolution Strategy

  • CMA-ES searches the continuous, multi-modal pose landscape using forward evaluations and ranked candidate scores.Its global search can bypass local optima that trap gradient methods, and it requires the fewest queries among global derivative-free optimizers to locate the canonical pose.Source: §3.4 · pp. 3–4

4. Experiments

The experiments evaluate pose recovery across point-cloud tasks, backbones, and corruption settings using frozen models and forward queries. CANDO recovers part of the oracle gap and outperforms tested baselines, with qualitative results showing recovered poses aligned to upright targets.

4.1. Experimental Setup

  • The evaluation covers ShapeNetPart part segmentation, S3DIS scene segmentation, and ModelNet40 classification using PointNet++ and PointNeXt-S.Both backbones are accessed solely through forward queries.Source: §4.1 · p. 4
  • CANDO is compared with gradient-free black-box methods and white-box gradient-based references, alongside a supervised oracle selecting the best pose by the true metric.The baselines include PCA, batch-normalization-statistic adaptation, external-prior canonicalization, entropy-minimization TTA, and gradient ascent.Source: §4.1 · p. 4
  • Inputs receive random pose, azimuth, horizontal-axis tilt, or random-axis SO(3) corruption; segmentation uses mIoU and classification uses accuracy.ShapeNetPart follows the category-masked mIoU protocol with object category supplied as a one-hot input.Source: §4.1 · p. 4

4.2. Results

  • PointNet++ recovers 76% of the oracle gap under azimuth rotations and 51% under arbitrary SO(3) rotations for part segmentation.Table 1 summarizes recovery across tasks, corruption settings, and backbones.Source: Table 1 · p. 4, §4.2 · p. 5
  • On ShapeNetPart, recovered orientations closely align with clean upright targets, restoring accurate per-part labels.Figure 2 shows the misoriented input, CANDO recovery, and upright target from left to right.Source: §4.2 · p. 5, Figure 2 · p. 5
  • 0.606 mIoU: CANDO outperforms the best gradient-free baseline by 0.247 and white-box gradient ascent by 0.106 under SO(3) corruption on ShapeNet part segmentation.White-box gradient ascent reaches 0.500 mIoU after becoming trapped in the non-convex confidence landscape.Source: §4.2 · p. 5, Table 2 · p. 5
  • CANDO reaches 0.523 mIoU on S3DIS tilt and 0.354 accuracy on ModelNet40 classification, outperforming the strongest classification baseline by 0.094.The S3DIS white-box gradient-ascent baseline reaches 0.440 mIoU; PCA reaches 0.047.Source: §4.2 · p. 5

5. Further Analysis

The analysis tests when confidence-driven recovery succeeds and how optimizer choice, surrogate reliability, the identity prior, and backbone equivariance affect it.

5.1. Transformation Recoverability

  • Recovery requires an augmentation gap and accuracy-aligned confidence; on ShapeNetPart, it is largest when test corruption is absent from training augmentation.Partial recovery can remain when training augmentation only partly matches the test rotation.Source: §5.1 · p. 6, Table 3 · p. 6, Table 4 · p. 6

5.2. Optimization Strategy

  • The confidence landscape is multi-modal, but its global maximum coincides with the best-mIoU pose; across shapes and rotations, its pooled Pearson correlation with accuracy is 0.51.Local optimizers can become trapped in suboptimal peaks, making global search important.Source: §5.2 · p. 6, Figure 3 · p. 7
  • With approximately 300 queries per shape, global search recovers 53%–57% of the oracle gap, versus 17%–27% for local estimators on SO(3) part segmentation.Random search reaches 53%, CMA-ES 55%, and Bayesian optimization 57%; four local methods plateau at 17%–27%.Source: §5.2 · p. 6, Table 5 · p. 7
  • At 60 and 120 queries, CMA-ES recovers 52% and 54% of the gap, respectively; the methods converge after several hundred queries.Bayesian optimization reaches 56% at 240 queries, while CMA-ES is adopted as the default for constrained budgets.Source: §5.2 · p. 7, Table 6 · p. 7

5.3. Surrogate Reliability

  • Across PointCloud-C corruption families, the segmentation objective has a mean gain of +0.002 and retains identity on 89% of inputs, unlike the less reliable classification softmax.The classification softmax has a mean change of −0.034 and retains identity on 62% of inputs.Source: §5.3 · p. 7, Table 7 · p. 7
  • For non-pose corruptions, confidence can rise without correctness improving; the marginal-entropy objective worsens mean change from −0.034 to −0.058.The authors note that these cases may require task-specific constraints or external priors.Source: §5.3 · p. 7, Table 8 · p. 8

5.4. Identity Prior

  • The λ=0.08 identity prior stays within 0.003 of clean accuracy at α=0, while removing it drops accuracy from 0.847 to 0.702 there.Without the prior, large rotations are better recovered, but well-posed inputs are harmed.Source: §5.4 · p. 8, Figure 4 · p. 8

5.5. Beyond Equivariance

  • Equivariant Vector-Neurons backbones remain invariant under SO(3), making CANDO an identity operation; non-equivariant backbones degrade and recover part of the oracle gap.The identity behavior allows a unified deployment policy without prior knowledge of backbone equivariance.Source: Table 9 · p. 8, §5.5 · p. 8

6. Conclusion

The conclusion presents CANDO as forward-query canonicalization for frozen point-cloud models facing rotation shifts. It highlights recovery conditions and identifies settings where confidence-only adaptation is insufficient.

6. Conclusion

  • CANDO searches a compact space of input rotations to recover performance without weights, gradients, or labels, using an identity prior to avoid unnecessary changes.The method maximizes a confidence surrogate using derivative-free optimization.Source: §6 · p. 8
  • The analysis identifies an augmentation gap, confidence-accuracy alignment, and global search over a multi-modal landscape as key recovery conditions.The conclusion also notes that confidence-only adaptation can be insufficient.Source: §6 · p. 8

A. Additional Results

  • The additional-results section covers optimizer behavior, augmentation gaps, surrogate reliability, and identity-prior safety.Source: A. Additional Results · p. 12

A.1. Optimizer and Query Budget

The appendix compares local and global search across evaluation budgets and reports how recovery scales with forward queries.

A.1. Optimizer and Query Budget

  • Even after 512 evaluations, gradient ascent reaches 0.494 mIoU, versus 0.522 for random search and 0.574 for CMA-ES.At 8192 evaluations, gradient ascent reaches only 0.528 mIoU, while both global methods exceed that value after 32 evaluations.Source: §A.1 · p. 12, Figure 5 · p. 12, Table 10 · p. 12
  • CMA-ES and random search saturate after roughly 128 evaluations, while increasing the budget does not rescue local optimization.Source: §A.1 · p. 12, Table 10 · p. 12
  • On part segmentation, performance improves from 0.43 to 0.49 with 16 queries and reaches 0.55 with 256 queries.Recovery increases with diminishing returns; classification improves from 0.21 to 0.32.Source: §A.1 · p. 12, Figure 6 · p. 12

A.2. Augmentation Gap

The appendix extends the augmentation-gap analysis to part segmentation and compares rotation sensitivity and recovery across architectures.

A.2. Augmentation Gap

  • On ShapeNetPart, recovery is consistently larger when training augmentation does not cover the test corruption and decreases as coverage increases.The clean cost of SO(3) augmentation is modest: performance decreases from 0.842 to 0.835 mIoU.Source: §A.2 · p. 12, Table 11 · p. 12
  • Across three backbones, unseen tilt reduces mIoU while yaw leaves performance unchanged, including for vanilla PointNet.The table caption reports that backbones remain robust to augmented transformations but degrade substantially under unseen tilt.Source: §A.2 · pp. 12–13, Table 12 · p. 13
  • Across additional non-equivariant architecture families, CANDO recovers a substantial portion of the SO(3) classification oracle gap, with positive confidence-accuracy correlation across backbones.The authors attribute weaker recovery to the single-softmax objective rather than architecture.Source: §A.2 · p. 13, Table 13 · p. 13

A.3. Surrogate Reliability

CANDO’s confidence surrogate remains reliable across several corruptions and score representations, but recovery varies with category geometry: ambiguous shapes can favor incorrect poses.

A.3. Surrogate Reliability

  • Every ShapeNet-C corruption change is +0.001 to +0.004 mIoU, and the dense per-point confidence surrogate remains reliable across all seven corruption families.The segmentation objective keeps the identity transform for 86–92% of inputs on shared corruptions.Source: Table 14 · p. 13, §A.3 · p. 13
  • SO(3) recovery on ModelNet40 ranges from 0.297 to 0.309 for temperatures T ∈[0.5, 2], reaches 0.277 at T=4, and is 0.309 with log-probability.Negative entropy reaches 0.307; these results indicate that an ordinal confidence signal suffices rather than calibrated probabilities.Source: §A.3 · p. 13, Table 15 · p. 13
  • Tables improve by 0.38 mIoU, skateboards by 0.37, chairs by 0.35, and airplanes by 0.30, while guitars lose 0.16, rockets 0.10, mugs 0.05, and cars 0.04.Rotational ambiguity lets multiple orientations explain the geometry equally well, so confidence can favor an incorrect pose.Source: §A.3 · p. 13
  • 63% of shapes improve by more than 0.01 mIoU and 26% degrade by the same margin; mean change is +0.23.Recovery rises with corruption magnitude: inputs below 1.2 radians improve from 0.571 to 0.662 mIoU, versus 0.244 to 0.551 above 2.2 radians.Source: §A.3 · p. 14, Figure 7 · p. 14

A.4. Identity Prior and Safety

The appendix evaluates safeguards for clean and mixed inputs, then details the baselines, optimizers, and implementation used in the experiments.

A.4. Identity Prior and Safety

  • With λ=0.08, only 6% of clean inputs degrade by more than 1%, with a mean −0.005 mIoU; without the prior, 41% degrade and mean change is −0.143.The identity prior suppresses unnecessary transformations while preserving recovery.Source: §A.4 · p. 14, Table 16 · p. 14
  • Larger identity-prior weights preserve identity more often while maintaining rotation recovery and reducing harm on non-pose corruptions; smaller values increase harm on jitter and outlier addition without improving rotation recovery.The confidence-margin gate reduces harm on jitter and dropout but has little effect on global outlier addition, where confidence remains confidently incorrect.Source: §A.4 · p. 14, Table 18 · p. 14
  • On a mixed stream, part segmentation has an operating point that preserves canonical inputs while maintaining positive net gain, whereas classification trades canonical harm for rotated-input gain at every threshold.Increasing the confidence threshold leaves more canonical inputs untouched but gradually reduces recovery on rotated inputs.Source: §A.4 · p. 15, Table 19 · p. 15

B. Experimental Details

  • The appendix expands the experimental framework with baselines, optimizer comparisons, implementation details, and dataset preprocessing.Baselines include black-box-compatible gradient-free methods and white-box gradient-based references.Source: B. Experimental Details · p. 15, §B.1 · p. 15

B.1. Baselines

  • Principal-axis canonicalization aligns inputs to principal axes, while its confidence-disambiguated variant selects among eigenvector sign flips; DUA adapts batch-normalization statistics, and an external-prior baseline scores poses with a separate frozen classifier.TENT and MEMO are also described as gradient-based baselines that update normalization parameters or model parameters, respectively.Source: §B.1 · p. 15

B.2. Optimization Baselines

  • The optimizer comparison includes random search, Bayesian optimization, coordinate finite differences, SPSA, natural evolution strategies, and a score-function estimator, using forward queries and matched query budgets.Methods use three random restarts; the section notes that richer transformations may make query- and memory-efficient zeroth-order estimators relevant.Source: §B.2 · p. 16

B.3. Implementation Details

  • Full SO(3) search uses CMA-ES with two restarts and task-specific populations, initial scales, and query budgets; azimuth recovery scans 16 angles before 15 SPSA steps.The identity-prior weight is selected on a small labeled validation split and fixed thereafter.Source: §B.3 · p. 16

B.4. Datasets and Preprocessing

The experiments cover object part segmentation, indoor scene segmentation, and object classification, with benchmark-specific inputs and pose corruptions.

B.4. Datasets and Preprocessing

  • ShapeNetPart uses unit-sphere-normalized shapes sampled with surface normals, and reports category-masked instance mIoU under the standard protocol.Transformed-point predictions are propagated to the original cloud by three-nearest-neighbor voting.Source: §B.4 · p. 16
  • S3DIS evaluation uses Area 5 with standard block preprocessing and 4096-point, nine-channel inputs; tilt recovery recomputes geometric channels while leaving color channels unchanged.Source: §B.4 · p. 17
  • ModelNet40 uses 1024-point shapes normalized to the unit sphere and represented only by point coordinates.Source: §B.4 · p. 17
  • Pose corruptions use independently sampled uniform azimuth, random-axis SO(3), or horizontal-axis tilt; SO(3) angles are sampled uniformly above a small minimum, and Haar-uniform experiments lead to the same conclusions.The non-pose study uses seven official PointCloud-C corruption types at five severity levels.Source: §B.4 · p. 17

C. Limitations and Future Work

The method’s scope is constrained by its confidence signal, transformation family, and evaluation setting; future work proposes broader transformations and external priors.

C. Limitations and Future Work

  • CANDO targets rotation-induced shifts and does not address noise or density corruptions when confidence is misaligned with task performance.It assumes low-dimensional input transformations, probability or logit access, and a moderate forward-query budget; evaluation uses synthetic pose shifts, leaving real sensor tilt and outdoor LiDAR for future validation.Source: C. Limitations and Future Work · p. 17
  • Future work could extend the transformation family to affine or automatically inferred transformations, or combine pose recovery with external generative or foundation-model priors.The proposed priors target shifts that confidence-only objectives cannot resolve.Source: C. Limitations and Future Work · p. 17

Related papers

Loading 2610.34850v1…