Source-linked AI summary
Polis: 3D Self-Supervision at City Scale
Alexander Rusnak, Sophia Kovalenko, Jingru Wang, Ismail Moudden, Xiru Wang, Frédéric Kaplan
TL;DR
City-scale 3D scenes are difficult targets for self-supervised learning because their sensing, density, spatial extent, and capture geometry differ from prevailing training data. Polis addresses this with an outdoor-specialized, distributionally regularized single-encoder system and finds stronger broad city-scale transfer but weaker localized terrestrial transfer. The results support domain- and geometry-aware self-supervision while exposing specialization limits.
Problem
City-scale point clouds are important across multiple applications, but existing 3D self-supervised models are trained mainly on indoor, object-level, or localized self-driving data and lack broad city-scale evaluation.
Method
Polis combines a 12.8k-scene outdoor mixture, gravity-preserving spatial view sampling, geometrically matched cosine invariance, SIGReg, and VICReg-style anti-collapse terms in a native single-encoder system.
Results
23.8% mean mIoU versus 16.3% for the next-best encoder on three pretraining-disjoint city datasets under high-capacity frozen probing, while localized terrestrial datasets favor broader baselines.
Takeaways & Limitations
Transfer improves when self-supervision is designed for outdoor capture geometry and spatial context, but this specialization is weaker on localized terrestrial streetscape and single-building scenes.
Takeaways & Limitations
Capture geometry, scene extent, and taxonomy granularity covary in the dataset suite, so their individual causal contributions remain unresolved.
Abstract
from arXiv · showhide
Reliable semantic representations derived from city-scale 3D models are increasingly important for urban analysis, infrastructure monitoring, autonomous systems, and heritage conservation. However, urban scenes of large spatial extent captured through aerial surveying differ substantially from the indoor, object-level, and self-driving LiDAR data used to pretrain most 3D self-supervised models. We introduce Polis, to our knowledge the first application of Sketched Isotropic Gaussian Regularization (SIGReg) as an objective for a native point cloud encoder, and evaluate it through a frozen-feature benchmark spanning fourteen city- and building-scale corpora. Polis combines geometrically matched cosine invariance, SIGReg, and VICReg-style anti-collapse terms with a 12.8k-scene outdoor pretraining mixture and gravity-preserving spatial view sampling. Controlled ablations show that this objective outperforms student--teacher architecture alternatives, as well as Polis versions without anti-collapse terms, on the same representative outdoor corpus. On three pretraining-disjoint city datasets, Polis reaches $23.8\%$ mean mIoU versus $16.3\%$ for the next-best encoder under high-capacity frozen probing, and $17.3\%$ versus $16.1\%$ at a matched point and voxel budget. The same city-scale lead holds on datasets whose training sets were seen in pretraining. On localized terrestrial captures with fine-grained facade and streetscape labels, the ranking reverses. Our results show that distributionally-regularized joint embedding architectures can be successful on challenging city-scale 3D scenes, and that transfer improves when self-supervision is designed for the capture geometry and spatial context of this domain while also revealing the limits of this specialization.
1. Introduction
Polis targets the underexplored transfer problem posed by heterogeneous, large-extent city-scale point clouds, combining outdoor-specialized data, spatial sampling, and a distributionally regularized single-encoder objective. It achieves stronger broad city-scale transfer while remaining weaker on localized terrestrial tasks.
- City-scale point clouds support urban planning, environmental monitoring, autonomous vehicles, infrastructure inspection, and heritage conservation, but their heterogeneous sensing, density, and extent challenge general-purpose 3D self-supervision.
- Polis combines the strongest held-out city transfer with the largest typical whole-scene occupied-voxel capacity, while localized terrestrial datasets favor broader baselines.
- Polis combines a 12.8k-scene outdoor mixture, gravity-preserving spatial views, a native LitePT-L encoder, matched global-local cosine invariance, SIGReg, and VICReg-style variance/covariance control.
- Polis evaluates city- and building-scale corpora separately, defining city-scale as broad, predominantly aerial scenes and building-scale as localized terrestrial captures with finer part taxonomies.
2. Related Work
Prior 3D self-supervision spans contrastive, masked, predictive, and self-distillation approaches, but broad comparisons focused on city-scale transfer and practical encoder capacity remain limited. The compared systems also differ in backbone, data, modalities, preprocessing, and objective.
- PointContrast, Masked Scene Contrast, and Point-MAE established contrastive and masked objectives for point clouds, while PointContrast pairs corresponding 3D points across views.
- Point-JEPA and 3D-JEPA use predictor-based point recipes on small object datasets, with targets originating from an EMA target encoder or a tuned 3D autoencoder.
- DINO-style self-distillation became a dominant 3D SSL paradigm, coupling student and momentum or EMA teacher networks while scaling to large mixed corpora.
- Broad cross-model evaluation focused on city-scale scenes and the practical capacity limits of released encoders remains comparatively limited.
- The benchmark compares pretraining scale, schedule, and pretrained DINOv2 image-tower bootstrapping across systems, using cited baseline figures and Polis’s headline-run figures.
3. Method
Polis combines an outdoor city-scale corpus, gravity-preserving multi-view sampling, and a distributionally regularized single-encoder objective. The objective aligns geometrically corresponding global–local tokens while using SIGReg and anti-collapse terms to control feature distributions, scale, and redundancy.
- System overview: Polis integrates an outdoor corpus mixture, spatial and gravity-preserving view sampling, the LitePT-L backbone, and a single-encoder distributionally regularized objective.The cross-model experiments evaluate this complete combination as a unit.
- Training data: The pretraining mixture contains 12,759 scenes from 22 outdoor and city-scale corpora spanning aerial imagery, LiDAR, photogrammetry, meshes, heritage buildings, and driving sequences.Large scenes may be partitioned into persistent spatial chunks, and corpus caps limit overrepresentation.
- View sampling: Each training pass samples one broad global view and two smaller local views from an area crop, with independent geometric and chromatic perturbations.Augmentations include differential yaw, tilt, scale, flips, jitter, elastic distortion, and cuboid dropout; these choices preserve gravity for scene-scale data.
- Geometric invariance: Nearest-neighbor matching in original coordinates pairs local and global tokens, discarding matches beyond three voxel widths so cosine invariance aligns spatially corresponding content.More than 80% of local tokens satisfy the correspondence criterion for most scenes.
- Distributional regularization: SIGReg is applied to final-stage tokens, while variance and covariance terms constrain feature scale, prevent collapse, and penalize redundancy.The added VICReg-style terms trade SIGReg’s single-hyperparameter formulation for explicit control over collapse, feature scale, and redundancy.
- Evaluation protocol: The locked evaluation uses fourteen datasets without selecting models by test metrics.The passage specifies that optimizer, precision, distributed-training, and exposure details are reported in Appendix G.
4. Experiments
Polis is evaluated through frozen-feature semantic segmentation on fourteen outdoor city- and building-scale corpora, with objective ablations and analyses of spatial, geometric, and density-related behavior. It leads broad city-scale transfer, while localized terrestrial tasks and some density conditions expose specialization limits.
- Evaluation protocol: Frozen-feature semantic segmentation covers fourteen outdoor city- and building-scale corpora, using Efficient Probing adapted for dense segmentation.The benchmark includes varied capture modalities and urban styles, with city- versus building-scale defined by capture regime and spatial extent.
- Objective ablations: Encoder-only leads City-4 and All-6 in the small-corpus ablation, while removing all anti-collapse terms remains weak.Three seeds do not establish meaningful accuracy gains from predictor removal; the encoder-only design is retained for simplicity.
- Main results: 21.4% mean mIoU versus 15.2% for Utonia gives Polis the high-capacity lead across eleven city-scale datasets.Overall per-dataset means are much closer at 19.8% versus 19.2%, reflecting weaker transfer to localized terrestrial datasets.
- Main results: 17.1% versus 13.6% preserves Polis’s city-scale lead at matched budget, despite near-tied overall means of 16.7% versus 16.4%.The gain therefore does not arise solely from Polis’s larger practical encoding capacity; localized terrestrial sets favor broader baselines.
- Qualitative spatial artifacts: Polis produces more coherent urban-object and ground separation, including a shared top-of-object direction, while Sonata and Utonia show center or axis-aligned artifacts.The top-down PCA labels are per-scene categorical keys, not cross-model encodings.
- Radial and geometric shortcuts: Polis remains above baselines across XY-radius rings, but lower rxy variance alone does not explain its center-accuracy advantage.Swiss3DCities favors tile centers for every backbone, whereas SUM results are stronger at the margin, indicating that label layout also matters.
- Density stability and encode max-fill: Polis has the lowest CKA similarity to its 400k-point reference under decimation, yet remains competitive in ridge mIoU at the lowest densities.The authors identify limited sparsity augmentations and dataset-density variation as possible sources of density shortcuts.
5. Discussion and Limitations
Polis transfers best to broad aerial city scenes but is weaker on localized terrestrial captures, while its density sensitivity may reflect limited sparsity augmentation and dataset-density shortcuts.
- Polis transfers best on broad aerial city scenes and is weaker on localized terrestrial streetscape and single-building scenes.
- Polis is more sensitive to decimation than Sonata, Concerto, and Utonia while remaining competitive on ridge-probe mIoU.
- The city-scale lead is a property of the complete system, not evidence that the encoder-only design alone produces better features.
6. Conclusion
Polis is an outdoor-specialized, encoder-only system designed for city-scale 3D understanding, using distributional regularization, matched views, outdoor data, and gravity-preserving augmentation. It improves frozen transfer on broad city scenes but trades off performance on localized terrestrial tasks.
- 23.8% mean mIoU versus 16.3% for the next-best encoder on three pretraining-disjoint city datasets under high-capacity probing.
- Polis combines an encoder-only distributionally-regularized objective with geometrically matched views, outdoor data, and gravity-preserving augmentation.
- On localized terrestrial captures with fine-grained building-part and streetscape labels, Polis reaches 14.1% mIoU versus 33.5% for the strongest baseline.
- The headline run indexes 12,759 scenes from 22 outdoor and city-scale corpora after dataset caps and removal of broken scenes.
C. Evaluation Corpora
The evaluation corpora comprise public-label city-scale datasets and localized building-scale datasets, with SensatUrban used only for validation because official test labels are unavailable.
- City-scale probe sets include DALES, Swiss3DCities, Turin3D, SUM, YTU3D, Hessigheim3D, UrbanBIS, UNS-Geo, HRHD-HK, and WHU-Urban3D.
- Building-scale sets include ArCH, UnderOneFacade, and SemanticUrban.
- SensatUrban is validation-only because its official test labels are not public.
D. Random-Initialization Probe
The random-initialization probe calibrates how much frozen segmentation performance comes from pretraining rather than the probe head, using an untrained LitePT-L under the matched-budget protocol.
- Random sparse-convolution features still encode local voxel geometry, so the performance gap measures a contribution from pretraining rather than only the probe head.
- Table 4 reports matched-budget frozen-probe mIoU, with Random-init as an untrained LitePT-L and Polis as the headline pretrained checkpoint.
- The high-capacity one-shot scores use model-specific point and voxel caps, including 1.5M/1M for Polis and smaller caps for the baselines.
F. Class-Conditional Spherical Variance
Frozen-feature within-class spherical variance is nearly stable across XY-radius rings for all three city datasets, so radial position does not explain the Swiss3DCities mIoU drop.
- 0.011 is the largest centre-to-margin variance range, observed for Utonia on SUM; Polis reaches 0.010 on DALES.
- Swiss3DCities has centre-to-margin spherical-variance ranges of at most 0.007.
- The Swiss3DCities centre-to-margin mIoU drop is therefore not explained by frozen-feature spherical variance.
G. Hyperparameters
The reported protocol combines spatially structured multi-view augmentation with reproducible probing and hyperparameter documentation, while retaining batch-level statistics.
- The recipe uses persistent spatial chunks, per-pass area crops, DINO global and local views, and differential yaw.
- Table 5 reports high-capacity native-preprocessing mIoU with model-specific one-shot caps and validation-based probe selection.
- Table 6 documents Polis pretraining and frozen-probing hyperparameters, including checkpoints, sampling, objective weights, and resource settings.
- Batch-level rather than per-scene statistics are used in the training recipe.
H. Sparse-Convolution Capacity Runtime
The sparse-convolution evaluation is constrained by a released-library 32-bit byte-addressing guard that can reject sufficiently large sparse tensors despite available GPU memory; unofficial rebuilds were excluded.
- Sufficiently large sparse tensors can trigger a released implicit-GEMM assertion when N×C×dtype-bytes exceeds a 32-bit byte-addressing guard.
- The guard is separate from 64-bit hashing support for large spatial shapes and may trigger while substantial GPU memory remains available.
- Community remedies involve rebuilding cumm and spconv with widened 64-bit limits and convolution assertions.
- The comparison excludes these unofficial CUDA 12.4 remedies because they would introduce an additional systems variable; one workaround is explicitly unsuitable for training.