Source-linked AI summary
SiZeUp: Fast 3D Proxy from Aerial Images via Depth Ordinal Loss
Wenjun Zhou, Yunshan Li, Qiaoyu Zhu, Weidan Xiong, Hao Zhang, Daniel Cohen-Or, Hui Huang
TL;DR
Large-scale urban proxy reconstruction needs a faster alternative to dense 3D workflows while preserving useful structural properties. SiZeUp extrudes footprints to estimated heights and uses ordinal depth alignment with selected views, achieving 23-52× speedups with comparable proxy-level coverage and volume consistency.
Problem
Large-scale urban proxy estimation often relies on lengthy, computationally costly point-cloud or dense reconstruction workflows.
Method
SiZeUp directly estimates building heights over extracted footprints using a differentiable renderer and ordinal consistency between rendered and prior depth orderings.
Results
23-52× speedup over state-of-the-art proxy reconstruction pipelines is achieved while maintaining comparable proxy-level coverage and volume consistency.
Takeaways & Limitations
A constrained footprint-and-height representation can preserve coherent proxy-level coverage and volume consistency while substantially improving efficiency.
Takeaways & Limitations
A single footprint extrusion cannot fully capture stepped roofs, multi-layer structures, or non-vertical details, and performance depends on footprint accuracy and reliable ordinal depth cues.
Abstract
from arXiv · showhide
We present SiZeUp, a fast and scalable approach for constructing large-scale 3D urban proxy models directly from calibrated oblique aerial imagery. Our method adopts a height-from-footprint representation, reducing 3D building abstraction to a low-dimensional optimization problem in which building footprints are extruded by a single height parameter. To enable efficient and robust height estimation, we introduce an ordinal depth consistency loss that enforces agreement between the relative depth ordering of rendered proxies and depth priors predicted by a monocular depth model. This is realized through a differentiable renderer that maps parametric building proxies into multi-view depth images, allowing gradients to be propagated from depth supervision to building heights. Our ordinal formulation produces stable optimization in practice and avoids explicit feature matching or dense point cloud reconstruction. Rather than relying on metric depth, which can be unreliable under monocular scale ambiguity, our ordinal depth consistency loss operates on relative depths, providing a more reliable signal across views. Combined with an efficient dynamic view selection, our approach achieves a 23-52$\times$ speedup over state-of-the-art proxy reconstruction pipelines while maintaining comparable proxy-level coverage and volume consistency, making it well suited for large-scale urban modeling tasks.
1 Introduction
SiZeUp directly estimates efficient 3D urban proxies from calibrated aerial imagery by extruding extracted footprints to inferred heights. Its ordinal depth objective and dynamic view selection support a substantially faster efficiency-quality tradeoff than conventional proxy reconstruction pipelines.
- Motivation: 3D urban proxies provide simplified but structurally meaningful building geometry for reconstruction priors and scalable downstream tasks.They capture spatial arrangement and core volumetric properties while prioritizing structural correctness over visual realism.
- Method: SiZeUp extrudes 2D building footprints into vertical volumes, reducing each building’s geometry estimation to a single height inference.This converts general 3D recovery into constrained image-space alignment using calibrated multi-view oblique imagery.
- Method: The ordinal consistency loss aligns relative depth ordering between rendered proxy depths and noisy depth priors instead of relying on absolute metric depth.A differentiable renderer maps building heights to multi-view depth spaces, with pairwise ordering errors penalized using a margin-based hinge loss.
- Efficiency: Dynamic view selection and low-dimensional footprint-plus-height optimization reduce computation while retaining comparable proxy-level quality.The pipeline combines learned footprint extraction with depth supervision and optimization over only a small selected set of views.
- Contribution: 23-52× speedup over state-of-the-art proxy reconstruction pipelines is achieved while maintaining comparable proxy-level coverage and volume consistency.The gain comes from avoiding lengthy point-cloud workflows and optimizing a reduced representation.
2 Related Works
Prior urban proxy methods commonly depend on reconstructed 3D data, manual procedural modeling, or specialized sensing assumptions. SiZeUp instead uses calibrated oblique imagery, footprint constraints, and ordinal depth cues to estimate building proxies directly.
- 3D proxy construction: Existing urban proxy methods often simplify precomputed dense meshes or point clouds into compact geometric abstractions.Without such 3D data, alternatives include manual modeling, interactive procedural tools, image-based inference, and procedural grammars.
- Image-based proxy reconstruction: 3D inputs are bypassed by reconstructing large-scale urban proxies directly from oblique RGB imagery.The central image-based challenges are precise 2D footprint segmentation and building height estimation.
- Footprint extraction: Traditional footprint extraction relies on edges, shadows, vanishing points, or indirect projection from SfM/MVS-derived 3D data.Large-scale approaches may also combine semantic instance segmentation with auxiliary remote-sensing products.
- Height and depth estimation: Image-based height methods can depend on task-specific annotations or sensing conditions such as satellite imagery with consistent lighting.These requirements limit generalization to oblique aerial photography with repetitive textures and varying perspectives.
- SiZeUp positioning: SiZeUp integrates footprint constraints with ordinal depth priors to mitigate scale ambiguity in oblique aerial imagery.The method treats footprints as reconstruction anchors rather than merely post-processing masks.
3 Method
SiZeUp estimates 3D building proxies from calibrated oblique aerial imagery by extruding extracted footprints and optimizing shared building heights. A differentiable renderer, ordinal depth loss, and dynamic view selection provide multi-view supervision while reducing computation.
- Footprint Extraction: SiZeUp classifies aerial views as nadir or oblique, extracts nadir-view footprints with FPSAM, and vectorizes them into polygonal building footprints.FPSAM combines semantic, texture, and depth-aware geometric cues from SAM 3 and Depth Anything 3.
- Footprint Extraction: FPSAM is designed for low-altitude oblique imagery, where perspective distortion, facade visibility, and self-occlusion challenge near-orthographic footprint extraction.The method addresses these limitations with texture, semantic, and depth-aware geometric cues from frozen foundation models.
- Building Height Estimation: Building proxies vertically extrude each footprint, reducing geometry estimation to one height parameter per building within scene-dependent bounds.Heights are initialized from a scene-dependent normal distribution and subsequently optimized across selected aerial views.
- Dynamic View Selection: Dynamic view selection periodically chooses informative oblique views using building coverage, spatial diversity, viewing-angle variation, and per-building observation criteria.In real-scene experiments, only 10%–27% of available views are used throughout optimization while retaining informative multi-view constraints.
- Ordinal Depth Consistency: Depth priors from Depth Anything 3 and rendered proxy depths are median-normalized over visible building regions before ordinal comparison.The valid region Ω_k comprises pixels with non-zero visible-building mask values.
- Ordinal Depth Consistency: The ordinal loss penalizes reversed or weakly ordered pixel-pair depths with a margin, and differentiable rendering propagates gradients to shared building heights.Per-view violations are averaged and aggregated into a piecewise-differentiable objective; multi-view coupling improves conditioning and avoids degenerate per-view solutions.
4 Experimental Results
SiZeUp is evaluated across footprint extraction, height optimization, and real-scene proxy reconstruction, showing accurate estimates, competitive proxy-level quality, and substantial speedups.
- Evaluation Protocol: The evaluation reports IoU, Precision, Recall, and F1-Score for footprint extraction, plus absolute and relative height accuracy for height estimation.
- 4.1 Footprint Extraction: FPSAM achieves the best IoU, precision, and F1-score among the compared footprint extraction methods.Near-orthographic methods degrade under oblique imagery, while SAM 3 has high recall but lower precision and IoU because of inaccurate boundaries.
- 4.2 Height Optimization: Height optimization moves building heights away from random initialization and settles into a stable LOD1 proxy despite fluctuating ordinal loss.Stability is assessed using average relative height change below 0.5% between consecutive updates, with experiments run for 20 epochs.
- 4.3 Evaluation on Real Scenes: SiZeUp produces accurate height estimates across seven real-world scenes with varied urban layouts.Table 2 reports selected-view statistics and height estimation accuracy using absolute and relative height metrics.
- 4.3 Evaluation on Real Scenes: 23-52× speedup is achieved over point-cloud-based proxy pipelines while SiZeUp remains competitive on proxy-level coverage and volume consistency.The method requires only tens of seconds; it ranks among the top two in Comp. and IoU across real scenes, despite higher point-level error from its extruded LOD1 representation.
- 4.3 Evaluation on Real Scenes: SiZeUp maintains coherent scene-level structure without SfM, and height optimization requires informative depth-ordering cues rather than a specific acquisition layout.Useful cues include building visibility, occlusion relationships, and relative depth structure between buildings and their surroundings.
5 Conclusion and Discussion
SiZeUp gains efficiency by constraining urban proxies to dominant structural degrees of freedom while preserving coherent proxy-level coverage and volume consistency. This formulation remains limited for complex stepped or multilayer buildings, though finer footprint decomposition can partially mitigate the loss of accuracy.
- Conclusion: SiZeUp aligns representation, supervision, and optimization with proxy-level task requirements rather than full geometric reconstruction.The framework deliberately constrains representation and optimizes dominant structural degrees of freedom.
- Conclusion: The task-specific low-dimensional formulation achieves substantial efficiency gains while preserving coherent proxy-level coverage and volume consistency.This conclusion emphasizes the practical trade-off between simplified representation and proxy quality.
- Limitations: Single-footprint extrusion is less accurate for buildings with stepped roofs, multilayer structures, or non-vertical details.These structures cannot be fully captured by one planar footprint and one vertical extrusion.
- Limitations: Finer footprint components with separate heights can partially mitigate inaccuracies in complex building proxies.The paper illustrates this mitigation strategy as a way to retain a low-dimensional representation while increasing structural flexibility.
- Limitations: Result quality depends on footprint accuracy and the availability of reliable ordinal depth cues.The paper identifies both inputs as constraints on the quality of the resulting proxies.