Source-linked AI summary
SVI2LoD3: Agent-Driven Reconstruction of LoD3 Facade Openings in Semantic 3D City Models from Volunteered Street View Imagery using Large Language and Visual Models
Elmehdi Kanna, Lukas Arzoumanidis, Huynh Duc An Son Nguyen, Youness Dehbi
TL;DR
Detailed facade openings are difficult to reconstruct at scale from noisy, incomplete imagery, while existing approaches often depend on manually annotated segmentation data. This paper combines agent-driven language-model reasoning, zero-shot vision segmentation, and expert-designed programs to generate CityGML-conformant LoD3 models and introduces FFD for feature-space evaluation. The pipeline achieves strong eTRIMS benchmark performance and produces ready-to-use models, while future work must extend reconstruction to additional facade characteristics and data sources.
Problem
Most city models remain at LoD2, and reconstructing detailed facade openings from noisy observations requires methods that handle heterogeneous architecture without extensive manual annotation.
Method
The pipeline combines agent-driven LLM reasoning, SAM 3 zero-shot segmentation, expert-designed geometric programs, and FFD feature-space evaluation for facade reconstruction.
Results
The pipeline achieves sufficiently strong eTRIMS benchmark results and produces ready-to-use, CityGML-conformant LoD3 building models.
Takeaways & Limitations
Zero-shot segmentation reduces dependence on labor-intensive training annotation while the agent-driven validation step supports directly usable CityGML-conformant LoD3 outputs.
Takeaways & Limitations
Future work must extend semantic enrichment to balconies, protrusions, dormer windows, and additional data sources for reasoning and GPS-based map matching.
Abstract
from arXiv · showhide
This paper presents an end-to-end, agent-driven pipeline for the LoD3 reconstruction of facade openings in 3D city models, producing directly usable CityGML-conform outputs. In contrast to existing approaches that rely on supervised semantic segmentation and therefore require large amounts of manually annotated training data, the proposed method employs a zero-shot segmentation strategy. This substantially reduces the annotation effort while still achieving strong performance in our benchmark on the eTRIMS dataset. A further key contribution is the enforcement of correct partonomic hierarchies, thereby producing CityGML-conform LoD3 building models. Beyond the reconstruction pipeline itself, this work also introduces a novel evaluation metric for facade reconstruction, termed Facade Feature Distance (FFD). Unlike conventional metrics such as mIoU or FRDS, which assess similarity primarily through pixel-wise overlap, FFD measures distance in a high-level feature space derived from a vision transformer. In doing so, it captures both semantic correctness and architectural layout, providing a more suitable assessment of facade reconstruction quality. The proposed pipeline and evaluation strategy together offer a practical and scalable contribution toward the automated generation and analysis of semantically enriched 3D city models. The developed code is published at: https://github.com/hcu-cml/citydb-SVI2LoD3-ai.
1. Introduction
Detailed LoD3 facade models are needed for semantically rich urban analysis, but most city models remain at LoD2 because large-scale reconstruction of heterogeneous facade openings is difficult. The paper proposes an agent-driven pipeline combining zero-shot vision segmentation, language-model reasoning, and expert-designed programs to produce CityGML-conformant models and evaluate both semantic and topological quality.
- Research gap: Most publicly available semantic 3D city models remain at LoD2, omitting detailed facade composition and opening geometry needed for fine-grained urban modeling.The missing detail reflects the complexity of acquiring and reconstructing facade information at scale.
- Research gap: Noisy, incomplete observations introduce measurement errors, occlusions, illumination variation, and perspective distortions into facade reconstruction.Volunteered street-view imagery offers broader coverage because camera images are less demanding to acquire and handle than large point clouds.
- Proposed approach: The proposed pipeline combines large language models, zero-shot large-vision-model segmentation, and expert-designed algorithms to reconstruct facade openings from volunteered street-view imagery.The expert-designed programs support the reconstruction process while targeting semantic, geometric, topological, and syntactic correctness under CityGML 2.0 LoD3.
- Contributions: The framework combines expert-designed programs with agent-driven reasoning to address diverse facade-opening arrangements and geometries.The paper identifies this combination as among the first of its kind and pairs it with a feature-space evaluation metric.
- Proposed approach: The approach targets end-to-end reconstruction of CityGML-conformant LoD3 building models from volunteered street-view imagery.This contribution is presented alongside the agent-driven framework and its model-generation objective.
- Contributions: The proposed evaluation metric emphasizes semantic and topological accuracy rather than only semantic accuracy from pixel-wise metrics.This complements the reconstruction pipeline by assessing facade quality beyond conventional overlap-based evaluation.
2. Related Work
Related work reconstructs buildings and facade openings from LiDAR, street-view imagery, and airborne RGB imagery using geometric, rule-based, image-to-mesh, and deep-learning methods. These approaches provide foundations for enhancing low-detail models, while the paper positions its method within this broader reconstruction landscape.
- Methodological landscape: The reviewed approaches span point-cloud reasoning, geometric rules, image-to-mesh reconstruction, and semantic segmentation for facade and building-model generation.Together, these studies establish the methodological context for reconstructing facade openings and increasing model detail from observational imagery.
- LiDAR-based reconstruction: LiDAR-based methods exploit geometrically rich terrestrial or airborne point clouds to infer building structures and facade elements.Examples combine support-vector machines, probabilistic relational reasoning, geometric algorithms, and predefined architectural rules.
- Street-view reconstruction: Street-view approaches include SfM with deep-learning segmentation for free-standing LoD3 buildings and image-to-mesh reconstruction from a single street-view image.These methods focus on detecting facade openings or increasing the detail of commonly available LoD1 block models.
- Model enrichment: Other work enriches LoD2 models by extracting semantic information from textured facades using deep-learning computer vision methods.Related approaches also combine low-detail building models with panoramic imagery and semantic segmentation to preserve facade properties.
3. Methodology
The methodology combines LLM reasoning, LVM zero-shot segmentation, and expert-designed programs to reconstruct facade openings from SVI in CityGML-conform LoD3 models. It also evaluates semantic, topological, and completeness quality while matching images to camera-facing LoD2 surfaces.
- Agent-driven reconstruction: The agent coordinates LLM reasoning with LVM segmentation and expert-designed programs for SVI-to-LoD3 facade-opening reconstruction.The programs support image-to-facade matching, opening extraction, and integration into LoD2 models.
- Evaluation: The evaluation focuses on semantic, topological, and completeness dimensions using established metrics and a new metric that jointly captures semantic and topological quality.Geometry is excluded from the three assessed OGC error dimensions in this work.
- Zero-shot facade segmentation: SAM 3 performs text-conditioned, zero-shot segmentation of windows, doors, and facade surfaces without task-specific fine-tuning or labeled training data.Prompts generate multi-class masks for the target facade elements.
- Geometry refinement: The workflow derives facade-element geometry from class-specific mask colors, then refines noisy and occluded opening geometries using agent-guided constraints.The refinement removes artifacts and checks overlaps while preserving partially visible genuine openings.
- Facade surface matching: Camera-facing LoD2 surfaces are selected by transforming camera data into the model CRS, filtering by visibility and normal alignment, and comparing camera-to-centroid distances.Small extrusions are retained when matching surfaces are within a 1 m perpendicular separation.
1 Coordinate transformation and view direction estimation
The method transforms camera parameters and position into the LoD2 model’s coordinate reference system before estimating the camera viewing direction.
- Camera parameters and position are transformed to the LoD2 coordinate reference system, followed by computation of the camera normal vector.
4 Visibility and geometric filtering
Visibility filtering identifies WallSurfaces that can be observed from the camera by using surface normals and a photogrammetric visibility criterion.
- Each WallSurface normal is evaluated against the camera normal to retain surfaces visible from the camera.
11 Candidate ranking
Candidate WallSurfaces are ranked using normal consistency and distance, with the nearest candidate selected after sorting.
- Candidate surfaces are first ordered by normal-consistency score and then sorted by ascending distance to select the nearest candidate.
21 Fac¸ade aggregation
Candidate WallSurfaces are aggregated when their perpendicular distance from the nearest matched surface is below 1 m.
- WallSurfaces are added to the matched set when their perpendicular distance from the nearest matched surface is less than 1 m.
27 Export
The pipeline matches facade geometry to CityGML wall surfaces, distributes openings consistently across complex facades, and exports validated LoD3 CityGML models. Evaluation uses adapted eTRIMS masks and combines conventional overlap metrics with the feature-based FFD metric.
- Export: Matched wall surfaces are exported with CityGML wall identifiers and semantic dimensions for geometrically consistent opening distribution.This supports facades represented by multiple WallSurface elements.
- Export: Facade-opening geometries and surface coordinates are matched to LoD2 wall geometry to preserve correct partonomy and scaling.
- Export: Segmentation masks are proportionally subdivided across multiple WallSurfaces according to their LoD2 widths, preserving opening arrangement and topology.
- Export: Windows and doors are inserted as explicit CityGML elements, schema-validated, and written as compliant CityGML 2.0 LoD3 documents.
- Evaluation: The eTRIMS benchmark provides street-view images and annotations for eight semantic classes across varied European architectural styles and cities.
- Evaluation: Evaluation removes irrelevant classes and adapts facade, window, and door colors to match SAM 3 outputs before computing reconstruction metrics.
- Evaluation: mIoU measures class-wise pixel overlap, while FRDS assesses overlap between ground-truth masks and re-projected modeled facades.Both metrics primarily evaluate pixel-level agreement.
- Evaluation: FFD addresses pixel-level metric limitations by comparing learned semantic and contextual image representations in a transformer feature space.
4. Experimental Results
Experiments on heterogeneous eTRIMS facades evaluate zero-shot segmentation, reconstruction quality, model behavior, and opening completeness using quantitative and qualitative analyses.
- Evaluation setup: Experiments combine mIoU, FRDS, FFD, and qualitative analysis of architecturally heterogeneous buildings in Hamburg.
- Segmentation: 0.7221 mIoU was achieved by SAM 3 without task-specific fine-tuning on heterogeneous eTRIMS facades.Dormer windows present in the ground truth contributed substantially to observed error because the approach does not reconstruct them.
- Reconstruction metrics: 0.7654 FRDS and 0.4945 FFD were obtained for LLM-refined facade masks, with dglobal 0.4523 and dpatch 0.5367.The reported FRDS provided limited additional expressive power, whereas FFD assessed semantic and topological reconstruction together.
- Model comparison: GPT-5.1 produced more accurate shape completion and fewer hallucinations than Qwen 3 in qualitative comparisons.Qwen 3 introduced nonexistent roof windows and an additional basement door, while expert programs prevented these predictions from degrading final reconstruction results.
- Qualitative results: Both models reconstructed partially occluded garage doors, basement windows, and small windows hidden by ornamentation or balconies.
- Completeness: Window detection reached 76.66%, while door detection reached 106% relative to eTRIMS annotations.The excess door rate was attributed to incomplete or inaccurate ground-truth annotations, including garage doors.
5. Outlook & Conclusion
The paper presents an agent-driven, zero-shot pipeline that reconstructs facade openings into ready-to-use CityGML-conformant LoD3 models and evaluates reconstruction beyond pixel overlap. Future work will extend semantic enrichment to additional facade characteristics and data sources.
- The end-to-end pipeline uses zero-shot segmentation and automatic validation to produce ready-to-use, CityGML-conformant LoD3 building models.This avoids the manual training-data annotation required by supervised semantic segmentation while achieving sufficiently strong eTRIMS benchmark results.
- Facade Feature Distance (FFD) evaluates reconstruction using vision-transformer feature distances rather than pixel-wise overlap alone.It is designed to assess both semantic correctness and architectural layout fidelity.
- Future work will reconstruct additional facade characteristics, including balconies, protrusions, and dormer windows.The paper identifies these additions as a way to further strengthen semantic 3D city models for urban simulation and analysis.