PaperlayerView exact revision on arXiv

Source-linked AI summary

AstraLOD3: Zero-shot multimodal agentic reconstruction of LOD3 building models

Bryan G. Pantoja-Rosero

arXiv:2609.28061v1cs.CV

TL;DR

Automated LOD3 reconstruction remains challenging across heterogeneous buildings and incomplete evidence. AstraLOD3 uses a general-purpose multimodal agent to dynamically select procedures from images, calibrated cameras, sparse SfM geometry, and a reconstruction specification. Its results show that constrained agentic reconstruction can produce structured LOD3 models with image-space agreement generally stronger and geometric consistency of a similar order to prior purpose-built approaches.

  • Problem

    Detailed building models from heterogeneous, incomplete observations often require specialized algorithms and assumptions, while inferring partially observed architectural elements remains challenging.

    Source: §1 · p. 3

  • Method

    AstraLOD3 uses a general-purpose multimodal model to dynamically select and execute computational procedures from multi-view imagery, SfM evidence, and a reconstruction specification without task-specific training or fine-tuning.

    Source: §1 · p. 4

  • Results

    Comparisons with prior purpose-built LOD3 and DADT approaches showed generally stronger image-space agreement and geometric consistency of a similar order, with richer architectural components in several cases.

    Source: §6 · p. 35

  • Takeaways & Limitations

    Building-scale LOD3 reconstruction can be formulated as a constrained agentic process in which a general-purpose multimodal model selects and combines procedures according to evidence and requirements.

    Source: §6 · p. 36

    Reconstruction quality depends on available evidence, and FRDS, DCD, and IMF measure consistency with image and SfM evidence rather than complete accuracy against independent geometric ground truth.

    Source: §6 · p. 35

Abstract

from arXiv · show

Automated LOD3 building modeling typically relies on purpose-built geometric or learning-based pipelines, limiting flexibility across heterogeneous buildings and input evidence conditions. This study investigates whether Astra, a general-purpose multimodal foundation model, can address these limitations through zero-shot reconstruction of LOD3 building models within an agentic framework under bounded autonomy. AstraLOD3 combines multi-view images, calibrated cameras, and a filtered sparse SfM point cloud with a natural-language reconstruction specification, while the Astra agent dynamically selects and executes computational procedures using Python and Blender. Across 35 runs, including 24 benchmark buildings, AstraLOD3 achieved a mean FRDS of 0.9647 and geometric agreement comparable to that of previous purpose-built methods. Controlled ablations further revealed the effects of reconstruction guidance, evidence modalities, model configuration, and run-to-run variability. The results demonstrate that structured LOD3 reconstruction can be formulated as a constrained agentic process rather than as a fixed pipeline. Future work will investigate adaptive refinement, user-guided correction, task-specific specialization, and damage-aware reconstruction.

1. Introduction

Structured building models are valuable, but converting incomplete reality-capture evidence into them remains difficult for fixed or task-specific methods. AstraLOD3 investigates a zero-shot agentic alternative using multimodal evidence and dynamic computational procedures.

1. Introduction

  • Lightweight LOD representations retain application-relevant geometry as editable surfaces and components, while reality-capture images and point clouds still require conversion into structured building models.Object-level organization can support workflows such as BIM, digital twins, asset management, and analysis.Source: §1 · p. 2
  • Fixed geometric pipelines depend on predefined thresholds, priors, and building assumptions, while learning-based methods generally require task-specific data and depend on its training distribution.These constraints complicate reconstruction across heterogeneous buildings and incomplete or occluded observations.Source: §1 · pp. 2–3
  • Whether a general-purpose multimodal model can autonomously reconstruct building-scale LOD3 models zero-shot from calibrated images and sparse 3D evidence, and how guidance or model configuration affects performance, remains insufficiently understood.Prior agentic building-modeling work does not resolve these reconstruction questions.Source: §1 · p. 3
  • The framework combines multi-view images, calibrated cameras, and a filtered sparse SfM point cloud with a reconstruction specification; its agent dynamically selects and executes procedures using Python and Blender.Experiments examine specification guidance, input modalities, model configuration, and run-to-run variability.Source: §1 · pp. 3–4
  • AstraLOD3 introduces a zero-shot framework that generates lightweight, semantically organized LOD3 models and records evidence provenance, reproducible code, and reconstruction metadata.The study also evaluates 24 building cases and contributes an open input–output benchmark.Source: §1 · p. 4

2. Background and Problem Formulation

The background reviews geometry-driven, learning-based, and agentic reconstruction approaches before framing AstraLOD3 as a bounded-autonomy transformation from multimodal evidence and a specification to a structured LOD3 model.

2.1. Automated reconstruction of structured building models

  • Purpose-built geometry methods use primitives, constraints, and optimization, while learning-based methods automate perception or reconstruction stages; both retain predefined procedures, assumptions, or training-data dependencies.These approaches range from PolyFit-style geometric modeling to hybrid learned and geometric LOD3 pipelines.Source: §2.1 · pp. 4–5

2.2. Foundation models and agentic 3D generation

  • Foundation-model agents can combine multimodal inputs with external tools, execute intermediate operations, inspect outputs, and refine solutions; programmatic 3D generation also leaves editable modeling procedures.Examples include tool-using language models and Blender code generation.Source: §2.2 · p. 6
  • Prior building agents address BIM interaction, Scan-to-BIM, or façade-opening enrichment from an existing LOD2 model, whereas AstraLOD3 targets complete building generation from imagery and SfM evidence.The target includes the envelope, roof configuration, architectural components, and semantic organization.Source: §2.2 · p. 6

2.3. Problem formulation and positioning of AstraLOD3

  • The task maps multi-view images, calibrated camera parameters and poses, and a filtered sparse SfM cloud, together with specification R, to a compact model consistent with observations and relevant components.The target includes geometry, semantic components, their hierarchy, and evidence provenance indicating observed or inferred geometry.Source: §2.3 · p. 7
  • Under bounded autonomy, the specification sets objectives, evidence rules, constraints, guidance, and outputs, while the agent dynamically chooses and composes computational procedures within those limits.Components must be supported by supplied evidence, remain in the SfM coordinate system, and satisfy output and refinement limits.Source: §2.3 · p. 7
  • AstraLOD3 tests whether a general-purpose multimodal model can select and coordinate procedures for building-scale LOD3 reconstruction without task-specific training or a prescribed complete algorithm sequence.The agent is tasked with recovering the envelope, roof, openings, and other architecturally relevant components under geometric and semantic constraints.Source: §2.3 · p. 7
  • Here, zero-shot means no task-specific training or fine-tuning, building-specific pretrained reconstruction pipeline, external 3D assets, or ground-truth geometry during reconstruction.The approach is framed as dynamic procedure selection under bounded autonomy, not as a replacement for any one existing algorithm.Source: §2.3 · p. 8

3. AstraLOD3 Methodology

AstraLOD3 uses Astra through Codex to transform multimodal building evidence and a natural-language specification into a structured LOD3 model, with computational steps chosen dynamically within specified constraints.

3. AstraLOD3 Methodology

  • Astra operates through Codex with Python and Blender, adapting its computational trajectory to each building and evidence set within the constraints in R.The agent can inspect and transform data, generate task-specific scripts, construct geometry, and inspect intermediate results.Source: §3 · p. 8

3.1. Multimodal reconstruction evidence

  • The framework combines multi-view images, calibrated camera information, and a filtered sparse SfM point cloud, keeping the generated model in the SfM reference frame.The modalities provide visual, projective, and explicit 3D geometric information, respectively.Source: §3.1 · pp. 9–10

3.2. Reconstruction specification

  • The reconstruction specification sets evidence rules, geometric and semantic requirements, procedural guidance, execution limits, and required outputs without prescribing a complete algorithm.Components need image or 3D support; incomplete observations may be completed with simple inferred geometry distinguished from directly observed geometry.Source: §3.2 · pp. 10–12

3.3. Agentic reconstruction process

  • Astra dynamically inspects evidence, plans the reconstruction, generates Blender geometry, validates it, refines it, and finalizes outputs rather than following a fixed sequence of computational operations.The specification defines six high-level stages, while the agent selects operations; refinement is limited to three iterations.Source: §3.3 · pp. 12–13

3.4. Model representation and generated outputs

  • The outputs combine lightweight geometry with semantic components and provenance metadata, while independent runs may produce slightly different geometric solutions.The executable build model.py regenerates an individual solution; this does not imply deterministic repeatability across complete agentic runs.Source: §3.4 · pp. 13–14

4. Experimental design

The evaluation combines a 24-building reference benchmark with controlled runs varying reconstruction guidance, evidence modalities, and Astra configuration, plus a repeatability test. It measures image-space and geometric agreement and compares shared cases with earlier approaches, while noting a point-cloud preprocessing difference.

4. Experimental design

  • The study comprises 35 runs: 24 reference reconstructions, controlled changes to guidance, evidence, and model configuration, and an independent repeat of Building 03.The reference setup uses the complete multimodal evidence, Normal specification, and High Astra configuration.Source: §4 · p. 14, §4.1 · p. 15, §4.2 · p. 17

4.1. Benchmark cases and reference configuration

  • The benchmark covers 24 buildings—23 real cases and one synthetic case—with varied architecture and observation conditions; 14 overlap with earlier LOD3 or DADT models.Seven overlapping cases come from each earlier study, enabling comparison using common metrics.Source: §4.1 · pp. 14–15, Table 2 · p. 15

4.2. Controlled ablation and sensitivity experiments

  • Controlled experiments vary specification guidance, evidence modalities, and Astra settings while holding applicable conditions fixed; a separate Building 03 repeat examines run-to-run variability.The guidance variants progressively reduce prescriptiveness; the repeatability analysis is limited to one demanding case because repeating the full benchmark would increase computational cost.Source: §4.2 · pp. 16–17

4.3. Evaluation metrics

  • FRDS measures projected-mask agreement across evaluation views, while symmetric squared Chamfer distance and IMF assess geometric consistency with the SfM point cloud; processing time is also recorded.Lower IMF indicates closer agreement; geometric metrics are directly comparable only within the same building or at a common scale.Source: §4.3 · pp. 17–18

4.4. Comparison with previous reconstruction approaches

  • For 14 shared buildings, archived LOD3 and DADT models are evaluated alongside AstraLOD3, with scale correction enabling case-by-case geometric comparison.Earlier studies used manually cleaned point clouds, whereas this study uses automatic filtering that may retain non-building observations, so comparisons do not use identical point sets.Source: §4.4 · pp. 18–19

5. Results and discussion

This section evaluates AstraLOD3’s constrained agentic LOD3 reconstruction across benchmark buildings, examining reconstruction performance and how guidance, evidence, model configuration, and execution trajectory affect outcomes. It also considers run-to-run variability and the motivation for further refinement and user guidance.

5.1. Overall reconstruction performance

  • Across 24 reference buildings, FRDS ranged from 0.9126 to 0.9937, with a mean of 0.9647 and median of 0.9671; all cases exceeded 0.90.Processing time ranged from 12:32 to 38:47, with a mean of approximately 22:29 and median of 21:23. Geometric metrics reflect consistency with each dataset’s sparse SfM reconstruction, not complete architectural correctness; representative comparisons show local differences in simplified, weakly observed, or occluded regions.Source: §5.1 · pp. 19, 22, Table 4 · p. 20, Figure 3 · p. 23

5.2. Comparison with previous reconstruction approaches

  • AstraLOD3 achieved generally stronger image-space agreement and comparable geometric agreement than previous purpose-built approaches, while recovering more architectural detail without their task-specific procedures.FRDS was higher for nearly all overlapping cases; for Ozcan, the two values were almost identical at 0.9126 and 0.9150. Geometric results were mixed, and comparisons are affected by differences in point-cloud preprocessing and image-supported details weakly represented in sparse clouds.Source: §5.2 · pp. 24–25, Table 5 · p. 24

5.3. Effect of reconstruction-specification guidance

  • The Short specification achieved the strongest quantitative agreement, whereas Nano autonomy produced a richer interpretation that included inferred concealed surfaces not necessarily favored by the evaluation metrics.Short removed the prescribed workflow but retained the reconstruction objective and geometric and evidence constraints; Nano also introduced source-derived facade textures and lower-building geometry concealed by terrain.Source: §5.3 · p. 27, Table 6 · p. 27

5.4. Effect of reconstruction evidence

  • The complete IVCP evidence configuration provided the strongest overall quantitative agreement; photographs supported architectural detail, point clouds constrained geometry, and calibrated cameras linked images to the 3D frame.Without photographs, ICP and IP recovered simplified principal structures; without a supplied point cloud, IVC triangulated image correspondences. The image-only IV case remained recognizable but had an independently inferred scale and coordinate system, so direct comparison in the SfM frame was not reported.Source: §5.4 · pp. 28–29, Table 7 · p. 28

5.5. Effect of Astra model configuration

  • High and Extra-High configurations achieved stronger quantitative agreement than Light and Medium, while all four recovered the principal building organization and Medium was fastest at 11:33.Extra-High had the highest FRDS and lowest IMF, while High had the lowest symmetric Chamfer distance; execution times and refinement iterations varied across configurations, and each result came from one run.Source: §5.5 · p. 31, Table 8 · p. 31

5.6. Run-to-run variability

  • Two identical-condition Petrinja runs recovered the same principal organization but differed in validation views, processing trajectory, and final quantitative results, with Run 1 obtaining higher FRDS and lower DCD and IMF.An individual generated model could be regenerated from its fixed script and parameters, but a new agentic execution was not deterministic; the two-run sample does not establish a systematic advantage for either trajectory.Source: §5.6 · pp. 32–33, Table 9 · p. 32

5.7. Agentic behavior and limitations

  • AstraLOD3 produced the required outputs while adapting computational procedures to building form; iterative validation corrected local issues, but the three-iteration cap bounded refinement rather than indicating convergence.Procedures varied across runs and architectures, and additional autonomous iterations or user-directed correction could address remaining discrepancies.Source: §5.7 · pp. 33–34
  • FRDS, DCD, and IMF measure consistency with available image and SfM evidence, not complete 3D architectural accuracy, especially for unobserved or weakly supported surfaces.Inferred hidden elements are recorded with observed/inferred provenance and should not be treated as survey-grade measurements.Source: §5.7 · p. 34
  • AstraLOD3’s contribution is a constrained agentic formulation that combines computational procedures to produce a common structured LOD3 representation across heterogeneous building cases.The formulation also operates under different levels of guidance and reduced evidence.Source: §5.7 · p. 34
  • Ablation and repeated-run experiments reveal variability in execution time, intermediate decisions, and final geometric details, motivating adaptive refinement, user-guided correction, and explicit modeling of damage.The authors also identify other application-specific architectural information as a future target.Source: §5.7 · p. 34

6. Conclusions

AstraLOD3 frames LOD3 building reconstruction as a bounded-autonomy, zero-shot agentic process using multimodal evidence and a natural-language specification. Across its benchmark and controlled experiments, it produced strong image-space agreement, while evidence quality and nondeterministic execution constrain interpretation and motivate further refinement.

6. Conclusions

  • AstraLOD3 combines multi-view images, calibrated cameras, filtered sparse SfM geometry, and a reconstruction specification while the agent selects procedures and builds models without task-specific training or manual correction.Source: §6 · p. 34
  • Across 24 benchmark buildings, mean FRDS was 0.9647; comparisons showed generally stronger image-space agreement and geometric consistency of a similar order versus previous LOD3 and DADT approaches.Source: §6 · pp. 34–35
  • Controlled experiments found complementary evidence roles, guidance-dependent reconstruction choices, configuration-dependent processing and refinement, and variable computational paths across repeated runs.Source: §6 · p. 35
  • Evaluation is limited by incomplete or noisy evidence, arbitrary SfM scale, evidence-consistency metrics rather than independent ground truth, a three-iteration refinement cap, and mostly single-run ablations.Source: §6 · p. 35
  • Proposed extensions include adaptive refinement, targeted user correction, uncertainty estimates, and denser sensing for weakly observed surfaces.Source: §6 · p. 35
  • Task-specific specialization and damage-aware reconstruction are future directions, but specialization would no longer be the zero-shot setting evaluated here.Source: §6 · pp. 35–36
  • The study plans to release configurations, specifications, models, scripts, reports, intermediate results, evaluation outputs, and eligible benchmark data to support reproducibility and comparison.Source: §6 · p. 36

Declaration of generative AI and AI-assisted technologies in the writing process

The authors used ChatGPT for grammar and readability, then reviewed and edited the content and retained responsibility for the publication.

Declaration of generative AI and AI-assisted technologies in the writing process

Appendix A. Reference reconstruction specification

The reference specification directs an autonomous agent to produce a lightweight, evidence-supported LOD3 model through a staged inspection, planning, generation, validation, and refinement workflow. It also constrains evidence use, geometry, coordinate handling, autonomy, and required outputs.

Appendix A. Reference reconstruction specification

  • The objective is a detail-enriched but lightweight LOD3 model that prioritizes SfM and projected-image consistency, architectural components, and compact geometry from calibrated photographs, cameras, and filtered sparse SfM points.Source: Appendix A · p. 37
  • The agent must model evidence-supported envelopes, façade planes, roofs, openings, frames, balconies, parapets, overhangs, chimneys, and major projections, while excluding bricks, image textures as geometry, and incidental objects.Source: Appendix A · pp. 37–38
  • Components must not be invented from architectural convention; when geometry is unobservable, the specification calls for the simplest geometrically reasonable interpretation.Source: Appendix A · p. 38
  • The model should use low-poly procedural forms such as planar surfaces, extrusions, boolean openings, instances, and simple profiles, avoiding unnecessarily dense meshes.Source: Appendix A · p. 38
  • The agent must verify camera transformations, preserve the original SfM coordinate system, and generate a reproducible Blender script that rebuilds the model from an empty scene.Source: Appendix A · pp. 38–39
  • The workflow inspects images, cameras, point-cloud distribution, building planes, envelope, roof, and visible components, then records dimensions, component inventory, inferred planes, and image support in a reconstruction plan.Source: Appendix A · pp. 38–39
  • Validation renders are compared with source photographs, with at most three cycles of script modification, model regeneration, rendering, and inconsistency inspection.Source: Appendix A · p. 39
  • The output package includes model files, a regeneration script, component dimensions and provenance distinguishing observed from inferred geometry, and a run report of decisions, iterations, statistics, and ambiguities.Source: Appendix A · pp. 39–40
Loading 2609.28061v1…