PaperlayerView exact revision on arXiv

Source-linked AI summary

BlueprintAgent: Constraint-Triggered Targeted Revisits for Simulation-Ready Generation from Scanned Structural Blueprints

Zhouyuan Xu, Chen Yang, Linhao Wang, Jiansheng Fan, Chen Wang

arXiv:2609.07362v1cs.CLcs.AIcs.CV

TL;DR

In-service RC blueprints are often converted into FEM models manually, while direct MLLM prompting can violate structural constraints. BlueprintAgent uses callable engineering validators to trigger local MLLM revisits, and achieves a Beam F1 of 0.994 versus 0.301 for single-MLLM zero-shot and 0.820 for a fixed pipeline. Its evaluation is limited by the absence of a fine-tuned detector baseline.

  • Problem

    Direct whole-sheet MLLM prompting can miss annotations, fabricate beam–column connections, and produce structurally inconsistent outputs from scanned blueprints.

    Source: §1 · p. 1

  • Method

    BlueprintAgent makes the MLLM the primary reader and uses callable engineering-constraint validators to trigger entity-level revisits over local visual evidence.

    Source: §1 · p. 2

  • Results

    Beam F1 was 0.994 for B5, versus 0.301 for single-MLLM zero-shot B2 and 0.820 for fixed pipeline B3.

    Source: §5.5 · p. 8

  • Takeaways & Limitations

    On 300 scanned blueprint sheets, constraint-triggered revisits improve beam topology, column identification, and text binding over single-MLLM zero-shot and a fixed OCR-plus-CV pipeline.

    Source: §6 · p. 9

    The evaluation omits a fine-tuned detector baseline that would compare a strong visual detector plus post-processing against the MLLM agent.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Converting in-service reinforced-concrete (RC) building blueprints into simulation-ready models---structured frame representations that support deterministic FEM export and qualified-engineer review---underpins safety assessment and seismic retrofit, but the process remains manual. Direct prompting of a multimodal large language model (MLLM) over a scanned sheet is unreliable: outputs often violate engineering constraints on beam--column support, span count, or 3D continuity. We present BlueprintAgent (BPA), a constraint-triggered multimodal agent for simulation-ready frame extraction from scanned blueprints. BPA treats the MLLM as the primary reader and decision maker, with OCR and computer vision supplying localized evidence. Its central mechanism realizes engineering constraints as callable validators whose entity-level conflict reports trigger targeted MLLM revisits over the local region---an inference-time control distinct from fixed pipelines and free-form self-reflection. We evaluate BPA on 300 real scanned blueprint sheets from 20 anonymized RC frame projects, against five baselines and six ablations. BPA reaches a macro-averaged Beam F1 of 0.994, against 0.301 for single-MLLM zero-shot and 0.820 for a fixed pipeline; removing MLLM-led axis adjudication collapses Beam and Column F1 on complex multi-sheet projects. For dense technical drawings, engineering constraints are best deployed as triggers for entity-level targeted revisits rather than as post-hoc output filters.

1 Introduction

Scanned structural blueprints require engineering semantics across sheets, not just OCR, because simulation-ready frames depend on correct entity bindings and topology. BlueprintAgent uses MLLM-led reading with constraint-triggered local revisits to produce evidence-linked structural models.

1 Introduction

  • Blueprint semantics depend on relationships: labels must bind to beams or columns, beams need correct supports and span counts, and columns must continue across floors.For example, the 4 in KL8(4) denotes a four-span frame beam, while an axis label defines a spatial coordinate.Source: §1 · p. 1
  • Direct whole-sheet prompting can miss small annotations, fabricate beam–column connections, and yield schema-valid JSON that violates geometric, annotation, or 3D-consistency rules.These failures motivate cross-checking perception output and re-reading evidence at conflict sites.Source: §1 · p. 1
  • BPA represents projects through sheets, regions, evidence records, and a simulatable 3D model, with the MLLM as primary perceiver and decision maker.It invokes OCR, CV-based axis probing, local re-OCR, axis-aware cropping, and table reading on demand.Source: §1 · pp. 1–2
  • The study focuses on frame-level RC reconstruction; shear-wall, steel, and masonry structures and detailed reinforcement are not main-task metrics.The stated scope includes axes, columns, primary beams, sections, floors, materials, 3D consistency, and simulation-ready output.Source: §1 · p. 2
  • BlueprintAgent couples MLLM-based reading with OCR and CV tools, using engineering-constraint failures to trigger entity-level targeted revisits rather than post-hoc filtering.The MLLM revisits relevant local evidence and may revise an entity, mark it uncertain, or accept a documented anomaly.Source: §1 · p. 2
  • The task takes all scanned blueprint sheets for a project and produces evidence-linked structural JSON, a 3D frame, and OpenSees-ready FEM input.The paper evaluates 300 real scanned sheets from 20 anonymized RC-frame projects, with five baselines, six ablations, and six end-to-end metrics.Source: §1 · p. 2

2 Related Work

Prior engineering-drawing work addresses local elements, annotation extraction, and BIM reconstruction, while BPA targets multi-sheet structural closure into a simulatable 3D model. Its distinction is using domain-validator conflicts to direct evidence collection and entity revision.

Engineering drawing digitization

  • BPA targets a multi-sheet simulatable 3D structural model whose correctness depends on cross-sheet parsing, beam–column topology, floor alignment, and FEM readiness.The related-work passage contrasts this with prior drawing work focused on local elements or annotation extraction within a single drawing.Source: §2 · pp. 2–3

Floor plan and CAD understanding

  • Unlike floor-plan systems centered on rooms and indoor elements, structural-blueprint understanding must recover axes, RC columns, frame beams, sections, elevations, and materials for engineering analysis.Its target is a load-bearing structural representation, not a room-topology or segmentation map.Source: §2 · p. 3

Visual document understanding and OCR

  • A number on a structural blueprint can denote a primary-axis span, sub-span, detailing dimension, or table value depending on its relationships to grids and components.BPA treats OCR and CV outputs as evidence interpreted under active constraint context.Source: §2 · p. 3

Tool-augmented and reflective agents

  • BPA differs from reflective and tool-augmented agents by using domain-level callable validators to produce structured conflict items that direct local visual re-inspection.The items contain an entity ID, violated constraint, and supporting evidence.Source: §2 · p. 3

Constrained and structured generation

  • Unlike token-level constrained decoding or textual neuro-symbolic inference, BPA uses constraint failures to reacquire visual evidence, re-extract entities, and update a simulatable 3D model.The paper positions its contribution as an MLLM-based reading-perception design, not a new OCR model, detector, or FEM solver.Source: §2 · p. 3

3 Task and Data

The task formalizes project-level reconstruction from complete sets of scanned RC structural blueprints into an evidence-linked state that can support 3D visualization and FEM export. The data and evaluation are organized around multi-sheet projects, with frame-level scope and manually reviewed project-level ground truth.

Input unit

  • The input is the complete set of scanned structural-blueprint sheets for one RC frame project, which may contain a few to several dozen sheets.The system aggregates evidence across sheets; individual sheets may include primary views and auxiliary regions whose roles govern how evidence enters the model.Source: §3 · pp. 3–4

Output

  • The canonical output is evidence-linked JSON containing axis grids, floors, columns, primary beams, materials, and validation status, alongside a 3D frame visualization.Once key geometric and material fields are complete, a deterministic exporter generates an OpenSees .tcl file.Source: §3 · p. 4

Scope

  • The task covers frame-level reconstruction of axes, columns, primary beams, sections, floors, materials, and 3D consistency, but not fine-grained reinforcement metrics.Reinforcement may be recorded as non-core content or auxiliary evidence, but it is not evaluated as simulation-ready frame recovery.Source: §3 · p. 4

Dataset

  • The evaluation corpus contains 300 real scanned blueprint sheets covering 102 floors from 20 anonymized RC-frame projects, with difficulty tiers based partly on project size, grid density, and scan quality.GPT-5.4 inference is used without parameter fine-tuning; project-level JSON and OpenSees models were manually annotated and reviewed entity by entity.Source: §3 · p. 4
  • The public academic release includes anonymized sheets, project-level ground-truth JSON, annotated OpenSees models, a manifest, annotation materials, evaluation code, and a datasheet.The terminology distinguishes sheets, semantic regions, structural entities, localized evidence, validator conflicts, and the project-level simulatable model.Source: §3 · p. 4

4 Method

BPA builds a project-level evidence pool and incrementally constructs a simulatable 3D model through staged extraction, validation, and targeted revisits. The MLLM makes structural decisions, while engineering validators identify conflicts for it to revisit against local visual evidence.

4 Method

  • The agent builds a project-level evidence pool, processes axes, columns, beams, floors, and materials, and progressively closes a simulatable 3D model.Its stages include axis-aware cropping, element extraction, semantic validation, 3D consistency checks, and FEM export.Source: §4 · p. 4
  • BPA treats the MLLM as primary reader and decision maker, calling OCR, computer vision, and table tools for localized evidence as needed.Its tool library includes local re-OCR, axis probing, axis-aware cropping, and an FEM exporter.Source: §4 · p. 5
  • Columns and beams are represented through grid intersections, axis labels, and node sequences rather than pixel coordinates.An axis-span sanity gate compares pixel-distance ratios with annotated span ratios using a ±15% tolerance and triggers a targeted revisit when they deviate significantly.Source: §4 · pp. 5–6
  • BPA uses validator-triggered targeted revisits: domain constraints emit structured conflicts that direct the MLLM back to relevant visual evidence.The conflict is supplied to the next plan call; corrective actions remain with the MLLM rather than hard-coded rules.Source: §4 · p. 6
  • Four validator families check axis grids, geometry and topology, annotation consistency, and 3D consistency across the model.Checks include beam support, span-count agreement, attribute completeness, and column continuity across floors.Source: §4 · p. 6
  • When conflicts arise, the MLLM can revise an entity, mark it uncertain for human review, or accept a legitimate engineering exception.Examples of accepted exceptions include cantilever beams and intentionally terminated columns.Source: §4 · p. 6

5 Experiments

The experiments test whether engineering-constraint feedback improves frame extraction, identify important components, and examine where the gains come from. BlueprintAgent outperforms single-MLLM prompting and a fixed pipeline, while targeted checks isolate the contribution of localized rereading.

5.1 Research questions; 5.2 Baselines; 5.3 Ablations; 5.4 Metrics

  • The experiments ask whether targeted revisits improve extraction, which components matter on complex projects, and where gains over three baseline approaches come from.Five systems share an MLLM backbone when applicable, and six ablations remove individual components; the suite measures axis, column, beam, and binding F1, JSON validity, and structured-model completeness.Source: §5.1 · p. 6, §5.2 · p. 6, §5.3 · p. 7

5.4 Metrics

  • B5 reaches 1.000 on Axis F1, Col F1, Bind F1, JSON Validity, and SMC, while its remaining 0.6% Beam-F1 error is concentrated in perception and dense-binding cases.The metrics assess distinct properties, including axis identity and position, column inventory, KL-to-beam binding, schema conformance, and structured-model completion.Source: §5.4 · pp. 7–8

5.5 Main results

  • B5 reaches Beam F1 0.994, versus 0.301 for single-MLLM B2 and 0.820 for fixed-pipeline B3; B5 also improves Col F1 and Bind F1 over B4.The B4–B5 comparison isolates constraint feedback: B4 retains the agent loop and tools but suppresses engineering-constraint feedback during inference.Source: Table 1 · p. 8, §5.5 · p. 8

5.6 Ablation study

  • A6, which removes MLLM-led axis adjudication, has macro Beam F1 0.369 and Col F1 0.614, with stronger effects on complex multi-sheet projects.A3 inflates column inventory by 2.0–3.0× through duplicate cross-sheet columns and reduces Col F1 to 0.516; A5 yields Beam F1 0.828 and Col F1 0.831, while A4 assigns SMC 0 when C3D validation is disabled.Source: §5.6 · p. 8, Table 2 · p. 8

5.8 Case studies and counterexamples

  • Three representative cases pair ground-truth and BlueprintAgent 3D models; an A6 counterexample shows ghost axes when CV candidates bypass MLLM re-checking.A separate case passes C3D and runs in OpenSees but produces anomalous top-floor displacement, motivating qualified-engineer review before engineering use.Source: §5.8 · pp. 8–9, Figure 4 · p. 9

5.7 Targeted robustness and isolation checks

  • On the prespecified five-project subset, B5 has Beam F1 0.9609 versus B4’s 0.7894, and exceeds post-hoc B6 by 0.1104.All 15 project-by-repeat contrasts favor B5; 23 of 32 conflicts classified as requiring localized visual rereading are resolved after rereading, and the B5-over-B4 direction holds for two replacement backbones.Source: §5.7 · p. 8, Table 3 · p. 9

5.9 Findings and analysis

  • The small B3–B4 and large B4–B5 gaps identify structured conflicts used as entity-level revisit triggers—not looping alone—as the operative mechanism.Post-hoc constraints do not recover the recall gained through localized rereading; the B5-over-B4 direction holds across three tested backbones, with larger gains on complex projects.Source: §5.9 · p. 9

5.10 Adjudicated consensus error profile

  • Residual B5 errors are adjudicated as 45% perception, 25% topology, 20% binding, and 10% completeness.Two structural engineers categorized the errors independently, with disagreements arbitrated by a third.Source: §5.10 · p. 9

6 Conclusion

BlueprintAgent uses engineering constraints to trigger targeted MLLM revisits to local evidence, rather than applying constraints only as a final filter. Across 300 scanned sheets, the paper reports improved beam topology, column identification, and text binding over single-MLLM zero-shot and a fixed OCR-plus-CV pipeline.

6 Conclusion

  • BlueprintAgent turns engineering constraints into triggers that return the MLLM to the local region where its first read went wrong.Its representation links sheets, regions, and evidence to a simulatable 3D model, with callable, entity-scoped validators.Source: §6 · p. 9
  • The evaluation covers 300 real scanned blueprint sheets across 102 floors.The paper reports consistent improvements in beam topology, column identification, and text-binding metrics over single-MLLM zero-shot and a fixed OCR-plus-CV pipeline.Source: §6 · p. 9
  • The paper’s scope is a multi-sheet, simulation-ready structural frame representation, with constraints used as auditable triggers for multimodal evidence collection and entity revision.The contribution is a reading-perception design built on an MLLM, not a new OCR model, detector, or FEM solver.Source: §6 · p. 9

Limitations

The evaluation is limited to RC frames, and several practical, methodological, and measurement boundaries constrain how broadly its results can be applied.

  • The evaluation covers RC frames; shear-wall, steel, masonry, and complex hybrid structures remain outside its empirical scope.Source: Limitations · p. 9
  • Each core entity requires structural-engineer-level annotation and review, so scaling the drawing library requires corresponding engineering-annotation effort.Source: Limitations · pp. 9–10
  • Portability evidence is limited to GPT-5.4, Claude 4.6 Opus, and Qwen 3.6 Plus, with replacements tested on a five-project subset.Source: Limitations · p. 10
  • The main evaluation matrix is single-run; three repeats per configuration assess only the B4–B5 contrast on a stratified five-project subset.Source: Limitations · p. 10
  • The exported OpenSees models are verified only for linear-elastic static analysis, not dynamic, push-over, time-history, or nonlinear analyses.Source: Limitations · p. 10
  • The evaluation omits a fine-tuned detector baseline that could isolate a strong visual detector plus post-processing from the MLLM-agent approach.Source: Limitations · p. 10
  • Targeted revisits increase tool calls and token cost on harder sheets, but the paper does not report wall-clock or token-cost comparisons.Source: Limitations · p. 10

Ethical considerations

The paper describes data handling and engineer oversight, uncertainty disclosure, and reproducibility considerations, alongside dataset statistics and an OCR reading protocol.

Ethical considerations

  • The 300 evaluation sheets came from partner-authorized archives and were stripped of project, client, designer, and address identifiers before processing.Source: Ethical considerations · p. 10
  • BlueprintAgent supports engineers’ modeling and review workflows but does not replace qualified-engineer judgment; safety-critical applications require engineer-of-record review and sign-off.Source: Ethical considerations · p. 10
  • The output’s confidence, status, evidence_refs, and unresolved_items fields are intended to reduce false confidence and should be displayed by downstream systems.Source: Ethical considerations · p. 10
  • Commercial MLLM API model versions and service-side policy affect reproducibility; supplementary materials provide configurations and traces for re-evaluation on other backbones.Source: Ethical considerations · p. 10
  • The authors used LLMs for phrasing, code drafting, and literature screening, while independently verifying technical claims, experimental numbers, and citations.Source: Ethical considerations · p. 10

A Dataset statistics

  • Table 4 summarizes dataset statistics by difficulty tier for 300 sheets and 102 floors from 20 projects.Source: Table 4 · p. 13

B OCR multi-scale protocol

  • The OCR subsystem supplies multi-scale candidates with confidence scores, and the MLLM selects readings using axis-grid, annotation, and constraint context.Source: B OCR multi-scale protocol · p. 13

C Metric definitions

The metrics define matching criteria for axes, columns, beams, and KL text, plus schema validity and structured-model completeness.

C Metric definitions

D Per-tier results

The section describes per-tier Beam and Column F1 reporting, a terminology table, feature comparisons with prior agents, and representative cases organized by failure and revisit.

D Per-tier results

E Terminology

F Feature comparison with prior agents

  • Table 8 compares BPA with prior agents by trigger source, signal type, revisit granularity, and evidence-grounded revisions.Source: Table 8 · p. 14

G Representative cases

H Adjudicated consensus error profile

The section describes an adjudicated four-category profile of residual B5 errors and a BPA main loop that proceeds through five stages with repeated conflict assessment and revisits.

H Adjudicated consensus error profile

I BPA main loop pseudocode

  • The BPA main loop walks through five stages, repeating a seven-step assess-to-revisit cycle until conflicts are resolved or the stage budget is exhausted.The stages are axes, columns, beams, validation, and 3D-check.Source: I BPA main loop pseudocode · p. 14
  • The loop takes a project, MLLM backbone, and constraints as input and is intended to ensure a simulatable 3D model.The listed constraints are axis, geometry, annotation, and 3D constraints.Source: I BPA main loop pseudocode · p. 14

Related papers

Loading 2609.07362v1…