PaperlayerView exact revision on arXiv

Source-linked AI summary

Structural Plan-to-Model Conversion with Deterministic Geometry and Guarded Agentic Vision-Language Refinement

Mohammad Talebi-Kalaleh, Qipei Mei

arXiv:2608.17237v1cs.CVcs.AI

TL;DR

Translating structural framing plans into editable finite-element drafts is labor-intensive, while existing drawing systems and structural language-model agents have limited coverage of the drawing itself. This paper combines deterministic geometry with guarded vision-language refinement and reports strong held-out detection results across five structural entity classes.

  • Problem

    Structural-plan conversion to editable finite-element drafts remains labor-intensive, while existing systems do not combine drawing-based structural detection with language-model refinement.

  • Method

    The framework combines deterministic PDF geometry extraction, scale consensus, drafting-grammar recognition, topology-based model assembly, and guarded typed vision-language corrections.

  • Results

    On the held-out half, walls and braces achieved 1.000 recall/precision, openings 1.000/0.964, and columns/beams 0.922/0.997 and 0.886/0.990.

  • Takeaways & Limitations

    The framework produces editable model drafts with an inspectable geometric core while constraining agentic actions through guarded review and fail-closed transactions.

  • Takeaways & Limitations

    Evaluation used vector PDFs from a shared generator and did not validate independently drafted plans, analytical connectivity, or solver validity.

Abstract

from arXiv · show

Converting structural framing plans into editable finite-element model drafts remains labor-intensive and prone to transcription error. Existing drawing-understanding systems for building components rely on task-specific trained neural detectors, and language-model agents in structural engineering operate on text or model data rather than the drawing itself. This paper presents, to the authors' knowledge, the first framework applying an agentic vision-language layer to structural component detection and model drafting from framing-plan PDFs, without task-specific detector training or fine-tuning. A deterministic stage extracts primitives, estimates scale by dimension-ratio consensus, recognizes five entity classes with a drafting grammar, and assembles an editable layout. The agentic stage proposes typed corrections constrained by deterministic candidates, operation-specific admission tests, change-level review, and fail-closed transactions. Evaluation used an author-generated benchmark of 100 plans: a development half that informed every rule revision, and a seed-disjoint held-out half generated after the rules froze, evaluated once. All reported scores are end-to-end results of the complete framework on the held-out half. Scale was estimated within 0.1% of the generator reference for every drawing. Recall and precision were 0.922/0.997 for columns, 0.886/0.990 for beams, 1.000/1.000 for walls, 1.000/1.000 for braces, and 1.000/0.964 for openings. A controlled study repeated two corruptions three times on three development drawings. Calibration passed all nine trials; member repair met every strict end-state predicate in five of nine. Guarded review corrected missed framing and false marks within explicit bounds. The held-out half shares the development generator, so the study excludes independently drafted plans, raster evaluation, analytical connectivity, and solver validation.

1 Introduction

The paper addresses the labor-intensive translation of structural framing plans into finite-element model drafts by placing deterministic geometric processing before guarded vision-language refinement. It contributes a deterministic PDF-to-model pipeline and, to the authors’ knowledge, the first agentic vision-language architecture for structural component detection and model drafting from drawings.

  • Motivation: Manual conversion requires reading coordinates, redrawing members, and re-entering section labels, and prior work found no complete automatic reconstruction pipeline.Even existing building information models require dedicated tooling to derive geometrically faithful analysis models.
  • Contributions: The second contribution is, to the authors’ knowledge, the first agentic vision-language architecture for building-component detection and structural model drafting from drawings.The claim is positioned against prior drawing-understanding work focused on architectural semantics and image- or primitive-space scores.
  • Framework: The framework assigns explicit quantities to deterministic geometry while using typed vision-language operations for semantic refinement.The deterministic layer covers primitive extraction, scale consensus, symbol signatures, bearing topology, and region analysis.

2 Related work

Prior drawing-understanding research spans rule-based vector interpretation, learned raster and vector recognition, scan-to-BIM, and language-model agents for structural workflows. The present work differs by targeting structural entities in model units and combining explicit heuristic detection with an agentic vision-language layer applied to drawing images.

  • Classical drawing understanding: Early architectural systems used hand-built rules over vector primitives for vectorization, symbol recognition, cross-floor matching, wall detection, and annotation separation.Later statistical methods extended recognition across notation and drawing styles.
  • Learned drawing understanding: Deep-learning systems recast drawing understanding as pixel prediction or vector-primitive recognition, including floor-plan recovery, multi-task segmentation, and panoptic symbol spotting.FloorPlanCAD contributed more than ten thousand annotated CAD drawings for primitive-level recognition.
  • Structural-design automation: Architectural layout studies support design reuse through learned wall representations, but their outputs remain pixel-level architectural information rather than structural model entities.The cited work includes limited-label wall segmentation and transformer-based retrieval of similar wood-frame layouts.
  • Agentic structural workflows: Language-model systems in construction and structural engineering generate scripts, coordinate workflows, or produce BIM and building-model outputs from textual briefs, scripts, or models.The paper’s agent architecture draws on interleaved reasoning and acting, tool delegation, iterative self-refinement, model-based judging, and process supervision.
  • Positioning of the present work: To the authors’ knowledge, no prior reviewed system applies an agentic vision-language layer to building-component detection from drawing images without relying on a fitted detector.The present work instead targets structural entities, constructs bearing relations after detection, resolves scale explicitly, and expresses coordinates in model units.

3 Proposed plan-to-model method · 3.1 Division of responsibilities

The method assigns geometric measurement and exhaustive detection to deterministic procedures, while a vision-language model interprets symbols and proposes only guarded, typed operations. This division addresses coordinate and geometric-reasoning limitations while retaining flexibility for annotations beyond fixed parsing rules.

  • 3.1 Division of responsibilities: The vision-language model interprets symbols, hatched bands, brace signs, and designations such as W14×90 without task-specific fine-tuning.Its role is semantic interpretation rather than sole geometric extraction.
  • 3.1 Division of responsibilities: At 1:100 scale, a 0.3 m model-space displacement occupies 3 mm on the printed page, making image-based coordinate estimation unreliable from downsampled prompt images.Coordinate proposals from images are estimates rather than measurements from vector geometry.
  • 3.1 Division of responsibilities: Four tested models averaged 58.07% across a seven-task synthetic benchmark for low-level geometric reasoning involving intersection, collinearity, and parallel-line counts.These results motivate restricting geometric reasoning authority.
  • 3.1 Division of responsibilities: Deterministic geometry measures coordinates, resolves scale, and applies named rules to candidate entities, rather than delegating metric extraction to the vision-language model.The architecture uses deterministic procedures for repeatable geometric operations and exhaustive entity detection.
  • 3.1 Division of responsibilities: Deterministic repeatability does not prevent misses or false detections when office conventions fall outside the encoded grammar, and unmodeled symbols can leave meaningful ink unclaimed.Annotations introduce an open vocabulary that fixed parsing rules cannot fully cover.
  • 3.1 Division of responsibilities: The architecture gives the vision-language model access only through a closed schema of typed operations, while geometry supplies coordinates, repair candidates, and the initial entity set.This explicitly enforces the division of responsibilities between semantic interpretation and geometric processing.
  • 3.1 Division of responsibilities: Additions require supporting axis or glyph ink, while other operations use class-specific caps, range checks, and judging because semantic changes lack one universal geometric predicate.These operation-specific admission tests constrain agentic proposals.

3.2 System architecture

The system converts floor-plan PDFs into editable finite-element model drafts through deterministic extraction, guarded agentic refinement, and three-dimensional model building. Explicit geometric procedures establish the initial layout, while language-model outputs remain proposals subject to checks.

  • Architecture: Three stages convert a floor-plan PDF into an editable finite-element model draft: deterministic extraction, guarded agentic refinement, and three-dimensional model building.The input may be a vector export from CAD or BIM software or a scanned drawing.
  • Deterministic stage: Deterministic extraction produces a typed two-dimensional model-space layout containing grids, columns, beams, walls, braces, slab regions, and openings.The implementation applies named rules in a fixed order, and the resulting layout remains editable.
  • Model builder: The model builder uses refined layouts and user-supplied story heights to place members, mesh slabs around openings, assign properties, and apply restraints and releases.Parsed designations determine sections and materials when available; configured defaults fill unresolved properties.
  • Design principles: Scale, collinearity, coverage, bearing, and containment are computed by explicit procedures rather than a task-specific detector, although their thresholds remain design choices.The language model is consulted only after the initial layout exists, and its answers remain proposals until they pass available checks.

3.3 Deterministic detection layer

The deterministic detection layer converts PDF drawings into editable structural-model geometry through primitive extraction, scale resolution, ordered entity detection, topology construction, and cross-drawing reconciliation. Its rule-based grammar uses explicit geometric signatures and constrained tolerances rather than learned appearance.

  • Pipeline: Three dependent phases extract primitives and scale, detect structural entities and bearing topology, then reconcile drawings before assembling an editable three-dimensional draft.The pipeline proceeds from drawing evidence to model geometry through shared intermediate representations and cross-drawing alignment.
  • Primitive extraction: Vector and image-only PDFs feed common geometric detectors, while vector parsing preserves attributed segments, polygons, arcs, positioned text, endpoints, and line weights.The vector path traverses PDF graphics operators and recovers arcs from flattened Bézier runs; image inputs are traced into stroke primitives.
  • Scale resolution: 1.5% is the configured reference-scale snapping tolerance after dimension-consensus estimates are formed from surviving dimension candidates.Each candidate yields si = Di/di; outliers are rejected using median absolute deviation, and confidence is the inlier fraction |I|/n.
  • Structural detection: Grid recovery, column detection, member classification, and topology construction occur in order so annotations and structural symbols are not chained into members.Topology chains fragments across gaps and seeks bearing endpoints against columns, walls, and beams using explicit geometric predicates.
  • Grammar and reconciliation: Eleven explicit grammar rules encode columns, beams, walls, framed openings, and braces through geometric signatures, while shared grid labels register related drawings.The column detector derives locations, footprint dimensions, and orientations; symbols are distinguished from member linework without assigning material from linework alone.

3.4 Guarded agentic refinement

Guarded agentic refinement uses vision-language review to propose typed corrections for residual drawing inconsistencies, while deterministic candidates, operation-specific validation, judging, and fail-closed transactions constrain edits. The workflow separates destructive and constructive reviews, with page-level metadata handled outside typed-operation admission and candidate-label bias identified as a limitation.

  • Refinement purpose: Agentic refinement reviews residual inconsistencies by proposing typed edits from geometry-derived candidates and evaluating resulting changes through a separate review call.The target inconsistencies include missed members, false detections, displaced column marks, and assumed scale.
  • Guarded transaction: Typed operations are admitted only after required-field and operation-specific evidence checks, with accepted and rejected edits applied to a layout copy.The validation routine is the sole procedure permitted to apply layout edits.
  • Guarded transaction: Fail-closed entity editing returns the original layout when no candidate survives or when neither the complete batch nor a re-judged subset is accepted.This guarantee does not cover separately reported page-level metadata.
  • Review decomposition: Two focused review calls separate false-mark deletions or column moves from missed-structure additions, attribute assignments, and scale calibration before shared validation.The normal workflow uses two proposal calls and one judging call; retries and subset review can raise this to six logical model invocations.
  • Candidate generation: Five deterministic generators seed repair candidates, including calibration, inkless-mark removal, unmarked columns, and missing members, under explicit geometric thresholds.Calibration requires at least three dimension-derived factors, with 60% within 2% of their median; unmarked spans require at least 0.60 axis coverage, while inkless marks fall below 0.25 axis coverage.
  • Limitations: Candidate labels encode geometric stroke evidence rather than structural proof, creating a prompt-bias limitation that motivates provenance-blinded ablation testing.The concern is that emphatic candidate wording may bias both the proposing model and its judge toward the seeded class.

4 Benchmark and evaluation

The evaluation uses a 100-plan procedural benchmark split into development and held-out halves, with end-to-end held-out processing after all revisions were frozen. Its controlled representation supports reproducible geometry tests but may overestimate performance relative to independently drafted drawings.

  • Benchmark design: 100 procedurally generated single-story framing plans are divided into equal PD-50 development and PD-50-T held-out halves.The benchmark provides vector PDFs, raster variants, ground truth in model-space meters, and world-to-sheet mappings.
  • Benchmark design: 1,082 columns, 2,721 beams, 194 wall panels, 47 braces, and 80 openings comprise the PD-50-T ground truth.The corresponding PD-50 totals are 1,100 columns, 2,819 beams, 189 wall panels, 44 braces, and 80 openings.
  • Limitations: Held-out results may remain optimistic because the generator and extractor share drafting assumptions, despite controlling tuning to specific drawings.The paper identifies independently drafted drawings as necessary for assessing external generalization.
  • Evaluation protocol: All rule, threshold, prompt, and guard revisions were completed on PD-50 before PD-50-T was generated and processed exactly once per condition.Held-out failures were not used to change the system, and each case passed through the complete framework from vector PDF to intermediate layout.
  • Evaluation protocol: Class-level recall and precision are primary outcomes, supplemented by a per-drawing regression gate with explicit class-specific recall thresholds.The gate requires column recall and precision ≥0.95, beam recall ≥0.85, wall recall ≥0.75, and opening recall ≥0.50; it excludes braces and constrains no listed precision except columns.

5 Results

On the held-out half, the complete framework achieved strong end-to-end detection results, with perfect wall and brace scores and lower recall for columns and beams. Guarded refinement operated conservatively, while controlled corruption trials showed reliable calibration and partial member repair.

  • Held-out end-to-end results: 0.086% maximum relative scale error was achieved on all 50 held-out drawings, with dimension-consensus confidence 1.0.The error was measured against the generator reference scale.
  • Held-out end-to-end results: 1.000 recall and precision were achieved for walls over 194 panels and braces over 47 members; openings reached 1.000 recall and 0.964 precision.Columns scored 0.922 recall and 0.997 precision, while beam recall was lower than the perfect wall and brace results.
  • Family variation: 0.659 and 0.790 column recall occurred in the mixed and skew-wing families, whose rotated wings accounted for nearly the whole column-recall deficit.Atrium contributed the remaining four column misses, with recall of 0.969.
  • Guarded refinement: 43 of 50 guarded transactions were accepted and 338 typed operations were applied; seven transactions failed closed without changing intermediate layouts.The applied operations comprised 218 deletions, 70 member additions, and 50 section-designation assignments.
  • Controlled corruption study: 14 of 18 controlled scenario executions passed strictly, all nine calibration repetitions passed, and member repair met every strict predicate in five of nine repetitions.The largest post-calibration primary-span relative error was 0.00182%, and all 2,073 full-layout scale-restoration checks passed.
  • Controlled corruption study: 36 of 36 fabricated marks were removed and 9 of 9 displaced columns were corrected, but none of 6 deleted beams were restored.The refiner restored six of nine deleted columns and six of six deleted walls, with no collateral deletions or unexpected entities observed.

6 Conclusions

The study presents a hybrid, training-free workflow that converts structural floor-plan PDFs into editable finite-element model drafts using deterministic geometry and guarded vision-language refinement. Held-out results were strong on the author-generated benchmark, but transfer, raster evaluation, analytical connectivity, and solver validation remain untested.

  • Contributions: The workflow combines deterministic primitive extraction, dimension-consensus scale estimation, drafting-grammar entity recognition, geometry assembly, and agentic vision-language refinement.No task-specific detector was trained or fine-tuned on plan annotations, although refinement uses a pretrained externally hosted vision-language model and manually designed rules.
  • Held-out evaluation: 0.1% scale error was achieved for every held-out drawing; column recall/precision were 0.922/0.997 and beam recall/precision were 0.886/0.990.The seed-disjoint held-out half was generated after rules froze and evaluated once; walls, braces, and openings achieved recall of 1.000.
  • Controlled study: 9/9 calibration repetitions passed, with maximum primary-span error of 0.00182% and no failures among 2,073 full-layout restoration checks.These controlled-corruption results characterize calibration performance under the tested repetitions.
  • Controlled study: 5/9 member-repair repetitions met every strict predicate: 36 fabricated marks were removed, nine displaced columns were corrected, and six deleted walls were restored, but zero of six deleted beams were recovered.The controlled study therefore demonstrated bounded correction while exposing missed-beam recovery as a failure mode.
  • Limitations: The benchmark may overstate foreign-producer performance because its generator and detector share representational assumptions, and the quantitative study is limited to vector PDFs.Encoded United States and Canadian variants show balance within the generator rather than transfer across design offices or regional practices; raster and model-assembly examples remain illustrative.
  • Limitations: Every generated draft requires engineering review before analysis, with future evaluation needed on independently drafted plans, additional conditions, repeatability, ablations, and operation-level error rates.The paper recommends extending the frozen-rule held-out evaluation beyond the author-generated benchmark.

Data availability

The benchmark release contains all 100 vector drawings across development and held-out halves, with clean and degraded rasters, exact ground truth, mappings, and an evaluation protocol. The archive is planned for public repository deposition upon acceptance, with its persistent identifier added to the published version.

  • Benchmark contents: 100 vector drawings span the PD-50 development and PD-50-T held-out halves, each with clean rasters, degraded variants, and exact ground truth.The release also includes world-to-sheet-to-pixel mappings.
  • Evaluation materials: The suggested evaluation protocol preserves separation between development and held-out test data.
  • Release plan: The benchmark archive will be deposited in a public repository upon acceptance, with its persistent identifier added to the published version.

A Case inventory and per-case recall counts

Table 9 inventories every held-out PD-50-T case and reports per-class end-to-end recall as matched detections over ground-truth counts. The inventory records family, notation variant, material, and drawing scale, with held-out variants 06–10 paralleling development variants 01–05.

  • Case inventory: Table 9 lists every held-out PD-50-T case with family, notation variant, material, drawing scale, and per-class end-to-end recall counts.Recall counts are reported as matched detections over the ground-truth count.
  • Case inventory: Held-out variant numbers run from 06 to 10 within each family, matching the structure of development variants 01 to 05.Per-case development and deterministic-arm records are included in the data release.
  • Case inventory: Drawing scales are abbreviated using notation such as 1/4′′ = 1′-0′′ (1:48) and 3/16′′.The passage introduces these abbreviations as part of the inventory conventions.

B Detection overlays for the held-out half

Figures 6 and 7 overlay the final end-to-end layout for every held-out PD-50-T case on its source drawing, while distinguishing detected counts from matched counts. Panel titles mark detected column-count discrepancies in amber, and the overlays show the discussed errors.

  • Held-out overlays: Figures 6 and 7 show the final end-to-end layout for every held-out PD-50-T case over its source drawing, cropped to the structure.The overlays depict the complete held-out set rather than selected examples.
  • Count interpretation: Detected counts are compared against ground truth rather than matched instances, so equal counts can conceal a false detection compensating for a miss.Matched counts are reported separately in Table 9.
  • Error visualization: Panel titles are amber when the detected column count differs from the ground-truth count, and the overlays make the Section 5.1 errors visible as unmarked columns.

C Prompt architecture of the agentic layer

The agentic layer uses a color-coded overlay, ordered structural checks, and typed, bounded corrections to refine deterministic markups. Parallel focused calls separate deletion/movement from calibration, additions, attributes, and labels, while a strict judge verifies only listed changes.

  • Overlay and evidence: The overlay legend maps colors and line styles to columns, girders, secondary beams, walls, braces, slabs, openings, and grids, while halos expose missed and false detections.A dark drawing line without a color halo indicates missed structure; a halo over blank paper indicates a false detection.
  • Ordered correction protocol: The reviewer performs checks in order and proposes at most 12 small, high-confidence typed corrections.The sequence covers scale, false marks, missed structure, wrong placement, attributes, and section labels.
  • Guarded operation admission: Candidate additions require a lower bayJoistCounts value and one image-verified unhaloed structural line, while scale calibration compares JSON grid distances with printed dimensions.The prompts prohibit adding more beams than the count difference and require exactly one calibration operation when calibration is necessary.
  • Disjoint focused calls: Call A handles only false marks and misplaced columns, whereas Call B handles scale, missed structure, attributes, printed levels, and section designations.Call A cannot add members or calibrate; Call B cannot delete or move anything, and required calibration returns exactly one calibration operation.
  • Change-level review: The judge verifies each authoritative listed change against the drawing, rejects unsupported or nonstructural additions, and accepts only when every listed change is supported.It distinguishes deletion from movement, checks calibrated scale within 0.1%, and records only wrong change numbers in bad_changes.

D Illustrative raster and model-assembly artifacts

Figure 8 illustrates raster-based processing and model assembly using three scanned architectural plans from a six-story building drawing set. The example applies an assumed 3.2 m story height and specified story mapping, alongside an independent six-story model draft.

  • Raster and model assembly: Three scanned architectural plans from a six-story building drawing set illustrate the raster and model-assembly pathways.Panels (a)–(c) show user-calibrated layouts processed through the raster path.
  • Raster and model assembly: 3.2 m story height was imposed, with the first plan assigned to the lowest level, the second repeated across four intermediate levels, and the third assigned to the top level.This story mapping was supplied to the model builder.
  • Raster and model assembly: Panel (d) presents an independent six-story model draft, while panels (a)–(c) show layouts with assumed lattice framing.The figure separates the scanned-plan raster layouts from the independently drafted model artifact.
Loading 2608.17237v1…