PaperlayerView exact revision on arXiv

Source-linked AI summary

Handoff-H1: An Orchestrated Vision-Agent System for Material Quantity Takeoff from Construction Blueprints

Bruno Chicelli, Henrique Alves, Rodrigo Anselmo, Joshua Weinberg, Felipe Lemos, Jan Baryla

arXiv:2608.15032v1cs.CLcs.AIcs.CV

TL;DR

Material quantity takeoff from architectural blueprints demands cross-sheet visual perception, dimensional reasoning, and unstated construction knowledge. Handoff-H1 combines specialized vision models, tool-using agents, and a persistent hierarchical project foundation, reaching an 81.6% composite from raw PDFs—roughly 20 points above the strongest frontier agent and above independent professional estimators.

  • Problem

    Material takeoff remains difficult because it requires cross-sheet dimensional extraction, multi-hop reasoning, visual measurement, and construction conventions absent from drawings.

  • Method

    Handoff-H1 combines specialized perception models, a persistent hierarchically structured project foundation, and orchestrated vision agents that audit takeoff outputs.

  • Results

    81.6% composite performance (86.1% coverage, 78.8% P@.25) exceeded the strongest of seven frontier and open-weight agents by roughly 20 points and surpassed independent professional estimators’ 77.6%.

  • Takeaways & Limitations

    Handoff-H1 produced a more thorough takeoff than independent professional estimators by covering more items and trades, despite lower quantity precision.

  • Takeaways & Limitations

    Remaining errors concentrate on quantities, with 78.8% P@.25 versus 87.9% for human estimators, especially in continuous-measurement trades.

Abstract

from arXiv · show

Converting a set of architectural blueprints into a complete material quantity takeoff requires visual perception across drawing sheets, dimensional and multi-hop reasoning, and grounding in construction conventions that the drawings never state. We present Handoff-H1, a takeoff system built from three layers: purpose-built computer-vision models that extract primitives; tool-using agents equipped with image operations and in-house visual-task tools, including CV-model-backed counting, detection and plan decomposition; and a persistent, hierarchically structured project foundation, grounded in a curated construction knowledge base. We evaluate on the Construction Blueprint Takeoff Benchmark: 10 real residential blueprint sets paired with consensus-validated expert takeoffs - 2,009 verified line items, restricted for scoring to the 1,348 primary-tier materials that drive an estimate - scored per trade by an LLM judge on material coverage and quantity Precision@25% (P@.25) and combined into a weighted composite. Under identical scoring from the raw PDF, seven frontier and open-weight models span composites of 35-61, and independent professional estimators - scored against the same reconciled gold standard - post 77.6% (65.5% coverage, 87.9% P@.25). Handoff-H1, working end-to-end from the raw PDF, reaches 81.6% (86.1% coverage, 78.8% P@.25): roughly 20 points above the strongest frontier agent, and above the independent estimators by pairing near-human quantity precision with coverage they do not reach. The evaluation harness is public for the open harbor framework; the blueprint sets and ground truth are available upon request for research use.

1 Introduction

Material quantity takeoff from construction blueprints requires cross-sheet dimensional extraction, multi-hop reasoning, and trade-specific construction knowledge that general-purpose systems lack. Handoff-H1 addresses this gap with purpose-built vision models, tool-using vision agents, and a persistent hierarchical project foundation, achieving an 81.6% composite versus 35–61% for seven other models and 77.6% for independent estimators.

  • Motivation: Construction takeoff requires cross-sheet dimensional extraction, multi-hop reasoning, and grounding in unstated construction conventions.A single material-quantity error can cause tens of thousands of dollars in overruns on a modest residential project.
  • Motivation: General-purpose systems produce fluent but dimensionally ungrounded estimates, while flat blueprint retrieval mixes trade contexts and disrupts estimator-like coherence.Observed failures include hallucinated quantities, confused units, and missing trade-specific logic.
  • System contribution: Handoff-H1 combines purpose-built construction-drawing vision models, tool-using vision agents, and a persistent hierarchical project foundation grounded in curated construction knowledge.The architecture is organized around perception, knowledge, and orchestration layers.
  • Evaluation: 81.6% composite: Handoff-H1 achieves 86.1% material coverage and 78.8% Precision@25% from raw PDFs under the shared evaluation harness.The benchmark uses consensus-validated expert takeoffs and a per-trade LLM judge.
  • Evaluation: 35–61% composites: seven frontier and open-weight models run as tool-using agents under the same harness, versus 77.6% for independent professional estimators.These comparisons use the same raw PDFs and reconciled evaluation standard.

2 Related work

Prior work spans modular retrieval and hierarchical knowledge, tool-using and self-refining agents, construction-drawing recognition datasets, and document assistants. These datasets generally support recognition rather than material estimation, while benchmark designs balance reproducibility against contamination risks.

  • Retrieval-augmented generation and hierarchical knowledge: Modular RAG inserts a structured synthesis layer between retrieval and generation, while hierarchical architectures organize retrieved knowledge into local, bridge, and global layers.Handoff-H1 is positioned within the Modular RAG family and draws on hierarchical knowledge organization.
  • Agentic systems and per-component optimization: Tool-using agents and self-refinement architectures are standard production-LLM components, and Handoff-H1 adopts both to handle conflicting information.The two patterns combine reasoning and acting with critique and revision.
  • Construction-drawing datasets and the takeoff ground-truth gap: Public architectural-drawing datasets target recognition: CubiCasa5K has 5,000 floor plans, FloorPlanCAD roughly 16,000 vector drawings, ArchCAD-400K hundreds of thousands, and ResPlan 17,000 residential plans.Their tasks include floor-plan annotation, vector panoptic symbol spotting, and graph-based residential-plan representation.
  • Construction-drawing datasets and the takeoff ground-truth gap: None of these public datasets carries the material quantities required for construction takeoff.This creates a ground-truth gap between drawing recognition and estimation.
  • Benchmark access models: Open benchmarks improve reproducibility but risk training-corpus contamination, motivating private holdouts, private evaluation splits, and credentialed-access datasets.The cited examples include FrontierMath, ARC benchmarks, and MIMIC.

3 The Handoff-H1 system

Handoff-H1 combines perception, knowledge, and orchestration layers over general-purpose base models to recover drawing structure, ground estimates in persistent project knowledge, and audit agent outputs. Vision agents use generic image operations and specialized construction tools within scoped foundation queries, producing structured takeoff line items.

  • System architecture: Three layers address distinct failure modes: perception recovers visual structure, the knowledge layer organizes it persistently, and orchestration runs and audits vision agents.The layers operate over general-purpose base models and include a curated construction knowledge base.
  • Perception layer: A dedicated perception pass extracts typed rooms, building elements, dimensions, scales, plan areas, and drawing callouts from every sheet.Construction-specialized computer-vision and vision–language models perform the extraction and label detected elements with provenance.
  • Orchestration layer: Vision agents combine generic cropping, zooming, and panning with specialized tools for framing decomposition and regional counting, detection, and measurement.Agents decide when to rely on their own drawing interpretation and when to delegate to purpose-built computer-vision models.
  • Knowledge layer: The project foundation is built once per project as a persistent hierarchy that retrieves perception outputs, partitions them into construction clusters, and grounds them with curated domain knowledge.The curated knowledge base contains construction references, estimating guides, material catalogs, pricing data, assembly conventions, default specifications, and code-driven defaults.
  • Verification: Scoped agents query the foundation and visual tools, while independent verification audits the assembled takeoff against source drawings before finalization.The resulting output is a structured set of primary-material line items, and verification catches missed items and implausible quantities.

4 The Construction Blueprint Takeoff Benchmark

TAKEOFFBENCH-V1 evaluates blueprint takeoff systems on real residential drawing packages using reconciled expert ground truth and item-level scoring. Its design emphasizes verifiable primary materials, shared raw-PDF inputs, public evaluation infrastructure, and controlled access to data.

  • Dataset: 10 residential blueprint sets from real projects include complete drawing packages spanning floor plans, foundations, elevations, wall sections, and roofs.Each set was de-identified through automated detection followed by human review.
  • Ground truth: Primary materials are included while secondary items and labor lines are excluded to keep the gold standard verifiable and comparisons fair.The benchmark focuses on framing members, sheets, and structural quantities that dominate cost; secondary quantities are considered preference-driven and difficult to reconcile to one number.
  • Dataset: 2,009 verified line items comprise the reconciled benchmark ground truth, with 1,348 primary-tier items in scoring scope.Evaluation is item-level: every primary item is treated as its own find, measure, and specify problem.
  • Ground truth: Independent estimators with at least five years of experience produced complete takeoffs, which construction experts reviewed in two stages before reconciliation.The paper frames multi-reviewer reconciliation as necessary because a single estimator’s opinion is not a defensible gold standard.
  • Scoring: LLM judging matches predicted and ground-truth material items within each trade using many-to-many groupings, while excluding labor and non-material rows.Reported results are not based on a single stochastic judge sample.
  • Scoring: Precision and coverage are combined multiplicatively, with precision weighted more heavily and quantities evaluated at a 25% tolerance across collapsed count, linear, area, and volumetric types.The 25% tolerance is intended to avoid penalizing ordinary measurement variance, while collapsing measurement types simplifies comparison but can overpenalize counts.
  • Access and limitations: The evaluation harness is public under Apache-2.0, while blueprint sets and ground-truth takeoffs are available upon request to reduce memorization risk.The public framework represents each task as a blueprint–trade pair with separate prediction and verification steps.
  • Access and limitations: LLM judging tolerates labeling differences but cannot replace human adjudication for genuinely ambiguous rows, and repeated runs address sampling variance rather than systematic judge bias.The benchmark’s interpretation is therefore bounded by both judge limitations and the use of a single reconciled quantity for Precision@25%.

5 Results

Across 10 benchmark sets, Handoff-H1 substantially outperforms frontier agents and independent estimators overall, with its advantage concentrated in derivation-heavy trades. Frontier systems generally locate more items than they quantify accurately, while Handoff-H1 leads on complex scope-derivation tasks.

  • 5.1 Main comparison: 34.7–61.4 composites place all seven frontier agents below the independent-estimator reference of 77.6.claude-fable-5 is strongest at 61.4, followed by gpt-5.6 at 58.6; Kimi-K2.6 reaches 54.5 versus claude-opus-4-8 at 54.6.
  • 5.1 Main comparison: 81.6 composite gives Handoff-H1 a roughly 20-point lead over the best frontier agent, claude-fable-5 at 61.4, from identical raw-PDF inputs.The comparison attributes the gap to Handoff-H1’s perception, knowledge, and orchestration layers rather than a stronger base model.
  • 5.1 Main comparison: 77.3% coverage versus 52.8% P@.25 for claude-fable-5 shows the frontier field consistently finds more primary items than it quantifies within the 25% band.The reported failure profile identifies dimensional reasoning as the principal bottleneck after locating takeoff scope.
  • 5.2 Estimator comparison: 6 of 10 sets favor Handoff-H1 on composite, while estimators lead on four, including one half-point wash.Estimator losses are attributed to missed primary items, especially Structural Framing members and detail-only drywall and rebar quantities.
  • 5.3 Per-trade analysis: 94, 89, and 90 are Plumbing composites for Handoff-H1, estimators, and claude-fable-5, while Windows & Doors gives estimators 89 versus Handoff-H1 88.Counting-like trades show small margins because fixtures are enumerable; Windows & Doors follows the same pattern.
  • 5.3 Per-trade analysis: 23, 14, 11, and 10 points are Handoff-H1’s four largest margins over estimators in Drywall, Insulation, Framing, and Roofing, respectively.These trades require deriving quantities and specifications from areas, pitches, and multi-sheet reasoning rather than reading single-plan callouts.

6 Failure modes and limitations

Handoff-H1’s main residual weakness is quantity precision, especially for continuous-measurement trades whose geometry is ambiguous or hidden in plan views. Its knowledge base and evaluation scope also limit generalization as codes, standards, pricing, regions, and project types change.

  • Quantity precision: 86.1% coverage and 78.8% P@.25 leave more than one matched quantity in five outside the 25% band, versus 87.9% for human estimators.Remaining errors concentrate on quantities rather than omissions or specification-level errors.
  • Continuous-measurement trades: Roofing quantities remain difficult because billed area depends on pitch and framing information hidden by the plan view, making disagreement with a single gold number legitimate.Roofing is sold in squares, and the true area cannot be read directly from the flat footprint.
  • Continuous-measurement trades: Continuous-measurement trades such as framing, siding, and roofing show higher error rates than discrete-count trades in Handoff-H1 and every baseline.Wall-length inference requires resolving whether dimensions include or exclude adjacent wall thickness, an ambiguity human estimators also debate.
  • Knowledge-base maintenance: The curated construction knowledge base requires ongoing maintenance as building codes evolve, material standards change, and pricing data ages, with regional practice variation posing an added challenge.These maintenance demands are a limitation of the system’s construction knowledge grounding.
  • Evaluation scope: Results are limited to residential blueprints, while commercial construction, multistory buildings, and partial or inconsistent drawing sets remain unevaluated.Those project types may behave differently.

7 Discussion and next steps

Handoff-H1 reaches human-level benchmark performance and exceeds professional estimators on composite score by covering more items, while retaining a 20-point advantage over the strongest frontier agent. The authors’ next steps are to update model comparisons, expand evaluation and estimator coverage, and support external research access.

  • Human comparison: 81.6 versus 77.6 composite: Handoff-H1 surpasses professional estimators while covering roughly a third more matched items.The estimator’s 87.9% precision is computed only over items and specifications accurately detected and named.
  • Human comparison: 78.8% precision is earned over roughly a third more matched items than estimators’ 87.9%.The discussion characterizes broader coverage as the harder road because every additional quantified item is scored.
  • The gap against frontier models: 20-point gap: Handoff-H1 remains ahead of the strongest frontier agent when all systems start from the raw PDF.The authors attribute the difference to combining on-demand perception, estimator-aligned knowledge partitioning, and scoped, verified trade reasoning.
  • Next steps: The next steps are to track new frontier models, evaluate beyond residential construction, broaden the estimator pool, and welcome benchmark and collaboration requests.Requests include benchmark access, leaderboard submissions, research partnerships, and individual collaborations.

8 Conclusion

Handoff-H1 is presented as a generally available system for material quantity takeoff from construction blueprints, alongside a benchmark for measuring it. Its design combines purpose-built vision models, task-specific visual tools, a persistent knowledge-grounded project foundation, and orchestrated estimation with independent verification.

  • Handoff-H1 provides material quantity takeoff from construction blueprints and is accompanied by a benchmark for evaluating performance.
  • The system follows a combination-over-silver-bullet design commitment rather than relying on a single capability.
  • Its architecture combines purpose-built vision models, task-specific visual tools, a persistent partitioned project foundation, a curated knowledge base, and independent verification.

A Running the benchmark with harbor

The benchmark runs on a public Apache-2.0 harbor harness, while blueprint sets and ground truth require approved access. Evaluation proceeds per (blueprint, trade) task, with agents producing line items that a verifier grades against held-out ground truth.

  • A Running the benchmark with harbor: The evaluation harness is public under Apache-2.0, but blueprint sets and ground truth require an approved access request.The harness is available through the public repository, whereas requested data are placed under tasks/ after access is granted.
  • A Running the benchmark with harbor: Each task pairs one blueprint with one trade, and an agent’s predicted line items are graded against held-out ground truth by a verifier.The raw-PDF track can be run with harbor run using a case, trade, agent, and model specification.
  • A Running the benchmark with harbor: Per-task rewards are aggregated into the leaderboard with harbor report.The documented workflow includes cloning the public harness before running and reporting benchmark tasks.
Loading 2608.15032v1…