PaperlayerView exact revision on arXiv

Source-linked AI summary

PlanSightRAG: A Visual-First Multimodal RAG for Automating Question Answering and Compliance Checking for Civil Standard Plans

Nabaraj Subedi, Shuvo Dip Datta, Ahmed Abdelaty, Shivanand Venkanna Sheshappanavar

arXiv:2608.26091v1cs.IRcs.CLcs.CV

TL;DR

OCR-centric compliance checking loses the layout and geometry needed to interpret dense civil engineering plans. PlanSightRAG retrieves and reasons directly over plan imagery, maintaining over 91% Recall@5 across five DOTs.

  • Problem

    Civil infrastructure compliance checking still depends on engineers reading dense 2D standard plans, while OCR-centric systems discard the layout, geometry, and symbolic cues needed to interpret them.

  • Method

    PlanSightRAG is a visual-first multimodal RAG that retrieves and reasons over plan imagery through patch-level retrieval, an agentic compliance workflow, and MaxSim grounding.

  • Results

    PlanSightRAG delivers robust retrieval, maintaining over 91% Recall@5 across five DOTs.

  • Takeaways & Limitations

    The results identify fine-grained visual reading, entity–dimension binding, and supplying pre-resolved rule thresholds as the remaining compliance bottlenecks.

  • Takeaways & Limitations

    Synthetic CAD compliance tests do not fully replicate real-world ambiguities, and autonomous threshold extraction succeeds on only 33% of numeric items in a real 931-page WYDOT corpus.

Abstract

from arXiv · show

Civil infrastructure compliance checking has long relied on engineers manually reading legacy 2D plans; however, OCR-based automation strips away the geometry and layout essential for interpreting these plans. We present a Visual-First Multimodal Retrieval-Augmented Generation (RAG) framework called PlanSightRAG. It indexes and reasons directly over plan imagery, integrates a ColNomic-3B multi-vector retrieval, an agentic Planner-Retriever-Auditor-Synthesizer, and MaxSim heatmaps as an evidence trail. We introduce a 4,056-pair benchmark from five state Departments of Transportation (DOT) standard plans (1,898 pages). PlanSightRAG achieves 91.47% Recall@5 on zero-shot retrieval, while on a held-out Michigan DOT corpus, it achieves 91.40%. On synthetic, parametrically-generated compliance drawings, our Qwen2.5-VL-72B pipeline reaches 100% verdict accuracy only when supplied a pre-resolved rule threshold, a controlled ceiling that a non-VLM OCR baseline already reaches at 76.4%. Finally, we demonstrate autonomous visual rule-grounding by extracting numeric limits directly from a specification corpus without any human-supplied rules.

1. Introduction · 2. Literature Review

The paper frames civil-plan compliance checking as a geometry- and layout-dependent task poorly served by OCR-centric methods. It positions PlanSightRAG as an integrated visual-first system combining patch-level retrieval, agentic auditing, and autonomous rule grounding across engineering-plan repositories.

  • 1. Introduction: State transportation agencies still rely on engineers to inspect dense 2D standard plans, making multi-sheet compliance review expensive, slow, and error-prone.Legacy DOT sheets remain authoritative references for design review despite their operational burden.
  • 1. Introduction: OCR-based RAG loses geometry, positional layout, and cross-view symbols, preventing reliable retrieval of the primary semantics encoded in engineering plans.Linearizing plans into token bags removes information needed to locate relevant plan content.
  • 1. Introduction: PlanSightRAG indexes plans directly as images and combines ColNomic-3B visual retrieval with an agentic Planner–Retriever–Auditor–Synthesizer compliance workflow.The framework uses high-resolution tiling and sharpened MaxSim heatmaps to provide a transparent evidence trail.
  • 1. Introduction: Recall@5 = 100% among 1,913 candidates enables autonomous retrieval of governing requirements, numeric-limit extraction, and compliance auditing without human-supplied rules.The system resolves symbolic 𝑑/2 and matches the 100% ceiling under a self-grounded threshold.
  • 2.1. Automated Compliance Checking in Civil Infrastructure: Traditional automated compliance methods encode requirements as executable rules or linear text, but legacy plans express them implicitly through geometry, layout, and cross-view symbols.This mismatch leaves manual visual inspection as the dominant current practice.
  • 2.2. OCR-Based Document QA and Text-Centric RAG Systems: Text-centric RAG successfully retrieves explicitly stated regulatory clauses and supports document QA, but these successes do not resolve the visual failure modes of standard plans.The literature therefore provides a textual reference point against which visual-first retrieval is compared.
  • 2.3. VLMs for Drawing Understanding: VLMs offer strong foundations for diagram understanding, yet engineering-drawing performance remains limited by implicit layout relationships, spatial grounding, symbolic interpretation, and multi-step reasoning.Most existing applications focus on single-image interpretation rather than scalable retrieval, evidence grounding, or compliance reasoning across repositories.
  • 2.4. Vision-First Document Retrieval and Multimodal RAG: Vision-first retrieval preserves spatial alignment through multi-vector late interaction, but prior multimodal systems largely remain limited to single-image QA rather than structurally linked compliance reasoning.PlanSightRAG addresses this gap by integrating visual-first retrieval with spatially grounded reasoning over large engineering archives.

3. Methodology

PlanSightRAG is a visual-first multimodal RAG architecture that preserves civil-plan geometry and layout through rasterized visual indexing rather than OCR. Its pipeline combines four phases: ingestion and metadata enrichment, ColNomic-3B multi-vector indexing, MaxSim retrieval with optional reranking or fusion, and visual QA or agentic compliance auditing.

  • Framework Overview: The framework comprises ingestion and preprocessing, ColNomic-3B visual indexing, MaxSim retrieval and grounding, and downstream VQA or agentic compliance auditing.Figure 2 identifies these as the framework’s four phases and includes heatmap grounding for VQA and Planner–Retriever–Auditor compliance auditing.
  • Document Ingestion and Preprocessing: 200 DPI full-page rasterization is used for the deployed index, headline retrieval, and compliance retrieval, while 400 DPI overlapping tiling is evaluated only as an ablation.The corpus contains 1,898 in-distribution pages from five state DOTs, with a separate 298-page Michigan DOT index for zero-shot evaluation.
  • Document Ingestion and Preprocessing: 400 DPI tiling uses 1024 × 1024 tiles, 256-px overlap, and 768 × 768 stride to preserve native-resolution detail in large engineering sheets.A typical approximately 4400 × 3400 landscape sheet produces approximately 30 tiles, each retaining document metadata and spatial coordinates for grounding.
  • Visual Indexing and Retrieval: ColNomic-3B encodes each page or tile as multi-vector patch embeddings on a Qwen2.5-VL backbone, enabling reusable visual representations across retrieval, VQA, and compliance.Pages and tiles are scored individually, supporting sub-page retrieval and spatial back-mapping to the source image.
  • Visual Indexing and Retrieval: MaxSim independently aligns query-token vectors with their most relevant visual patches, ranks pages, and returns Top-K visual evidence.This preserves contributions from localized cues such as dimension values, legend symbols, and table entries that dense single-vector retrieval may average out.
  • Retrieval Enhancements: Optional VLM cross-encoder reranking and BM25 fusion improve discrimination or lexical precision without modifying the visual index.The VLM verifies Top-K candidates sequentially with early stopping after two positive verifications, while BM25 fusion targets alphanumeric queries using normalized metadata scores.

3. Fine-Grained Tile … 4. Preserved Annotations + Grid Position

PlanSightRAG preserves fine-grained plan geometry and annotations through overlapping visual tiles, then grounds VQA and compliance decisions with localized visual evidence. Its agentic compliance workflow decomposes multi-plan queries into auditable steps, while deployment measurements expose the latency cost of sequential reasoning.

  • 3. Fine-Grained Tile: 400 DPI rasterization produces overlapping 1024 × 1024 tiles with 256-pixel overlap, preserving fine-grained annotations and enabling sub-page retrieval.The sliding-window decomposition supports retrieval at finer-than-page granularity.
  • 4. Preserved Annotations + Grid Position: Patch-level late interaction identifies the regions driving retrieval decisions, making ColNomic inherently explainable rather than opaque.This visual attribution supplements retrieval with interpretable regional evidence.
  • 4. Preserved Annotations + Grid Position: Top 5% patch scores are retained and normalized activations are raised to γ= 3.0, concentrating sharpened heatmaps on the strongest page regions.The resulting map is aligned to page layout and bicubic-upscaled to the original resolution.
  • 4. Preserved Annotations + Grid Position: Top-K high-activation patches matched to OCR regions return bounding-box evidence for annotations, dimensions, notes, and graphics across VQA and auditing.The mechanism creates a persistent visual evidence trail for answers and agent steps.
  • 3.5. Visual Question Answering (VQA) Pipeline: Each VQA response pairs a natural-language answer with an evidence gallery of supporting plan sheets for direct visual verification.The prompt supplies retrieved images, lightweight plan anchors, and an optional user diagram to Qwen 2.5-VL.
  • 3.6. Automated Compliance Checking Workflow: Single-pass RAG cannot handle compliance queries spanning multiple plans and regulatory rules, motivating an agentic compliance pipeline.The primary high-accuracy evaluations use Qwen 2.5-VL-72B, while the framework also supports Qwen 2.5-VL-7B for resource-constrained deployment.
  • 3.6. Automated Compliance Checking Workflow: The Planner, Retriever, Auditor, and Synthesizer decompose queries, retrieve evidence, produce preliminary findings, and consolidate final verdicts.Planner steps specify focused searches and verification targets; the Synthesizer includes per-step evidence and ambiguity notes.
  • 3.7. Implementation and Deployment Details: 60.9 s/query is the average agentic compliance latency, versus 2–5 s for single-shot VQA, because planning expands verdicts into ∼8.6 sequential VLM steps.Indexing costs 7.3 min for 1,898 pages, while full-index retrieval takes ≈0.10 s p50; step pruning targets the dominant compliance cost.

4. Evaluation Setup

The evaluation combines a 4,056-pair five-DOT benchmark, a held-out Michigan DOT transfer set, and a synthetic CAD-generated compliance set. It uses a 1,898-page visual index, four reasoning categories, and retrieval and judge-accuracy metrics to assess the framework.

  • Evaluation datasets: The evaluation includes a 4,056-pair benchmark across five state DOTs, a held-out Michigan DOT zero-shot set, and a CAD-generated compliance test set.The benchmark is generated through an iterative refine-loop procedure, while the compliance set provides known ground truth for the agentic verdict pipeline.
  • Evaluation datasets: 1,898 pages form the five-DOT visual index, comprising WYDOT, Caltrans 2025, Arizona DOT 2025, Colorado DOT 2025, and Florida DOT 2026 standard plans.Pages are rendered at 200 DPI and stored as multi-vector records with agency, plan ID, and sheet-title metadata.
  • Benchmark validity: 23.9% of finalized benchmark pairs are ColPali misses, indicating that the refine-loop benchmark retains questions ColPali failed to retrieve after five rephrases.This addresses the potential construct-validity concern that curation could favor ColPali-retrievable phrasings.
  • Evaluation categories: The benchmark covers Dimensional Accuracy, Visual Interpretation, Logical Reasoning, and Hallucination Rate, with the last category primarily functioning as a specification/material-lookup stress set.Because finalized pairs are predominantly answerable, literal “not specified” answers are a minority; abstention is measured separately.
  • Evaluation metrics: Recall@5 measures whether the gold page appears among the top five retrieved results, while Judge Accuracy measures binary end-to-end correctness assigned by a local Qwen2.5-VL-72B judge.Recall@5 evaluates visual retrieval independently of generation, whereas Judge Accuracy includes retrieval and reasoning.

5. Results

PlanSightRAG delivers strong visual retrieval and agentic compliance results across heterogeneous DOT plans, while experiments show that spatially resolved visual representations outperform global, OCR-based, and OCR-free alternatives. Its strongest compliance outcomes occur with tight visual context and self-correction, though multi-plan scaling exposes remaining failure modes.

  • Retriever comparisons: 92.69% Recall@5 for ColNomic-3B exceeds ColPali’s 76.89%, BGE-M3 + OCR’s 36.79%, CLIP’s 1.89%, and OCR-free VLMs’ 0.47–7.31%.Nemotron-ColEmbed-8B reaches 95.28% Recall@5; OCR-free Nougat, Pix2Struct, and UDOP achieve 0.47%, 7.31%, and 0.00% Recall@5, respectively.
  • Efficiency: 65.09% Recall@5 is retained by HPC-ColPali while compressing the visual index 16.0×, from 477.74 MB to 29.86 MB.This is within 11.79 percentage points of full-precision ColPali.
  • Retrieval performance: 91.47% zero-shot Recall@5 is achieved across 4,056 benchmark pairs, with agency-level retrieval spanning 89–93% and category-level Recall@5 reaching at least 87%.Dimensional Accuracy is strongest at 95.37%, while Logical Reasoning is weakest at 87.64%.
  • Answering ablations: 82.31% is the best answering result, achieved by critic self-correction with Qwen2.5-VL-72B, while retrieval-augmented prompting performs worst at 62.50% on 7B and 70.75% on 72B.Self-correction surpasses Qwen-72B zero-shot by 5.42 percentage points and Qwen-7B zero-shot by 4.01 percentage points.
  • Visual grounding: Top-5%-activation MaxSim heatmaps consistently localize evidence in a 16-query pilot and improve response specificity, while audit-step overlays create a verifiable evidence trail.The study treats grounding qualitatively and defers quantitative IoU evaluation with human-annotated regions.
  • Compliance checking: 100% verdict accuracy is achieved on the 8-query PASS set, while the 24-query FAIL set reaches 79.17% overall True Positive Rate and multi-plan pilots reach 90% and 88% accuracy.The PASS set clears the 90% false-positive threshold, the FAIL set clears the 60% sensitivity threshold, and scale-up misses concentrate in the N=5 pier subset.

6. Sample Cases

The sample cases contrast a successful interpretation of a WYDOT standard plan with a hard failure involving component–dimension binding. The failure motivates the taxonomy presented in Section 7.

  • Representative cases: Judge score = 1.0 for the Qwen2.5-VL-7B generator’s correct interpretation of a WYDOT standard plan.This success case is shown in Figure A.1 (Appendix H).
  • Representative cases: Judge score = 0.0 for a hard failure binding the V-mesh end-strip height to the wrong component dimension.This failure case is shown in Figure A.2.
  • Failure analysis: The component–dimension binding error motivates the failure-mode taxonomy in Section 7.The error concerns the V-mesh end-strip height.

7. Failure Case Analysis

The failure analysis identifies systematic visual–textual reasoning errors, especially binding correct dimensions to the wrong components and inferring values from non-referenced views. High-DPI tile retrieval improves recall and judge accuracy only modestly, so the predefined tiling-improvement hypothesis is rejected.

  • Systematic reasoning errors: The most common failure was assigning a correctly detected numeric value to the wrong physical entity when nearby annotations referred to different components.A salient 4′-10′′ [1.5 m] dimension was repeatedly selected for queries about steel-stake spacing and V-mesh end-strip details.
  • Systematic reasoning errors: Cross-view inference produced hallucinated precision by extracting dimensions from another detail instead of the explicitly referenced view.Under the rubric, this engineering-convention violation yielded a hard 0.0 score.
  • Tiling ablation: 82.08% tile-level Recall@5 exceeded 76.89% full-page Recall@5 by 5.19 pp, as focused high-DPI crops preserved fine detail and reduced competing annotations.Crops largely resolved component–dimension binding errors, but introduced loss of cross-reference context when notes and referenced content were separated.
  • Tiling ablation: 68.77% tile-level judge accuracy exceeded 64.23% full-page accuracy by only 4.53 pp, below the pre-declared 20 pp threshold, so H4 was rejected.The ablation used 400 DPI 1024×1024 crops with 256-pixel overlap versus 200 DPI full-page retrieval on the 424-pair test split.
  • Tiling ablation: The narrow WYDOT-only dense-subset gain did not generalize to the heterogeneous five-DOT test set, where tiling produced a consistent but modest lift rather than a step change.Nearly all 400 DPI plan pages in the five-DOT test set were uniformly tile-dense.

8. Discussion

PlanSightRAG’s visual-first design preserves plan structure and sustains robust cross-state retrieval, while agentic decomposition and MaxSim grounding support auditable compliance reasoning. Reasoning performance depends on model scale and prompting regime, with larger models benefiting from structured self-correction rather than zero-shot inference alone.

  • Visual-first retrieval: Over 91% Recall@5 was maintained across five DOTs using off-the-shelf patch-level embeddings, supported by a hand-curated, manually verified anchor set.The result reflects retrieval over preserved layout, geometry, and symbolic relationships rather than OCR-only representations.
  • Reasoning and prompting: 78.30% vs 76.89% shows the 7B answerer marginally exceeding the 72B model in zero-shot reasoning.The 72B model’s advantage emerges under structured reasoning, where question decomposition and critic self-correction reach 81.60% and 82.31%.
  • Auditable reasoning: Agentic decomposition converts complex cross-plan queries into structured verification steps, while MaxSim heatmaps provide a transparent, auditable evidence trail for regulatory and safety-critical reviews.This architecture addresses limitations of single-pass RAG by making the reasoning process explicitly grounded in retrieved visual evidence.
  • Cross-state generalization: Pretrained ColNomic-3B visual encoders support exceptional zero-shot cross-state generalization, while domain-specific LoRA fine-tuning is ineffective or harmful on small training corpora.The discussion recommends treating pretrained multi-vector retrievers as fixed components and allocating compute to advanced reasoning and prompting strategies.

9. Limitations

PlanSightRAG’s evaluation is limited by synthetic or machine-generated data, incomplete real-world threshold extraction, and persistent visual reasoning errors. High-resolution requirements and sequential VLM calls also constrain robustness and production latency.

  • Evaluation Data: The 4,056-pair benchmark uses machine generation and automated verification rather than direct licensed-engineer annotation, mitigated by a curated anchor set and structured human validation.These safeguards are described as mitigating major failure modes.
  • Real-World Generalization: 33% of numeric items are successfully extracted on a real 931-page WYDOT standard corpus, leaving autonomous threshold extraction from dense production standards an open challenge.Parameterized CAD test sets isolate VLM discrimination but do not reproduce ambiguities such as overlapping callouts or smudged scans.
  • Robustness: Incorrect component–dimension binding and symbol misalignment occasionally persist even with optimal retrieval, while low-quality scans degrade performance.The system’s reliance on high-resolution visuals makes it sensitive to scan quality.
  • Deployment: Sequential VLM calls introduce latency, requiring further optimization before real-time production use.The latency arises from the agentic pipeline’s sequential calls.

10. Conclusion

PlanSightRAG addresses the limitations of OCR-driven compliance checking by retrieving and reasoning directly over plan imagery. Its results support visual-first retrieval and agentic verification as a foundation for automated engineering plan review, while identifying fine-grained visual reading and rule-threshold provision as remaining bottlenecks.

  • Framework: PlanSightRAG combines visual-first plan-image retrieval, ColNomic-3B patch-level retrieval, agentic compliance processing, and sharpened MaxSim grounding.The framework is designed to preserve layout, geometry, and symbolic cues discarded by OCR-driven, text-centric pipelines.
  • Results: 92.69% Recall@5 was achieved by ColNomic-3B on the 4,056-pair five-DOT benchmark.The page-disjoint test split contains 424 pairs and runs against the full 1,898-page joint index.
  • Results: 100% verdict accuracy was achieved by the agentic Planner–Auditor–Synthesizer pipeline when supplied with per-drawing pre-resolved rule thresholds.The conclusion identifies fine-grained visual reading, entity–dimension binding, and supplying pre-resolved rule thresholds as decisive bottlenecks.
  • Future directions: Symbol-substitution robustness, multi-agency adaptation, and scale-invariant retrieval remain open problems for automated engineering plan review.The findings characterize visual-first retrieval, multimodal reasoning, agentic verification, and visual grounding as a viable foundation.

11. Future Work · CRediT authorship contribution statement · Funding

Future work targets broader, versioned DOT archives, stronger perception of degraded scans, and more precise visual annotations. The paper also records individual author contributions and WYDOT funding support.

  • 11. Future Work: Future research should expand the index to historical archives and versioned standards from additional state DOTs.This would enable year-aware comparisons and change tracking.
  • 11. Future Work: Perception robustness should improve for low-quality or handwritten legacy scans through image enhancement and multi-resolution indexing.The passage identifies these capabilities as necessary extensions.
  • 11. Future Work: Visual grounding should be extended to precise, structured annotations such as dimension-entity links.These annotations should be validated against CAD-derived ground truth.
  • CRediT authorship contribution statement: Nabaraj Subedi contributed conceptualization, methodology, software, and original-draft writing.Subedi’s listed contributions also include the original draft of the paper.
  • CRediT authorship contribution statement: Shuvo Dip Datta contributed literature-review investigation and original-draft writing.Ahmed Abdelaty contributed conceptualization, supervision, review, and editing; Shivanand Venkanna Sheshappanavar contributed supervision, review, and editing.
  • Funding: The Wyoming Department of Transportation financially supported the work under knowledge-management project grant RS03225.Ahmed Abdelaty is identified as the Principal Investigator.

B. Human validation of the generated datasets … E. Compliance pipeline confusion matrix and decomposition analysis

The appendix qualifies dataset-label validation, reports retrieval and judge robustness on a page-disjoint test split, and analyzes compliance-pipeline accuracy and decomposition. Human review remains ongoing, while automated filtering, transfer retrieval, cross-family judging, and all-PASS compliance evaluation provide bounded evidence with explicit limitations.

  • B. Human validation of the generated datasets: All 4,656 generated items were submitted through a web-based human-review protocol alongside the original drawings and complete contextual information.The set comprised 4,056 QnA pairs, 500 single-document compliance drawings, and 100 multi-plan drawings.
  • B. Human validation of the generated datasets: Human validation is ongoing, so label-quality guarantees are currently limited to the 78-pair anchor set and the automated two-stage filter.An independent Qwen-VL-7B verifier rejects roughly half of drafted questions before the verified corpus is used.
  • C. Bootstrap CIs on the 424-pair test split: Bootstrap intervals are reported on the full 424-pair page-disjoint test split over the 1,898-page five-DOT visual index, including category, generator, baseline, and agency analyses.The appendix uses percentile bootstrap confidence intervals with 10,000 resamples; generator judgments cover Qwen2.5-VL-7B, Qwen2.5-VL-72B, and InternVL-2.5-8B.
  • D. Additional retrieval and judge analyses: 92.45% Recall@5 versus 66.75% Recall@1 yields an approximately 26-point gap, motivating optional cross-encoder reranking for rank-1-sensitive deployments.The adopted ColNomic-3B retriever is also evaluated with MRR and nDCG@10.
  • D. Additional retrieval and judge analyses: 91.40% Recall@5 on the joint Michigan index essentially matches the 91.47% in-distribution benchmark and is reported instead of 93.55% from the smaller Michigan-only pool.The joint-index comparison removes the candidate-pool-size confound.
  • D. Additional retrieval and judge analyses: 92.92% judge agreement with Cohen’s κ=0.75 accompanies stricter same-family Qwen judging at 79.95% versus 85.61% cross-family judging.The comparison re-judged 424 Qwen2.5-VL-7B zero-shot answers using InternVL2.5-8B on identical inputs.
  • E. Compliance pipeline confusion matrix and decomposition analysis: The Planner averages 8.6 steps per query, or approximately 4.3× the two-step minimum, increasing latency linearly without compromising correctness in this evaluation.Observed decompositions range from 7–10 steps, with each expanded sub-step retrieving a relevant plan region.
  • E. Compliance pipeline confusion matrix and decomposition analysis: 8/8 = 100.0% compliance accuracy reflects avoidance of false-positive rejections on valid all-PASS plans, not sensitivity to violations.The eight queries target internally consistent WYDOT configurations, and malformed-plan evaluation is deferred.

E.1. False-Negative Sensitivity Study

The false-negative study shows that the pipeline detects 79.17% of deliberately non-compliant queries while maintaining 100% specificity on valid designs. Symbol/component swaps are the principal weakness, especially when the substituted component appears visually plausible.

  • Study motivation: The reported 100% accuracy from Section 5.6 reflects an all-PASS set and therefore does not measure false-negative violation-detection sensitivity.This study was introduced to probe missed violations directly.
  • Study design: The evaluation used 32 blind synthetic queries: eight PASS queries and 24 FAIL queries evenly divided across dimension mutation, note omission, and symbol/component swap violations.The FAIL queries describe deliberately non-compliant proposed designs rather than mutated plan images, matching the system’s query-driven deployment mode.
  • Results: 100% specificity (TNR = 100%) was achieved on eight PASS queries, with zero false positives, while overall TPR on 24 FAIL queries was 79.17%.The balanced-class Cohen’s κ was 0.655.
  • Results: Mdim and Mnote violations each reached 87.5% (7/8) sensitivity, whereas Msym violations reached only 62.5% (5/8).The three Msym misses involved visually plausible substitutions, such as wooden for steel stakes or concrete for cable end anchors.
  • Limitations and follow-up: The symbol-swap failure mode is identified as the next iteration’s specific target, with an explicit component-comparison step proposed after retrieval.The numerical results were measured from 24 completed FAIL queries, with average pipeline latency of approximately 62 seconds per query.

F. Latency distribution … Rule Grounding and Real-Plan Auditing Cases

The appendix reports retrieval and end-to-end latency, a small but operational grounding-IoU pilot, representative success and failure cases, and autonomous rule-grounding inputs for real-plan auditing. It also documents that the pilot’s apparent IoU=1.0 is invalid because predicted boxes seeded the ground truth.

  • F. Latency distribution: 101 ms median and 103.71 ms p99 characterize tightly concentrated full-page retrieval latency with a heavy cold-cache right tail.The mean is 101.24 ms/query over the 1,898-page five-DOT index, reflecting linear corpus scaling from brute-force MaxSim scoring.
  • F. Latency distribution: Millisecond-scale unpacking overhead accompanies binary-quantized HPC-ColPali, which compresses the index by an order of magnitude.The tile-level variant instead trades higher per-query cost for finer spatial granularity.
  • F. Latency distribution: 2–5 s is the estimated interactive latency per single-shot query because Qwen-7B VQA generation dominates retrieval cost.The agentic Planner–Auditor–Synthesizer turns typically multiply this latency by 3–5× when cross-referencing multiple plans.
  • G. Grounding IoU pilot: 48 predicted boxes are produced end-to-end for the 16-query pilot, with three ColPali candidates per query and reviewer-facing overlays.The pilot tests whether sharpened MaxSim heatmaps yield spatially localized boxes overlapping human-identified answer regions, rather than supporting per-category claims.
  • G. Grounding IoU pilot: IoU=1.0 on every query is not meaningful because the ground-truth JSON was seeded with predicted boxes rather than human annotations.Domain-informed human annotation and a larger fully annotated study are planned follow-up work.
  • Success Cases (Judge Score = 1.0): Judge score 1.0 cases show correct-sheet retrieval followed by direct reading of the queried dimension from the drawing.The model response matches the benchmark ground-truth answer.
  • Failure Cases (Judge Score = 0.0): Judge score 0.0 failures occur when retrieval succeeds but the VLM binds the query to a neighboring dimension or conflates related quantities.These are the two dominant error modes analyzed in Section 7; the accompanying figure describes component–dimension binding and semantic confusion.
Loading 2608.26091v1…