PlanSightRAG: A Visual-First Multimodal RAG for Automating Question Answering and Compliance Checking for Civil Standard Plans

1. Introduction

• PlanSightRAG combines OCR-free visual plan indexing, MaxSim grounding, and agentic compliance auditing, validated on a five-DOT benchmark.Its Planner–Retriever–Auditor–Synthesizer pipeline uses sharpened MaxSim heatmaps as a transparent evidence trail.Source: §1 · p. 3
• Recall@5 = 100% among 1,913 candidates; the agent extracts a numeric limit by resolving symbolic d/2, then audits against its self-grounded threshold without human-supplied rules.This matches the stated 100% ceiling when the rule threshold is supplied in advance.Source: §1 · p. 3
• The introduction frames four falsifiable hypotheses testing visual retrieval, agentic auditing, held-out domain adaptation, and whether high-resolution tiling improves judge accuracy.Each hypothesis is evaluated against an explicit numeric threshold.Source: §1 · p. 3

2.1. Automated Compliance Checking in Civil Infrastructure

• Traditional rule-based and text-driven compliance methods assume requirements can be structured or linearized, although legacy plans encode them implicitly in geometry, layout, and cross-view symbols.This leaves manual visual inspection error-prone.Source: §2.1 · p. 3

2.2. OCR-Based Document QA and Text-Centric RAG Systems

• RAG works for explicitly textual rules and answers, but layout-aware text encoders and image-to-structure methods recover only part of OCR-lost spatial information, which can still degrade engineering compliance.The passage positions these methods as the text-centric reference point for visual-first retrieval.Source: §2.2 · p. 3

2.3. VLMs for Drawing Understanding

• Civil-domain studies find generic VLMs struggle with engineering drawings; visual benchmarks expose weak spatial grounding and hallucination risks [30], [31], [32], [33], [34], while repository-scale evidence-grounded compliance remains unaddressed.Dense, domain-specific technical-document applications remain underexplored despite VLMs’ strength on general multimodal benchmarks.Source: §2.3 · pp. 3–4

2.4. Vision-First Document Retrieval and Multimodal RAG

• Vision-first late-interaction retrieval preserves spatial alignment without OCR [7], [35], but prior multimodal RAG remains focused on isolated tasks; PlanSightRAG integrates visual indexing with agentic VLM reasoning for cross-sheet inspection.The approach targets structural linking across large engineering plan archives.Source: §2.4 · p. 4

3.1. Visual-First Multimodal RAG Framework Overview

• The four-phase framework rasterizes plans, indexes ColNomic multi-vector patches, retrieves and grounds evidence with MaxSim, then performs multi-VLM VQA or agentic compliance auditing.The compliance pipeline uses a Planner–Retriever–Auditor–Synthesizer, while VQA and compliance use different model configurations; optional VLM reranking and BM25 fusion refine retrieval.Source: §3.1 · p. 4, Figure 2 · p. 5

3.2. Document Ingestion and Preprocessing

• The five-state DOT corpus uses 200-DPI full-page images for headline retrieval and compliance; overlapping 400-DPI tiles preserve fine detail and are evaluated only as an ablation.Tiles are 1024 × 1024 pixels with 256-pixel overlap; tile metadata records grid position and bounding boxes for spatial back-mapping, and a 298-page Michigan corpus is held out for zero-shot evaluation.Source: §3.2 · pp. 4, 6

3.3. Visual Indexing

• ColNomic-3B encodes pages and tiles as dynamic-resolution multi-vector patch embeddings, preserving spatially distinct drawing details and enabling the same index to serve retrieval, VQA, and compliance.Document-level metadata is extracted once per PDF, and the 1,898 in-distribution pages form one joint index; Michigan is stored separately for zero-shot evaluation.Source: §3.3 · p. 6

3.4. Query Processing, Retrieval & Visual Grounding

• MaxSim ranks pages by aligning each query token with its most relevant visual patch; optional VLM cross-encoder reranking and BM25 fusion address relevance verification and alphanumeric queries.The reranker checks Top-K candidates sequentially and stops after two positive verifications; BM25 scores from metadata are min-max normalized and fused with dense scores.Source: §3.4 · p. 6

4. Preserved Annotations + Grid Position

• At 400 DPI, engineering sheets are divided into overlapping 1024 × 1024 tiles with 256-pixel overlap, preserving fine annotations and enabling sub-page retrieval.Source: Figure 3 · p. 7
• ColNomic’s patch-level late interaction exposes retrieval-driving regions, while OCR-region scoring returns bounding-box evidence for annotations, labels, notes, or graphics.The same grounding mechanism attaches to VQA answers and each auditor step, creating a persistent visual evidence trail.Source: §4 · p. 7
• The sharpened MaxSim heatmap retains only the top 5% of patches, raises normalized values to γ=3.0, and overlays the colorized map on the page at α=0.4.The map is bicubic-upscaled to the original resolution before overlay.Source: §4 · p. 7

3.5. Visual Question Answering (VQA) Pipeline

• VQA supplies retrieved plan images, lightweight plan-ID and reference-index anchors, and the query to Qwen 2.5-VL, returning an answer with an evidence gallery.An optional user-supplied diagram can also be included, and users can verify answers against the supporting plan sheets.Source: §3.5 · p. 7

3.6. Automated Compliance Checking Workflow

• Compliance audits classify findings as unit, dimension, label, or logic and report the requirement, observed condition, and verdict under an enforced JSON schema.Verdicts are pass, fail, unclear, or hallucination_suspected.Source: §3.6 · p. 8
• To handle compliance queries spanning multiple plans and rules, the system uses a Planner–Retriever–Auditor–Synthesizer pipeline.The Planner creates focused steps; the Retriever finds plan pages; the Auditor produces findings and preliminary verdicts; and the Synthesizer consolidates findings into a final verdict and report.Source: §3.6 · p. 8
• Each audit step produces an overlaid MaxSim heatmap, and the pipeline outputs a machine-readable Markdown report interleaving findings, verdicts, and grounding images.Source: §3.6 · p. 8

3.7. Implementation and Deployment Details

• 60.9 s/query is the average agentic compliance latency, compared with ≈0.10 s p50 for full-index ColNomic retrieval and 2–5 s for single-shot VQA.Indexing the 1,898-page corpus takes 7.3 min offline; the agentic pipeline averages ∼8.6 sequential VLM steps per verdict. The system targets a single GPU and uses 4-bit BitsAndBytes quantization with NF4 weights and bfloat16 compute.Source: §3.7 · p. 9

4.1. Benchmark Dataset Construction

• The benchmark contains 4,056 verified question–answer pairs across five state DOTs, plus 93 verified Michigan pairs for zero-shot transfer evaluation.The visual index contains 1,898 plan pages, rendered at 200 DPI; reported retrieval results re-embed those pages with ColNomic-3B.Source: §4.1 · p. 9
• The refine loop generated page-conditioned questions across dimensional, visual, logical, and hallucination categories, rephrasing retrieval misses before independent answer verification.It allowed up to five rephrases per question (eight for Colorado), followed by a separate verifier that filtered questions whose answers were not page-supported.Source: §4.1 · p. 9
• On ColPali-miss test questions, ColNomic-3B achieved 77.89% Recall@5, versus 96.66% on ColPali-hit questions, countering a material curation-bias concern.ColPali missed 23.9% of finalized pairs overall (968/4,056; 95/424 on the page-disjoint test split).Source: §4.1 · p. 9

Manually-curated anchor set (78 WYDOT

• On 78 human-curated WYDOT pairs, Recall@5 was 91.03% and end-to-end judge accuracy was 80.77%, independently checking the machine-generated benchmark.The retrieval profile matched the large benchmark, with dimensional queries strongest and hallucination and visual queries hardest.Source: Manually-curated anchor set (78 WYDOT · pp. 9–10, Table 1 · p. 10
• The “Hallucination Rate” subset is predominantly answerable and should be read as a specification/material-lookup stress set, not a pure abstention test.Abstention is measured separately in the CAD false-negative study.Source: Manually-curated anchor set (78 WYDOT · p. 10
• The compliance test uses parameterized synthetic plan drawings with controlled single-rule violations and three ground-truth sets: 14 pilot, 500 single-document, and 100 multi-plan drawings.The generator covers five drawing archetypes and three density tiers; multi-plan verdicts require two to five plans, without plan IDs on the drawings.Source: Manually-curated anchor set (78 WYDOT · p. 10

4.2. Evaluated Models

• Qwen2.5-VL-72B and Llama-3.2-90B-Vision draft benchmark questions, while Qwen2.5-VL-72B judges compliance; ColNomic retrieval is fixed across generator comparisons.Open-source models use 4-bit NF4 quantization, and evaluation runs through an automated retrieval, answer-generation, and metric-computation pipeline.Source: §4.2 · p. 10

4.3. Evaluation Metrics

• Evaluation reports Recall@5 for retrieval and binary LLM-as-judge accuracy for end-to-end correctness, including retrieval and reasoning.Recall@5 measures whether the gold plan page appears among the top five candidates passed to the generator [41]; judge accuracy compares the predicted answer with ground truth [42].Source: §4.3 · pp. 10–11

5.1. Comparison with Existing Solutions

• 92.69% Recall@5 for ColNomic-3B exceeded BGE-M3 + OCR (36.79%) by 55.90 pp and VisionRAG (Pyramid, RRF) (23.11%) by 69.58 pp, clearing H1’s 15-pp thresholds.These comparisons use the 424-pair page-disjoint test split against the full 1,898-page index; the adopted ColNomic-3B backbone is openly licensed and deployable.Source: §5.1 · p. 12, Table 4 · p. 12

5.2. Overall End-to-End Performance on the Five-DOT Benchmark

• 91.47% Recall@5 was achieved zero-shot on the full 4,056-pair five-DOT benchmark.Recall@5 ranged from 89–93% across agencies: WYDOT was highest at 93.37%, while CDOT reached 90.29% despite near-duplicate detail sheets.Source: §5.2 · p. 13

5.3. Category-Wise Performance Analysis

• All four categories reached ≥87% Recall@5; Dimensional Accuracy led at 95.37%, while Logical Reasoning was lowest at 87.64%.Logical Reasoning asks the agent to infer a rule from a note or table rather than match a page-unique numeric anchor.Source: §5.3 · p. 13, Table 7 · p. 13

5.4. Prompting and Reasoning Technique Ablation

• 82.31% judge accuracy was the best prompting result, from critic self-correction with Qwen2.5-VL-72B, 5.42 pp above its zero-shot result.Retrieval-augmented prompting was worst on both models (62.50% on 7B; 70.75% on 72B); the union of all nine techniques reached 97.16% accuracy.Source: §5.4 · pp. 13–14, Table 8 · p. 13

5.5. Visual Grounding and Explainability Results

• In a 16-query qualitative pilot, thresholding sharpened MaxSim heatmaps at the top 5% of activations localized evidence, and the resulting crops improved response specificity.The heatmaps preserve spatial relationships between notes and geometric details; quantitative IoU evaluation with human-annotated regions remains future work.Source: §5.5 · p. 14, Figure 5 · p. 14

5.6. Agentic Compliance Results

• The agentic pipeline achieved 100% verdict accuracy on the 8-query PASS set and 79.17% True Positive Rate on the 24-query FAIL set.The FAIL-set rate was 87.5% for dimension/note mutations and 62.5% for symbol substitutions; the PASS set measured false-positive avoidance and contained no FAIL cases.Source: §5.6 · p. 14

5.7. Compliance Validation on a CAD-Generated Test Set

• With per-drawing pre-resolved numeric thresholds, the Qwen2.5-VL-72B recipe achieved 500/500 = 100% verdict accuracy, while the non-VLM OCR baseline reached 76.4% on scale500.The VLM’s advantage was robust value localization on rotated or distractor-dense layouts, rather than arithmetic; the reported 100% is a ceiling given a resolved rule and localizable value. The same recipe reached 50/50 = 100% on near-threshold cases, while multi-plan accuracy rose from 88/100 with textual d/2 to 100/100 when that threshold was resolved.Source: §5.7 · pp. 16–17

5.8. Autonomous Rule-Grounding: Retrieving the Governing Threshold

• With no injected threshold, rule-grounding achieved 80.0% Recall@1 and 100.0% Recall@5 for specification retrieval, 100% numeric-limit extraction, and 100% end-to-end verdict accuracy.The 15-sheet specification corpus was indexed among 1,898 plan pages; top-1 misses did not affect verdicts because the correct specification was always within the top five.Source: §5.8 · p. 17
• Across 500 single-document compliance drawings, the agent retrieved specification sheets and extracted numeric limits from rasterized imagery without OCR, BIM, or hand-coded rules.The clean, modest corpus demonstrates closed-loop feasibility rather than complete code coverage; testing on full agency standards is the decisive stress test.Source: Table 13 · p. 18, §5.8 · p. 18

5.9. External Validity: Real WYDOT Project Plans

• On real WYDOT plans, Qwen2.5-VL-72B achieved 100% verdict accuracy on 12 controlled compliant and violating examples when given the governing rule.ColNomic retrieved a topical governing standard in the top 5 for 10/12 real design items; the two misses lacked dedicated sheets in the index. On a 931-page real standard corpus, autonomous extraction returned the correct numeric limit for 3/9 items (33%) and correct verdicts for 44% of items; failures centered on retrieving governing sentences and resolving exhibit tables.Source: §5.9 · p. 18

5.10. Cross-Agency Generalization and Scaling

• 91.40% Recall@5 on held-out Michigan DOT over the combined index essentially matched 91.47% overall, with 89–93% Recall@5 across five agencies.Michigan was unseen during indexing, drafting, and training; the smaller Michigan-only pool reached 93.55% but is not directly comparable. The paper reports H3 rejected: domain adaptation did not improve the 5-DOT test split and degraded unseen-agency transfer.Source: §5.10 · pp. 18–19

5.11. Domain-Adaptive Retrieval: LoRA Fine-Tuning of the Adopted Retriever

• Full-LM LoRA reduced in-distribution retrieval from 92.69% to 55.66% and Michigan transfer from 93.55% to 75.27%, while head-LoRA changes were under 0.5 pp.The off-the-shelf zero-shot retriever remained the preferred deployment choice. Orthogonal options include VLM reranking to sharpen Recall@1 and BM25 fusion for identifier queries; reranking Top-20 candidates added about 5.91 s per query.Source: §5.11 · p. 19

6. Sample Cases

• 1.0 judge score marks Qwen2.5-VL-7B’s correct interpretation of a WYDOT plan, while the 0.0-score failure misbinds the V-mesh end-strip height.The hard failure motivates the failure-mode taxonomy in Section 7.Source: §6 · p. 19

7. Failure Case Analysis

• The most common failure was binding a real numeric dimension to the wrong component, such as using 4′-10′′ [1.5 m] for stake spacing or scarification depth.That dimension actually described enclosure height or panel geometry, despite being visually salient.Source: §7 · p. 19
• Other failures included inferring dimensions across views, confusing semantic categories, misreading reused symbols, and placing features by generic heuristics instead of exact plan language.Examples include confusing scarification depth with fill thickness and wire-tie-center markers with steel-stake symbols.Source: §7 · p. 20

Impact of high-resolution tiling on failure

• Tile-level retrieval reaches 82.08% Recall@5 versus 76.89%, while judge accuracy rises from 64.23% to 68.77% (+4.53 pp), rejecting H4’s 20 pp threshold.The test used a 424-pair page-disjoint split; the retrieval gain was consistent but modest on the heterogeneous five-DOT set.Source: Table 16 · p. 20, Table 17 · p. 20, Impact of high-resolution tiling on… · pp. 20–21
• High-DPI tiles preserve fine linework and annotation density, and focused crops reduce competing annotations, but may separate notes from referenced sections, motivating joint full-page and tile retrieval.The trade-off is greater spatial precision at the cost of cross-reference context.Source: Impact of high-resolution tiling on… · p. 20

• Over 91% Recall@5 across five DOTs with off-the-shelf patch-level embeddings, backed by a hand-curated, manually verified anchor set, supports robust retrieval.The framework preserves layout, geometry, and symbolic relationships rather than relying on OCR-centric pipelines.Source: §8 · p. 21
• At zero-shot, 7B scored 78.30% vs 72B’s 76.89%; structured reasoning raised 72B to 81.60% with decomposition and 82.31% with critic self-correction.Zero-shot prompting suffices for 7B-class models, while 70B-class judges benefit from critic self-correction; the single top-ranked page performs best, and retrieval-augmented prompting is weakest for both models.Source: §8 · p. 21
• Agentic decomposition overcomes single-pass RAG limitations on complex cross-plan queries, while MaxSim heatmaps provide transparent, auditable grounding for engineering reviews.The evidence trail is intended for regulatory and safety-critical engineering reviews.Source: §8 · p. 21
• Pretrained ColNomic-3B shows zero-shot cross-state generalization, while domain-specific LoRA fine-tuning is ineffective or harmful on small training corpora.The findings suggest keeping pretrained multi-vector retrievers fixed and directing compute toward advanced reasoning and prompting.Source: §8 · p. 21

• The 4,056-pair benchmark uses machine generation and automated verification rather than licensed-engineer annotation, mitigated by a manually curated anchor set and structured human validation.Source: §9 · p. 21
• 33% of numeric items are extracted from a real 931-page WYDOT standard corpus, while parameterized CAD tests do not capture ambiguities such as overlapping callouts or smudged scans.Autonomous threshold extraction from dense production standards remains an open challenge.Source: §9 · p. 21
• Incorrect component–dimension binding and symbol misalignment occasionally persist despite optimal retrieval, while high-resolution visual dependence makes the system sensitive to low-quality scans and sequential VLM calls add latency.The latency requires further optimization for real-time production use.Source: §9 · p. 21

10. Conclusion

• 92.69% Recall@5 on the 424-pair test split was 55.90 pp above BGE-M3 + OCR and 69.58 pp above VisionRAG (Pyramid, RRF).The test split was page-disjoint and searched the full 1,898-page joint index.Source: §10 · p. 21
• 100% verdict accuracy on both compliant and non-compliant cases was achieved on 500 single-doc CAD drawings and 100 multi-plan drawings, with pre-resolved rule thresholds.The multi-plan set included N∈{2, 3, 4, 5} standards per design.Source: §10 · p. 21
• 91.40% Recall@5 on an unseen sixth agency essentially matched the 91.47% in-distribution result, showing zero-shot transfer across agencies.Source: §10 · p. 21
• No tested LoRA recipe improved the zero-shot retriever, while full-LM LoRA catastrophically degraded it; high-resolution tiling gave gains below H4’s pre-declared 20 pp bar.Tile-level retrieval reached 82.08% Recall@5 versus 76.89% for full-page (+5.19 pp); judge accuracy rose from 64.23% to 68.77% (+4.53 pp).Source: §10 · p. 21
• The benchmark’s scale and manual anchor-set validation remove “small-sample” as a threat to validity.Source: §10 · pp. 21–22
• The compliance bottleneck is fine-grained visual reading and entity–dimension binding, and decisively the need to supply pre-resolved rule thresholds.Source: §10 · p. 22
• Symbol-substitution robustness, multi-agency adaptation, and scale-invariant retrieval remain open problems for automated engineering plan review.Source: §10 · p. 22

11. Future Work

• The index should expand to historical archives and versioned standards from additional state DOTs for year-aware comparisons and change tracking.Source: §11 · p. 22
• Image enhancement and multi-resolution indexing are proposed to improve perception robustness on low-quality or handwritten legacy scans.Source: §11 · p. 22
• Visual grounding should produce precise structured annotations, such as dimension-entity links, validated against CAD-derived ground truth.Source: §11 · p. 22
• The agentic compliance framework should handle cross-agency regulatory conflicts autonomously for production deployment.Source: §11 · p. 22
• The framework should support autonomous version-aware checks as part of large-scale production deployment.Source: §11 · p. 22
• The framework should autonomously generate variance documentation for large-scale production deployments.Source: §11 · p. 22

Funding

• Financial support came from the Wyoming Department of Transportation under knowledge-management project grant RS03225.Ahmed Abdelaty was the principal investigator.Source: Funding · p. 22

B. Human validation of the generated datasets

• All 4,656 generated QnA and compliance items are undergoing independent human review with full-resolution drawings, supporting context, and correct, incorrect, or unsure verdicts.The large-scale validation is ongoing, so confirmation rates, agreement, and flagged-item counts are not yet available; label-quality claims are scoped to the 78-pair human-curated anchor set and automated two-stage filter, whose verifier rejects roughly half of drafted questions.Source: B. Human validation of the generated… · p. 25

C. Bootstrap CIs on the 424-pair test split

• Bootstrap confidence intervals use the full 424-pair page-disjoint test split over the 1,898-page five-DOT index, with 10,000 resamples and seed 42.The appendix reports per-category Recall@5 and per-generator judge accuracy, plus per-baseline, per-agency Recall@5.Source: C. Bootstrap CIs on the 424-pair test… · p. 25, Table A.1 · p. 26, Table A.2 · p. 26

D. Additional retrieval and judge analyses

• Recall@5 was 92.45% versus Recall@1 at 66.75% for zero-shot ColNomic-3B, a rank gap relevant because the single-pass auditor consumes the rank-1 page.The appendix notes that this motivates an optional cross-encoder re-ranker for rank-1-sensitive deployments; a separate predecessor-ColPali table reports lower logical-reasoning Recall@5 than Table 7, while noting ColNomic-3B closes that gap.Source: D. Additional retrieval and judge… · p. 25, Table A.3 · p. 26, Table A.1 · p. 26
• 91.40% Recall@5 on the joint index makes Michigan transfer comparable to the 91.47% in-distribution benchmark, unlike the easier 93.55% Michigan-only result.The joint index controls for candidate-pool size; the paper reports the joint-index figure as the comparable transfer number and does not claim Michigan exceeds in-distribution agencies.Source: D. Additional retrieval and judge… · p. 25, Table A.4 · p. 26
• The two judges agreed on 92.92% of Qwen2.5-VL-7B answers (Cohen’s κ=0.75), and the same-family Qwen judge was stricter than the cross-family judge.The reported judge accuracies were 79.95% for Qwen and 85.61% for InternVL2.5-8B, which the authors interpret as evidence against same-family self-preference.Source: D. Additional retrieval and judge… · p. 25

E. Compliance pipeline confusion matrix and decomposition analysis

• The Planner produced 7–10 steps per compliance query, averaging 8.6 versus a two-step minimum, increasing latency linearly without compromising correctness in this evaluation.The expanded sub-steps still retrieved relevant plan regions; the eight-query scale and missing bounding-box metadata also limit the evidence-trail metric and domain-level analysis.Source: E. Compliance pipeline confusion… · p. 27, Table A.5 · p. 27
• 8/8 (100.0%) compliance verdict accuracy reflects avoiding false-positive rejections on eight valid all-PASS plans, not sensitivity to violations.No deliberately malformed plans were tested; the eight queries cover distinct domains, with one query per domain, and the small sample precludes per-domain breakdowns.Source: E. Compliance pipeline confusion… · pp. 26–27, Table A.5 · p. 27

E.1. False-Negative Sensitivity Study

• The earlier 100% accuracy was measured on an all-PASS ground-truth set and therefore characterized false-positive rejection avoidance, not false-negative violation-detection sensitivity.The query corpus, runner, submission wrapper, and scoring harness were released with the evaluation artifacts.Source: §E.1 · pp. 27–28
• Because the Auditor checks proposed values in query text against a retrieved standard-plan page, the study used non-compliant design queries to isolate its violation detection from retrieval artefacts.This query-based setup reflects the stated deployment mode in which an engineer submits a proposed value for checking against a relevant WYDOT standard.Source: §E.1 · p. 27
• The blind test comprised 24 synthetic FAIL queries—eight each for dimension mutations, note omissions, and symbol/component swaps—plus eight PASS queries, for N=32.The full query list is provided in the released artefacts.Source: §E.1 · p. 27
• A FAIL counted as detected only when the final rendered report explicitly stated a violation using one of the specified verdict patterns, making the criterion conservative.The study also reports confusion-matrix metrics and stratified TPR/FNR by violation type.Source: §E.1 · pp. 27–28
• TPR was 79.17% on 24 FAIL queries, while TNR was 100% on eight PASS queries; balanced-class κ was 0.655.Table A.6 reports per-stratum verdict counts and identifies GT=FAIL rows as the sensitivity measure.Source: Table A.6 · p. 28, §E.1 · p. 28
• Mdim and Mnote sensitivity was 87.5% (7/8) each, versus 62.5% (5/8) for Msym; overall FNR was 20.83%.The three missed swaps involved visually plausible components; retrieval surfaced the correct plan, but VQA hedged PASS without explicitly comparing the proposed and depicted components. Explicit numeric values triggered a more reliable numeric check.Source: §E.1 · p. 28

F. Latency distribution

• 101.24 ms/query is the mean retrieval latency across the 1,898-page five-DOT index, with a 101 ms median and p99 of 103.71 ms.The passage attributes the heavy right tail to cold-cache encoding on the first queries and mean scaling to the MaxSim brute-force scoring loop; binary-quantized HPC-ColPali adds millisecond-scale unpacking overhead while compressing the index by an order of magnitude.Source: F. Latency distribution · pp. 28–29
• 2–5 s per single-shot query is the estimated interactive latency, with Qwen-7B VQA taking 2–4 s and agentic turns multiplying latency by 3–5×.The estimate uses a 1024×1024 crop and 256 generation tokens on an A100 80GB; VLM generation dominates the wall-clock budget.Source: F. Latency distribution · p. 29

G. Grounding IoU pilot

• The 16-query grounding pilot derives predicted boxes from the top 5% of sharpened MaxSim patch activations and evaluates many-to-many maximum IoU, with IoU@0.5 as its primary metric.ColPali produces three candidate boxes per query (48 total); the pipeline renders reviewer overlays and scores against hand-drawn ground-truth boxes, but is intended to establish feasibility rather than support per-category claims.Source: G. Grounding IoU pilot · p. 29
• Human ground-truth boxes remain future work: the seeded boxes yield IoU=1.0 trivially, so the 16-query pilot reports no meaningful grounding measurement.The planned follow-up is human annotation and a larger-scale study; the paper's main results rely on Recall@5 and Judge Accuracy rather than pixel-level IoU.Source: G. Grounding IoU pilot · p. 29

H. Sample case figures

• The appendix figures document the Visual-First RAG pipeline on representative WYDOT benchmark queries, showing each query, response, and top-three retrieved pages with MaxSim scores.Success or failure is judged by comparing the model response with the benchmark ground-truth answer.Source: H. Sample case figures · p. 29

Success Cases (Judge Score = 1.0)

• In the success cases, retrieval surfaces the correct sheet and the VLM reads the queried dimension correctly, matching ground truth with judge score 1.0.Source: Success Cases (Judge Score = 1.0) · p. 29

Failure Cases (Judge Score = 0.0)

• In the failure cases, retrieval succeeds but the VLM binds the query to a neighboring dimension or conflates related quantities.Source: Failure Cases (Judge Score = 0.0) · p. 29

Rule Grounding and Real-Plan Auditing Cases

• The rule-grounding case requires the agent to retrieve a governing limit from a table row among distractors before auditing a design value.The illustrated specification sheet embeds seven governing limits among distractor rows, and the agent must select the correct sheet from 1,913 candidates.Source: Rule Grounding and Real-Plan Auditing… · p. 29, Figure A.3 · p. 32