PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounded and Faithful P&ID Reasoning: Constraining Vision-Language Models with Recovered Evidence Graphs

Prathamesh Gadekar, Sagar Srinivas Sakhinana, Venkataramana Runkana

arXiv:2609.05880v1cs.LGcs.AI

TL;DR

P&ID topology is consequential, yet dense raster sheets challenge pixel-only reading and VLMs need their connectivity claims checked against graph evidence. This paper recovers a process graph and requires a VLM to answer through graph tools with cited support. On TopoPID-VQA, graph grounding improves pooled exact-match accuracy across three open VLMs, while graph-recovery errors remain a boundary on reliability.

  • Problem

    P&ID decisions depend on connectivity, but dense sheets make pixel-only reading a poor fit and topology claims need explicit graph facts.

    Source: §1 · pp. 1–2

  • Method

    The system recovers a typed process graph from detected symbols, lines, and text, then requires VLM answers to use seven graph operators and cite supporting entities.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    Pooled accuracy reached 76.0 ± 1.8% vs. 40.1 ± 1.6% for Qwen3-VL-4B, 75.1 ± 1.4% vs. 41.3 ± 1.7% for Qwen3-VL-8B, and 74.3 ± 1.5% vs. 36.7 ± 1.6% for Gemma-4-E4B.

    Source: §5.5 · p. 7

  • Takeaways & Limitations

    Answer performance tracks both recovered-graph fidelity and multi-step tool use, so topology reasoning remains tied to the quality of its substrate.

    Source: §7 · p. 8

    Digitize-PID is synthetic, Gauto has exact edge F1 0.742, and the bundled evaluation lacks calibration or abstention, so safety-relevant use still needs human review.

    Source: §6 · p. 8

Abstract

from arXiv · show

Piping and Instrumentation Diagrams (P&IDs) are the authoritative maps of process plants: isolation, maintenance, and HAZOP decisions depend on what connects to what. Vision-language models describe these sheets fluently, yet they often invent or miss process connections---and an invented or missed link can reverse an isolation or reachability call, so a plant decision cannot trust a fluent answer that was never checked against the linework. We instead recover an explicit graph of the drawing---its symbols, the process connections between them, and the tags that name them---and then require the model to answer only by querying that graph through seven read-only operators, so a topology claim is returned only when it cites the query results that support it. On TopoPID-VQA, a new suite of 3000 topology questions over these sheets, Graph-Grounded Harness (Ours) raises exact match accuracy from 36.7--41.3% under image-only prompting to 74.3--76.0% for Qwen3-VL-4B, Qwen3-VL-8B, and Gemma-4-E4B. It does so on an imperfect substrate: on Digitize-PID dataset the recovered graph scores F1 0.742 on exact process connections, and 0.801 once symbols and tags are pooled in. The residual errors track that gap---grounding pays off where the recovered graph is right, and perception error still breaks topology questions where it is not.

1 Introduction

P&ID decisions depend on process connectivity, but dense raster diagrams challenge pixel-only interpretation and fluent VLMs can invent unsupported links. The paper proposes a recovered graph and evidence-citing harness, alongside a topology benchmark and measurements of answer and graph accuracy.

1 Introduction

  • P&IDs record process connections used in isolation, maintenance, HAZOP, and other decisions that depend on what connects to what.Invented or missed connections can constitute operational errors.Source: §1 · p. 1
  • Dense sheets with near-identical symbols, long pipe runs, crossings, and similar-looking tags make pixel-only reading a poor fit for topology questions.Source: §1 · p. 2

Grounding mechanism

  • Graph-Grounded Harness (Ours) requires a VLM to query seven graph operators and cite supporting entities for topology claims.The graph is recovered from detected symbols, lines, and text using deterministic construction rules.Source: §1 · p. 2

Topology benchmark

  • TopoPID-VQA is a 3,000-question suite covering easy, medium, and hard topology questions on recovered process graphs.Source: §1 · p. 2

Deployment-facing measurement

  • Graph grounding lifts pooled accuracy by 33.8–37.6 percentage points over image-only inference across Qwen3-VL-4B, Qwen3-VL-8B, and Gemma-4-E4B.Exact edge F1 on Gauto is 0.742, indicating remaining substrate error.Source: §1 · p. 2

1 Introduction

  • Cloud OCR and localization APIs and zero-shot line segmenters serve as perception controls; the paper reports them as weak substitutes for a P&ID-tuned substrate.This motivates finetuned perception plus deterministic graph construction rather than relying only on off-the-shelf detectors.Source: §1 · p. 3

2 Related Work

The related work spans document and relational reasoning, graph-augmented agents, and engineering-diagram understanding. The paper distinguishes its focus as end-to-end reasoning over an imperfect graph recovered from raster drawings.

Grounding and faithful multimodal reasoning

  • Document and chart VQA address reading and layout, while compositional benchmarks isolate relational reasoning; this paper recovers a typed graph and requires topology claims to come from tool results.Source: §2 · p. 3

Graph- and tool-augmented agents

  • Prior graph-query and tool-using systems commonly assume a clean graph or database, while this work operates on Gauto built from predicted perception and measures its drift from Goracle.Source: §2 · p. 3

Engineering-diagram understanding

  • Earlier P&ID QA often treats a labeled graph or smart source model as the knowledge base, whereas this work measures recovered-topology fidelity in a raster-to-agent pipeline.Dashed instrument links are kept in a separate signal-edge layer.Source: §2 · p. 3

3 Problem Formulation

The formulation separates the oracle answer, deployed inputs, and evaluation measures so answer errors can be distinguished from graph-recovery errors.

3 Problem Formulation

  • Each instance pairs a marked sheet and English question with a gold answer computed from the oracle process graph Goracle, which is withheld at inference.Source: §3 · p. 3
  • The image-only baseline receives the sheet and question, while the grounded system additionally receives the recovered process graph Gauto and may call a fixed tool library.Source: §3 · p. 3
  • Evaluation reports exact-match accuracy on TopoPID-VQA and graph F1 against Goracle to separate perception and reasoning failures.Source: §3 · p. 3

4 Evidence-Grounded VLM Framework

The framework separates graph recovery from question answering: perception and deterministic geometry build Gauto, then a VLM answers through graph tools under an evidence-citing contract.

4 Evidence-Grounded VLM Framework

  • The answer contract rejects responses without a tool call, a query for each named entity, or matching cited-entity counts on closed counting questions.The final schema includes a typed answer and references to supporting tool steps and entities.Source: §4 · pp. 4–5
  • Stage 1 detects symbols, solid and dashed line masks, and text before deterministic geometry constructs the process graph from predicted solid ink.The reported detection measures include symbol detection F1 0.983, line-mask Dice micro 0.982, and OCR detection F1 0.833.Source: §4 · pp. 3–4
  • Intersection noding and sheet-adaptive snapping join line runs at junctions; a first-symbol walk links symbols only when the drawn process path reaches a shared junction.A fixed 10 px snap reduces exact edge F1 to 0.193 on predicted solids.Source: §4 · p. 4
  • Gauto contains typed symbol nodes, undirected process edges, and line-tag attachments; an optional dashed-line signal-edge layer is evaluated separately from QA.Source: §4 · p. 4
  • Stage 2 gives the planner the sheet, question, and frozen Gauto; mark letters are snapped to graph nodes and question tags are linked through OCR tags before tool use.Source: §4 · p. 4
  • Seven tools return structured graph facts such as neighbors, traversals, paths, distances, and membership, while the VLM performs counting, comparison, and routing over those results.Every topology claim must be fetched through a tool, leaving a trace of the graph facts supporting the answer.Source: §4 · pp. 4–5
  • Recovery on 100 test sheets takes roughly 1.2 seconds for symbol stitching, 4.6 seconds for line inference, and 22.7 seconds for text detection and OCR on paired 16 GB GPUs.The stated profile targets offline batch recovery followed by interactive queries, not low-latency full-sheet chat.Source: §4 · p. 5

5 Experiments

The experiments evaluate graph recovery and topology question answering on Digitize-PID and TopoPID-VQA. Graph grounding improves pooled accuracy across all three tested VLMs, while graph-fidelity scores measure residual substrate errors.

5.1 Experimental Methodology

  • The held-out evaluation uses one 100-sheet test set for perception, graph recovery, and QA, comparing image-only and grounded conditions for three open VLMs.The grounded condition adds Gauto, seven tools, and an answer contract; the graph, tools, and contract are scored as one system.Source: §5.1 · p. 5, §5.3 · p. 6

5.2 Benchmark Suites

  • The 500-sheet Digitize-PID corpus is synthetic, with 350 training, 50 validation, and 100 test sheets; TopoPID-VQA contains three 1,000-question difficulty suites.Each TopoPID-VQA suite uses 100 sheets with 10 questions per sheet.Source: §5.2 · p. 5

5.3 Experimental Setup

  • Gauto is recovered from predicted symbols, solid lines, and OCR using intersection noding and sheet-adaptive snapping; the optional signal layer is not queried by TopoPID-VQA.The reference Goracle is used for scoring and supervision, not as agent input.Source: §5.3 · pp. 5–6, Figure 3 · p. 6

5.5 Results and Analysis

  • +35.9, +33.8, and +37.6 percentage-point pooled exact-match gains over image-only for Qwen3-VL-4B, Qwen3-VL-8B, and Gemma-4-E4B, respectively.Hard remains the weakest difficulty slice for each checkpoint.Source: Table 1 · p. 7, §5.5 · p. 7

5.4 Evaluation Metrics

  • TopoPID-VQA is scored by exact-match accuracy, while exact edge F1 measures matched undirected process connections and connectivity-consistency gives credit for any connecting path.Attributed micro-F1 pools nodes, exact process edges, symbol tags, and line tags.Source: §5.4 · p. 7

5.5 Results and Analysis

  • 0.742 exact edge F1 and 0.801 attributed micro-F1 summarize Gauto fidelity; class-aware symbol F1 is 0.984, while connectivity-consistency is 0.958.Line-tag F1 is 0.730, and symbol-tag F1 is 0.866.Source: §5.5 · p. 7, Table 2 · p. 8
  • 0.610 to 0.824 signal exact F1 is the reported gain from adaptive snapping; TopoPID-VQA evaluates process-connection questions only.Typed process-plus-signal exact F1 is 0.748, and attributed micro-F1 is 0.802.Source: Table 3 · p. 8, §5.5 · p. 8
  • Under the reported protocols, the finetuned perception stack outperforms the tested cloud OCR, object-detection, and zero-shot line-segmentation controls.For example, OCR joint F1 is 0.810 versus 0.779 for Azure Read and 0.663 for Textract.Source: §5.5 · p. 8

6 Limitations

The reported evaluation does not establish performance beyond synthetic Digitize-PID or isolate the contribution of individual harness components.

6 Limitations

  • Digitize-PID is synthetic, so these scores do not measure transfer to scanned plant drawings, handwritten redlines, or site-specific standards.The reported topology QA covers three open VLMs under a bundled grounded condition, without ablations of the harness factors.Source: §6 · p. 8
  • Exact edge F1 on Gauto is 0.742, so an incorrect recovered edge can produce a faithful but incorrect tool answer.The paper reports no calibration or abstention and states that human review remains necessary for safety-relevant use.Source: §6 · p. 8
  • The signal-edge layer is scored as recovered evidence, not as TopoPID-VQA input.The benchmark questions concern process connections only.Source: §6 · p. 8

7 Conclusion

The conclusion presents graph grounding as a way to make topology answers inspectable, while emphasizing that answers still depend on recovered graph fidelity and tool use.

7 Conclusion

  • Grounded access raises pooled TopoPID-VQA accuracy by 35.9, 33.8, and 37.6 percentage points for Qwen3-VL-4B, Qwen3-VL-8B, and Gemma-4-E4B, respectively.Grounded accuracies are 76.0%, 75.1%, and 74.3%, compared with 40.1%, 41.3%, and 36.7% for image-only.Source: §7 · p. 8
  • Gauto scores 0.801 attributed micro-F1, including 0.742 exact edge F1 and 0.958 connectivity-consistency.The conclusion says hard-tier misses and family-level checkpoint gaps indicate that answers track substrate fidelity and multi-step tool use.Source: §7 · p. 8
  • The stated future work includes scanned plant sheets, harness-factor ablations, abstention, and TopoPID-VQA questions over signal edges.Source: §7 · p. 8

A.1 Graph construction

Graph construction keeps learned detections in sheet coordinates and uses deterministic geometric rules to build Gauto. Process edges arise from a first-symbol walk over the noded and snapped line scaffold.

A.1 Graph construction

  • Perception outputs for symbols, solid and dashed line masks, and OCR tags enter a fixed construction order; learned modules do not create edges alone.Tag association uses containment, overlap, proximity, and line-seed propagation along segments of the same line type.Source: §A.1 · pp. 11, 13, Figure 4 · p. 11
  • Endpoints within the per-sheet adaptive radius merge into junctions, each symbol attaches to a nearby segment, and the walk traverses empty junctions until reaching occupied ones.Symbols on the same junction form a clique; the recovered graph attaches symbols to the nearest segment within 120 px of their centers.Source: §A.1 · pp. 11–12
  • The first-symbol walk emits edges only at the first occupied junction on a continuation, not from page proximity or bounding-box overlap.A fixed 10 px snap on predicted solids reduces exact edge F1 to 0.193.Source: §A.1 · pp. 11–12, Figure 5 · p. 12
  • Intersection noding splits or trims solid polylines at tees and corners so branches share exact endpoints before the process-layer walk.Crossings without a terminating tip and fly-overs remain separate rather than becoming connected.Source: §A.1 · pp. 11–12, Figure 6 · p. 12
  • The optional signal-edge layer applies a separate adaptive snap and walk to dashed instrument polylines.Attributed micro-F1 with typed edges is 0.802 for adaptive predicted signal and 0.794 for predicted fixed j=10.Source: §A.1 · pp. 12–13

A.2 Metric definitions

The section defines how perception and topology are scored, distinguishing exact matches from broader reachability and pooled layer-level measures. It also specifies how OCR, line pixels, and boolean answers contribute to evaluation.

A.2 Metric definitions

  • OCR detection F1 scores matched-word geometry, while joint F1 also requires normalized string equality; tag F1 applies the string test to tags linked to matched symbols or line ids.Combined tag F1 is the unweighted mean of symbol-tag and line-tag scores; CER is reported for matched pairs, with lower values better.Source: §A.2 · p. 13
  • Class-aware symbol F1 requires both matched geometry and matching class labels, while class-agnostic F1 uses geometry alone.Headline Digitize scores use class-aware F1; Appendix A.8 API comparisons use class-agnostic F1.Source: §A.2 · p. 13
  • Exact edge F1 counts a mapped predicted pair as a true positive only when it matches an unused ground-truth process edge; unmatched or collapsed endpoints count as false positives.False negatives equal the number of ground-truth edges minus true positives.Source: §A.2 · p. 13
  • Connectivity-consistency scores reachability rather than one-hop identity, so any connecting path can qualify even when local wiring is wrong.Its sheet score is the harmonic mean of reachability precision and recall; exact edge F1 remains stricter.Source: §A.2 · p. 13
  • 0.801 is the attributed micro-F1 across matched nodes, exact process edges, symbol-tags, and line-tags; adaptive predicted signal edges extend the pool to 0.802.TP, FP, and FN are summed across layers and sheets before applying the formulas.Source: §A.2 · p. 13
  • Line-mask Dice and IoU use pooled pixel counts, while solid and dashed channels reuse the same formulas and boolean-item counts treat true as the positive class.Dashed coverage F1 uses many-to-one matching of ground-truth chunks to predicted segments at τ=25 px; reported coverage F1 is 0.993.Source: §A.2 · pp. 13–14

A.3 Perception module scores

The perception modules are trained and evaluated on a held-out Digitize-PID split, using separate systems for symbols, line masks, and text. The supplied tables identify the test-set evaluation scopes but do not provide their score values.

A.3 Perception module scores

  • The three perception heads use Digitize-PID’s 500 full-sheet rasters, split into 350 training, 50 validation, and 100 held-out test sheets; reported scores use test only.The test sheets are not used for weight or graph-knob selection.Source: §A.3 · p. 14
  • The symbol detector is tiled YOLO11s, stitched on full sheets with 20% overlap; Table 5 reports full-sheet stitched-box detection scores on the test set.Table 4 compares symbol detector architectures on the same 100-sheet test set.Source: §A.3 · p. 14, Table 5 · p. 14, Table 4 · p. 14
  • Line masks come from a two-channel U-Net with an EfficientNet-B0 encoder, and Table 6 reports test-set pixel scores for the line U-Net.The MobileNetV2 encoder is the ablation; solid and dashed masks are trained as separate channels.Source: §A.3 · p. 14, Table 6 · p. 15
  • Text boxes use YOLO11n and word recognition uses fine-tuned TrOCR; Table 7 reports test-set text detection and OCR scores.Detection-only and ground-truth-crop comparisons separate box error from recognition error.Source: §A.3 · p. 14, Table 7 · p. 15

A.4 VLM decoding and run protocol

The appendices detail the evaluation protocol, graph-grounded inference, benchmark construction, and diagnostic analyses. Together they show gains across answer types alongside remaining errors tied to graph recovery and model-specific weaknesses.

A.6 Topology overclaim proxy

  • Grounding lowers pooled fabricated-true rates for all three models, while 45.8%, 33.4%, and 52.6% of their incorrect harness answers retain nonempty evidence_ids.The fabricated-true measure is an overclaim rate on gold-false topology questions, not overall boolean error.Source: §A.6 · pp. 15–16, Table 8 · p. 16

A.7 Graph ablations and compute

  • Adaptive junction snap is essential: fixing j=10 drops exact edge F1 from 0.742 to 0.193, while omitting intersection noding reduces it to 0.729.With ground-truth boxes and lines, topology exact F1 is 1.000; stock cloud and zero-shot vision controls also trail the tuned stack on diagram perception.Source: §A.7 · p. 16, Table 9 · p. 16, §A.8 · p. 17

A.9 Grounded Inference Details

  • The planner uses seven fact-returning operators on Gauto, must make at least one tool call, and cites supporting graph entities under a fixed answer contract.Topology reads use graph adjacency; breadth-first traversal returns hop distances, while edge check returns a boolean verdict.Source: §A.9 · pp. 18–19, Figure 8 · p. 18

A.10 TopoPID-VQA question generation

  • TopoPID-VQA contains 3,000 items sampled from 28 question families across 100 sheets, with gold answers computed on Goracle and hidden at inference.The frozen suite uses template wording; prompts were fixed before evaluation, and the gold-query ceiling is an upper bound rather than a deployable method.Source: §A.10 · p. 19, §A.11 · p. 20, §A.14 · pp. 21–22

A.12 TopoPID-VQA answer distributions and chance baselines

  • Image-only pooled accuracy of 36.7–41.3% falls below the 48.1% majority floor and near the 35.6% random floor; grounded harness accuracy is 74.3–76.0%.The gold-query ceiling reaches about 88.9% pooled, so the suite remains unsolved; the passage attributes the remaining gap to substrate error and multi-step tool use.Source: §A.12 · p. 21, Table 21 · p. 21

A.15 TopoPID-VQA accuracy by answer type

  • Harness gains extend beyond booleans: integer accuracy rises into the 70–90 range, and Qwen3-VL-4B string accuracy roughly doubles versus image-only prompting.Hard booleans remain the weakest harness slice for Qwen3-VL-4B at 63.4%.Source: §A.15 · p. 22, Table 22 · p. 22

A.16 TopoPID-VQA boolean metrics

  • For the boolean subset, harness pooled F1 is 73.3% for Qwen3-VL-4B, 74.9% for Qwen3-VL-8B, and 78.2% for Gemma-4-E4B, versus image-only 44.7%, 52.6%, and 53.0%, respectively.Source: §A.16 · p. 22

A.18 Qualitative overlays

  • The overlays trace the pipeline from raster and perception masks through vectorized line geometry to recovered process connections and a dual-layer graph with signal edges.The dashed-ink zoom highlights masks and vectorized dashed segments.Source: §A.18 · p. 25, Figure 9 · p. 25, Figure 10 · p. 25
Loading 2609.05880v1…