When Can Text Replace Vision? Structural Bottlenecks in Diagram Reasoning

1 INTRODUCTION

• A wrong answer after textualization can reflect either missing question-critical information or a solver’s failure to use information that is present.Whole-graph fidelity and end-task accuracy alone do not distinguish these explanations.Source: §1 · pp. 1–2
• Gold structure reaches 87% accuracy, while direct vision and learned text both remain below 30% on the reserved 240-diagram FlowGen holdout.The aggregate comparison includes source-derived relation labels that may not be printed and uses different learned and gold graph encodings.Source: §1 · p. 2
• Retrying only invalid public extractions makes nearly all representations schema-valid but leaves QA accuracy essentially unchanged.The public accuracy comparison does not isolate extraction error alone because it includes source-derived labels and different graph encodings.Source: §1 · p. 2
• The learned-text deficit relative to gold more than doubles across node-count bins and grows with branch depth.This public holdout pattern survives validity recovery; controlled recovery largely removes the low-budget deficit in a separate diagnostic.Source: §1 · p. 2
• Question-relevant topology predicts QA better than whole-graph topology, while matched interventions show that error location matters beyond edit count.For the primary solver, changing one answer-relevant edge nearly eliminates original-answer accuracy, whereas matched irrelevant edits largely preserve it.Source: §1 · pp. 2–3
• A fixed-solver input ladder and validity recovery separate the utility of supplied structure from failures in acquiring it.The learned–gold gap also widens with structural difficulty on the reserved holdout.Source: §1 · p. 2

2.1 FIXED-SOLVER INPUT LADDER

• The fixed-solver ladder compares answers from the original image, question-blind extracted structure, and source-derived structure.Learned and gold inputs use different graph encodings, so their utility gap is not a controlled measure of extraction error alone.Source: §2.1 · p. 3

2.2 VALIDITY RECOVERY AND STRUCTURE-SENSITIVE EVALUATION

• Validity recovery retries only schema-invalid extractions, while question-relevant fidelity compares topology exact match and edge F1 on the same learned representations.The recovery policy retains valid but semantically incorrect extractions and does not use the question, answer, or downstream QA to select text.Source: §2.2 · p. 3
• End-to-end QA counts invalid acquisitions as failures, whereas matched-valid fidelity prediction compares properties of representations that can be evaluated.Using identical cases for relevant and whole-graph metrics prevents differences in validity coverage from driving that comparison.Source: §2.2 · p. 3
• Matched interventions compare answer-changing edits with answer-preserving edits of the same type and count, testing the placement rather than nominal size of corruption.This exposed mechanism study measures sensitivity to deliberate edits, not their contribution to the natural acquisition gap.Source: §2.2 · pp. 3–4

2.3 CONFIRMATORY CRITERIA AND COST

• Confirmatory tests specify acquisition-gap trends with node count and branch depth, relevant-topology prediction, and a −2-point margin for text comparability with vision.Comparability requires the lower paired 95% confidence bound for Atext − Adirect to exceed −2 percentage points; non-significance alone is insufficient.Source: §2.3 · p. 4
• The cost analysis includes learned-representation acquisition tokens and solver tokens, reporting single-question use and reuse across three observed questions.Gold is a solving-only reference with zero modeled acquisition cost because its structure is supplied.Source: §2.3 · p. 4

3 EXPERIMENTAL SETUP

• The public holdout contains 240 previously unevaluated FlowGen charts and 720 deterministic source-derived questions across four query families.The questions cover incoming and outgoing neighbors, source edge relations, and shortest-path length; source answers may include relations not explicitly labeled in images.Source: §3 · p. 4
• The primary extractor and solver are Qwen3.5-122B-A10B, with fixed extraction and solving settings; the secondary 27B solver receives the same fixed-extraction strings.The common solving cap was selected using exposed checks before holdout evaluation, and extraction does not access the downstream question.Source: §3 · p. 4
• Separate controlled cohorts use program-flowchart and Boolean-circuit tasks, while exposed interventions test three solver sizes on a preselected chart subset.The controlled tasks ask for program or circuit outputs; their recovery results are not pooled with public FlowGen.Source: §3 · pp. 4–5
• QA uses normalized exact match, invalid acquisitions count as failures, and chart-cluster bootstrap intervals use 10,000 resamples.Hypotheses, policies, bins, comparability margin, and analysis were frozen before holdout access.Source: §3 · p. 5

4.1 A LARGE PUBLIC ACQUISITION GAP SURVIVES VALIDITY RECOVERY

• The public aggregate includes source-derived relation labels that may not be printed in images, so its gap is not a pure image-extraction effect.Learned and gold inputs also use different graph encodings.Source: §4.1 · p. 5
• 87.1% QA for gold versus 22.8% for fixed learned text on the 240-chart public holdout shows a large supplied-structure advantage.Direct vision scores 27.2–28.8%; the measured gap reflects both acquisition and representation–solver compatibility because learned and gold inputs use different graph encodings.Source: Table 1 · p. 5, §4.1 · p. 5
• Recovery raised schema-valid public representations from 196/240 to 238/240 charts but improved accuracy by only 0.7 points.In a separate controlled diagnostic, recovery made all 541 representations valid and brought learned QA near direct vision; that confirmation was ceiling-limited.Source: §4.1 · pp. 5–6, Figure 2 · p. 6

4.2 STRUCTURAL DIFFICULTY WIDENS THE ACQUISITION GAP

• The learned–gold deficit grows from 38.8 points on charts with at most ten nodes to 85.2 points on charts with 21–40 nodes.The paired deficit also grows with branch depth, and recovery barely changes the pattern.Source: §4.2 · p. 6, Figure 3 · p. 7

4.3 QUESTION-RELEVANT TOPOLOGY PREDICTS DOWNSTREAM CORRECTNESS

• Question-relevant topology exact match predicts QA better than whole-graph exact match, reducing out-of-fold Brier loss from 0.1406 to 0.1147.This comparison uses matched valid cases; the advantage is metric-dependent and does not establish improvement for edge F1.Source: Table 2 · p. 7, §4.3 · p. 7

4.4 MATCHED INTERVENTIONS REVEAL THE EFFECT OF ERROR LOCATION

• A single relevant edge edit yields 0–3.5% QA for 122B, while matched irrelevant edits preserve 92.4–93.9%.Reversal, deletion, and redirection show the same pattern with one or two edits; the study tests sensitivity to deliberate corruption, not its share of the natural acquisition gap.Source: §4.4 · pp. 7–8, Table 3 · p. 8

4.5 COST AND SOLVER COMPATIBILITY

• At K = 1, learned acquisition makes both learned policies more token-intensive, although supplied gold structure requires 81.8% fewer solving tokens than direct vision.Reuse across three questions lowers usage without repairing missing information.Source: §4.5 · p. 8
• On identical fixed extractions, the 27B-minus-122B holdout difference is −0.97 points (95% CI [−2.22, 0.28]), while QZhou comparisons show no established superiority.QZhou scores 69.8% for direct vision, 67.7% with source structure, and 71.0% for learned extraction under a 2,048-token cap; these are numerical differences without an established superiority claim.Source: §4.5 · p. 8

5 RELATED WORK

• Structured interfaces recover figure data, support chart comprehension, and combine textual, structural, and visual evidence for fact-checking.DePlot separates plot-to-table conversion from reasoning, while Pix2Struct and MatCha use parsing or derendering in pretraining.Source: §5 · p. 8
• FlowGen evaluates graph properties and triplet fidelity, but its illustrated triplet-assisted QA prompts retain the image, unlike this paper’s text-only learned and gold conditions.The paper also tests difficulty-dependent utility gaps, validity recovery, relevant-support metrics, and matched interventions.Source: §5 · p. 9
• Prior visual-dependence evaluations include question-conditioned descriptions, image-retaining oracles, and question-agnostic descriptions without a source-derived textual reference.The paper instead uses question-blind extraction and text-only source references to separate supplied-structure utility from acquisition.Source: §5 · p. 9

6 CONCLUDING DISCUSSION

• On the reserved FlowGen holdout, the learned–gold gap widens with chart size and branch depth, and question-relevant topology predicts QA better than whole topology.Matched interventions show that error location matters beyond edit count.Source: §6 · p. 9
• Useful textual representations must retain question-needed relations, and evaluation should consider intended-question support, validity, and end-to-end cost.Practical substitution also requires savings at comparable accuracy.Source: §6 · p. 9
• Confirmation is limited to FlowGen’s Diagrams renderer, source-generated questions, and one model family; pretraining exposure is unknown.Different learned and gold encodings, including source-derived relation targets that may not appear in images, confound a pure extraction-error interpretation.Source: §6 · p. 10

APPENDIX

• The appendix documents experimental design, recovery, fidelity prediction, perturbations, cost accounting, secondary results, solver-cap calibration, and human audits.Recorded extraction examples are included in Appendix K.Source: APPENDIX · p. 14

A DATASETS AND TASK CONSTRUCTION

• The public FlowGen holdout contains 240 unchanged Diagrams-renderer charts, balanced across easy, medium, and hard strata, with three source-based questions per chart.Questions cover direct predecessors, direct successors, source relations, or shortest-path length; source relations may not be printed in the image.Source: A DATASETS AND TASK CONSTRUCTION · p. 14
• OLD271 comprises executable flowcharts and Boolean circuits; NEW270 adds harder flowcharts, while QZhou is a secondary exposed sample with native Chinese questions.The controlled tasks use source execution for answers; QZhou results are kept secondary and do not verify its entire cohort.Source: A DATASETS AND TASK CONSTRUCTION · p. 14

B MODEL SETTINGS AND COMPARISON BOUNDARIES

• The direct-image and text-only conditions use the same solver and cap, while the extractor sees the image without the question or answer.Learned and gold graph encodings differ: learned inputs use entity and relation objects, whereas gold inputs embed labeled source triplets in a JSON string.Source: B MODEL SETTINGS AND COMPARISON… · p. 15, §B.1 · p. 16

B.1 EXACT PUBLIC INPUT CONTRACTS AND EXAMPLES

• The public gold structure provides source topology and labels, not all visual attributes requested from the extractor, so learned–gold utility differences are not pure extraction-error effects.The two constructions jointly differ in acquired content and representation–solver compatibility.Source: §B.1 · p. 16
• The learned textualization contract requires image-grounded visible primitives in a fixed schema, forbids solving or adding answer-bearing fields, and withholds the downstream query.The direct-image and text-only solver prompts specify their respective image or representation inputs and require a compact answer output.Source: QUESTION-BLIND EXTRACTOR · p. 17, DIRECT-IMAGE SOLVER · p. 17, TEXT-ONLY SOLVER: LEARNED AND GOLD · p. 18

C EVALUATION DESIGN AND COHORTS

• All three prespecified holdout tests support their specified directions: the learned–gold QA gap worsens with node count and branch depth, and relevant topology lowers Brier loss versus whole topology.Table 8 reports the paired estimates and chart-cluster intervals for these primary tests.Source: C EVALUATION DESIGN AND COHORTS · p. 19, Table 8 · p. 21

D SCORING, ESTIMANDS, AND MISSING OBSERVATIONS

• Invalid acquisitions count as pipeline failures, rejected responses are unknown, and paired estimates use complete pairs with separate full-sample missing-outcome bounds.The bootstrap moves questions and conditions from each chart together.Source: D SCORING, ESTIMANDS, AND MISSING… · p. 21
• Gold meets the two-point paired comparability criterion against direct vision, while fixed and recovered learned text do not.Gold also declines more slowly than direct vision across node-count and branch-depth bins.Source: D SCORING, ESTIMANDS, AND MISSING… · p. 21

D.1 LENIENT ANSWER-SET SCORING

• The gold–direct–learned ordering survives post-hoc partial-credit scoring for neighbor sets, while relations and scalar shortest-path answers retain exact scoring.Invalid-input skips remain zero and rejected direct outputs remain missing.Source: §D.1 · pp. 21–22

E RECOVERY POLICY AND ACQUISITION ACCOUNTING

• Recovery retries only schema-invalid outputs, so it tests bounded format recovery rather than uniform re-extraction at the largest budget.Valid but semantically incorrect extractions are retained.Source: E RECOVERY POLICY AND ACQUISITION… · p. 23
• On the reserved holdout, recovery raises valid charts from 196 to 238 while QA changes from 164/720 to 169/720.The two remaining invalid acquisitions end in unterminated JSON and are retained as six intended QA failures.Source: E RECOVERY POLICY AND ACQUISITION… · p. 23

• Directed topology exact requires matching node and directed-edge sets, while directed-edge F1 compares ordered endpoint pairs and labeled-edge F1 compares source-label-target triples.Direction and label metrics are reported separately.Source: F FIDELITY DEFINITIONS AND PREDICTION… · p. 23
• Question-relevant support masks select incident edges, a labeled ordered pair, or a conservative shortest-path certificate; unsupported or global questions retain the whole graph.The masks are offline, withheld from extractor and solver, and may exceed a minimal sufficient subgraph.Source: F FIDELITY DEFINITIONS AND PREDICTION… · p. 23
• −0.0184 recovered-topology Brier difference (descriptive 95% CI: [−0.0346, −0.0029]); relevant labeled-edge F1 also improves prediction on valid source-relation cases.Source: Figure 8 · p. 24, F FIDELITY DEFINITIONS AND PREDICTION… · p. 24
• Matched one-feature logistic models use identical cases and chart-held-out folds; Brier loss is primary, with AUC secondary.The primary analysis covers 588 fixed-learned schema/adapter-valid observations; all-intended sensitivity analysis covers 720 questions.Source: F FIDELITY DEFINITIONS AND PREDICTION… · p. 24, Figure 8 · p. 24

G COMPLETE EXPOSED PERTURBATION RESULTS

• Matched relevant and irrelevant edits vary corruption placement while holding edit type and count fixed; the intervention effect is conditional on selected edits and does not estimate their contribution to the natural acquisition gap.Relevant edits alter the original question’s answer semantics; irrelevant edits leave them unchanged.Source: G COMPLETE EXPOSED PERTURBATION RESULTS · p. 24, Figure 9 · p. 25

H CONTROLLED POST-CONFIRMATION DIAGNOSTIC

• +0.49 pp recovered-minus-direct QA on flowcharts (95% CI [−0.74, 1.72]) and zero on circuits ([−2.22, 2.22]); recovery made all 541 controlled charts valid, versus 111/406 flowcharts and 116/135 circuits at fixed 1,024.The controlled task and public holdout are different populations and question constructions and are not pooled.Source: H CONTROLLED POST-CONFIRMATION… · p. 25, Table 9 · p. 25

I TOKEN, CALL, AND LATENCY ACCOUNTING

• 9,484,994 tokens covered the holdout’s actual prompt and completion usage, with 3,032 physical calls and 70.81 allocated GPU-hours over approximately 4.25 wall-clock hours.Accounting distinguishes acquisition from solving; three-question reuse amortizes acquisition, while invalid representations retain acquisition cost.Source: I TOKEN, CALL, AND LATENCY ACCOUNTING · pp. 25–26, Figure 10 · p. 26

J QUESTION-FAMILY RESULTS AND SECONDARY SCOPE

• 66.7% shortest-path gold QA falls below the 70% floor for structural-bin gap interpretation, though occupied node and depth bins remain above it.Source edge relation gold QA is 99.0%, with label-sensitive fidelity reported separately.Source: J QUESTION-FAMILY RESULTS AND… · p. 26, Figure 11 · p. 27

J.1 SOURCE EDGE RELATION VISIBILITY AND LITERAL-LABEL AUDIT

• Image spot checks found unlabeled links referenced as connectedTo and containment represented as partOf, so source-triplet correctness is not equivalent to recovering an explicitly printed relation.The frozen generator asks for source relation labels without checking whether they appear in the image; 63 relation questions had other labels without exhaustive visibility audit.Source: §J.1 · p. 27
• 123 of 124 connectedTo answers were correct in the recorded gold arm; direct, fixed learned, recovered learned, and fixed 27B arms answered none correctly on that subset.The aggregate and paired results are retained; interpretation includes source-annotation access as well as acquisition and representation–solver effects.Source: §J.1 · p. 27

J.2 QZHOU AS A SECONDARY COUNTERPOINT

• On QZhou’s global/counting slice, source-derived text trails direct vision, while a larger learned-extraction cap changes the learned arm’s ordering; this is a recorded-cap interaction.The comparison uses a 200-chart cohort, a 512-token solver cap, and a taxonomy slice that includes global graph queries as well as counting.Source: §J.2 · pp. 27–28

K QUALITATIVE EXAMPLES: WHICH STRUCTURE SURVIVES EXTRACTION?

• The three public-holdout examples illustrate contrasting relations among valid JSON, answer-critical structure, and correct answers; they were selected to explain outcomes, not estimate their frequency.Source: K QUALITATIVE EXAMPLES: WHICH… · p. 28

K.1 A VALID EXTRACTION CAN OMIT THE EDGE NEEDED FOR THE ANSWER

• The valid 422-token extraction omitted Underwriting → Legal Review of Claims; learned text returned only Loss Prevention, unchanged by recovery, and relevant topology exact was zero.The answer followed the extracted outgoing neighbors rather than the source structure.Source: §K.1 · pp. 28–29, Figure 13 · p. 29

K.2 AN IMPERFECT WHOLE GRAPH CAN PRESERVE EVERYTHING THIS QUESTION NEEDS

• Direct, fixed learned, recovered learned, and gold all returned the two correct incoming neighbors even though the learned graph reversed unrelated edges; its question-relevant topology was exact.This example’s whole-graph directed-edge F1 matched the K.1 example, illustrating that whole-graph error alone does not show whether an answer is affected.Source: §K.2 · p. 30, Figure 14 · p. 30

K.3 RECOVERY CAN COMPLETE THE JSON WITHOUT REPAIRING THE RELEVANT STRUCTURE

• Gold was correct while direct and recovered learned answers were incorrect; recovery produced schema-valid JSON but relevant directed-edge F1 remained 0.500 and relevant topology exact remained zero.Whole-graph edge F1 was 0.738; the recovered answer matched its extracted outgoing-neighbor set despite missing and spurious destinations.Source: Figure 15 · p. 31, §K.3 · p. 31

L SOLVER-CAP CALIBRATION AND RESIDUAL OUTPUT HITS

• Path QA changed −1.23 pp on 162 complete pairs from 4,096 to 8,192 tokens (95% CI [−6.79, 4.32]); local-neighbor and source-relation cells remained 40% and 35%, respectively.Each of those two cells still had 1/20 cap hits at 8,192; the protocol makes no claim that every question type is saturated.Source: L SOLVER-CAP CALIBRATION AND RESIDUAL… · p. 32

M HUMAN AUDITS OF SECONDARY EVIDENCE

• Three reviewers independently assessed 30 model-flagged QZhou charts and 150 historical query-conditioned textualizations using image, question, text, and contract evidence.A contract violation required prohibited downstream task content, such as supplying an answer; malformed syntax alone did not count as leakage.Source: M HUMAN AUDITS OF SECONDARY EVIDENCE · pp. 32–33
• Both audit samples were enriched for suspected issues, so their outcomes cannot estimate population rates.The QZhou packet came from a model-flagged discrepancy subset, while the leakage packet was stratified across ordinary and flagged or uncertain outputs.Source: M HUMAN AUDITS OF SECONDARY EVIDENCE · p. 32, Table 12 · p. 33
• Among 29 adjudicable QZhou charts, 28 matched the source and one differed cosmetically; 88 of 90 question checks showed no observed answer-affecting discrepancy.One chart and two question-level checks remained unresolved.Source: M HUMAN AUDITS OF SECONDARY EVIDENCE · p. 33, Table 12 · p. 33
• Five of 115 reviewable query-conditioned outputs violated the contract, 110 showed no observed violation, and 35 were unreviewable.Unreviewable outputs remain unknown, and the five violations describe this enriched sample rather than a population leakage rate.Source: M HUMAN AUDITS OF SECONDARY EVIDENCE · p. 33, Table 12 · p. 33
• All query-conditioned outputs remained excluded from confirmation, and the audit did not alter the question-blind holdout or frozen score.No rescoring or new inference was performed.Source: M HUMAN AUDITS OF SECONDARY EVIDENCE · p. 33
• Initial reviewer unanimity was 83.3% for QZhou charts and 88.0% for leakage items; several question-level and leakage-stratum outcomes were not reported.The summary also does not report why 35 outputs were unreviewable or whether the five violations reached a solver.Source: M HUMAN AUDITS OF SECONDARY EVIDENCE · p. 33