Source-linked AI summary
CVIF: A Criticality-Driven Visual Intervention Framework for Geometric Diagram Understanding in MLLMs
Jiahui Kang, Bifan Wei, Lingling Zhang, Tianwen Jiang, Qiuyong Xiao, Jihong Zhang, Jun Liu
TL;DR
Geometric diagram understanding remains challenging because MLLMs may rely on textual priors rather than visual evidence. CVIF intervenes at selected layers, steering attention toward vertex-centered geometric cues. It improves Overall F1 over Qwen3-VL-32B-Instruct on both PGPS9K and PGDP5K.
Problem
MLLMs may infer geometric information from textual context rather than genuinely perceiving diagrams, complicating the separation of perception errors from reasoning failures.
Source: §1 · pp. 1–2Method
CVIF uses layer-wise prediction diagnostics to select intervention layers and directs attention toward visual evidence organized around relevant geometric vertices.
Source: §1 · p. 2, §3.2 · p. 4, §4.1 · p. 4Results
Overall F1 rises from 77.85 to 85.58 on PGPS9K and from 75.23 to 82.84 on PGDP5K, compared with Qwen3-VL-32B-Instruct.
Source: §5.2 · p. 7Takeaways & Limitations
CVIF improves both individual predicate recovery and diagram-level completeness and correctness on the evaluated geometric parsing benchmarks.
Source: §5.3 · p. 8, §6 · p. 9
Abstract
from arXiv · showhide
Despite significant progress in visual tasks by Multimodal Large Language Models (MLLMs), geometric diagram understanding remains challenging due to the presence of sparse visual cues and ambiguous symbol-primitive associations. MLLMs may therefore rely on textual priors, producing interpretations that conflict with visual evidence. We introduce the training-free Criticality-Driven Visual Intervention Framework (CVIF), an inference-time method that localizes critical layers and executes visual interventions during the transition from evidence aggregation to semantic decoding. At these layers, a Geometry-Constrained Local Relation Reconstruction (GCLR) module selects and weights vertex-centered visual evidence, while an Adaptive Visual Steering Operator (AVSO) redistributes attention mass toward the selected tokens. Experiments on PGPS9K and PGDP5K show that CVIF raises Overall F1 from 77.85 to 85.58 and from 75.23 to 82.84, respectively, establishing a novel inference-time visual intervention paradigm.
1 INTRODUCTION
The introduction frames geometric diagram understanding as a perception problem: MLLMs can favor textual priors over visual evidence, and perception errors are a primary source of reasoning failures. CVIF intervenes inside the model at critical locations, steering attention toward relevant geometric cues.
Motivation
- End-to-end geometric problem-solving benchmarks conflate perception errors with reasoning failures, while diagnostic studies identify perception errors as a primary source of failure.The introduction argues for decoupling diagram parsing from logical reasoning to enable objective evaluation.Source: §1 · p. 1
Failure analysis
- Deeper layers may overlook early visual cues and focus on symbols rather than geometric structure, allowing prior-consistent interpretations to gain influence.The example of angle assignment shows that recognizing a label or number alone does not recover the relations among incident edges.Source: §1 · p. 1, Figure 1 · p. 2
Motivation
- Once visual cues lose influence in internal representations, output-level correction alone cannot reliably restore them, motivating intervention inside the model.The paper contrasts this with external prompting and post-hoc correction.Source: §1 · p. 2
Contributions
- CVIF uses layer-wise prediction diagnostics to select intervention layers and steers attention toward vertex-guided visual features.Its proposed components are GCLR for reconstructing local topological relations and AVSO for reshaping multi-head attention distributions.Source: §1 · p. 2
2 RELATED WORK
Related work describes GPS benchmarks and diagram-parsing approaches as ways to study geometric reasoning and separate perception from reasoning. It also connects layer-wise representation analysis and energy-based views of attention to the study of visual grounding.
Geometric Problem Solving and Geometry Diagram Parsing
- End-to-end GPS benchmarks can conflate perception errors with reasoning failures, while diagram parsing translates diagrams into formal languages to decouple perception from reasoning.The section notes that sparse, abstract topological features and textual-prior reliance complicate geometric diagram interpretation.Source: §2 · pp. 2–3
Mechanistic Interpretability and Representation Dynamics
- Mechanistic interpretability examines neural-network computations, including depth-dependent representations and feed-forward networks as key-value memories contributing to vocabulary predictions.Prior work also describes inference stages and offers an energy-based perspective on associative retrieval.Source: §2 · p. 3
- Studies of multimodal models report reliance on textual priors and limitations in visual grounding, motivating investigation of where visual evidence can be reinforced during prediction formation.The section connects this motivation to layer-wise analyses and energy-based accounts of attention.Source: §2 · p. 3
3 ANALYSIS OF INTERVENTION LOCATIONS AND GEOMETRIC CUES
The analysis treats inference as movement toward stable semantic states and identifies a critical interval when visual evidence can still alter competing interpretations. It then motivates vertex-centered cues because local geometric relations are distributed across vertices, edges, and annotations.
3.1 CRITICAL LOCATIONS FOR VISUAL INTERVENTION
- Intervening too early precedes a relevant semantic judgment, while intervening too late may leave the prior-favored interpretation stable and difficult to redirect.The energy-landscape account describes competing interpretations as driven by textual priors and local visual evidence.Source: §3.1 · p. 3, Figure 2 · p. 3
- The entropy trajectory moves from multimodal evidence aggregation through nonlinear semantic decoding to low-entropy prediction consolidation, and its inflection points guide critical-layer selection.The method uses linear fitting on the entropy curve to identify intervention layers.Source: §3.1 · p. 4
3.2 VERTEX-CENTERED GEOMETRIC CUES
- Because geometric relations jointly involve vertices, thin edges, and nearby annotations, vertex-centered representations can organize local evidence for relation recovery.GCLR uses a vertex as a spatial anchor to identify incident edges and nearby annotations, then forms target tokens and a topological weight matrix.Source: §3.2 · p. 4, Figure 3 · p. 4
4 METHOD
CVIF intervenes at entropy-identified critical layers: GCLR forms vertex-centered token targets and weights, while AVSO steers attention toward selected visual evidence.
4.1 CRITICALITY-DRIVEN VISUAL INTERVENTION FRAMEWORK
- Entropy identifies critical layers; there GCLR derives target tokens and topology-aware weights from historical attention and detected vertices, and steered attention feeds Value features.Layers outside the critical set retain the native feed-forward path.Source: §4.1 · pp. 4–5
4.2 GEOMETRY-CONSTRAINED LOCAL RELATION RECONSTRUCTION
- GCLR scores vertices using spatial priors and historical attention, then combines representative nearby tokens with an anchor token into weighted targets.The candidate set for each vertex includes representative tokens and a vertex-centered anchor.Source: §4.2 · pp. 5–6
4.3 ADAPTIVE VISUAL STEERING OPERATOR
- AVSO compares current and historical attention, selectively reshapes target-token attention for under-aligned visual heads, and compensates text attention before normalization.It uses GCLR’s target tokens and topology-aware weights without modifying model parameters.Source: §4.3 · pp. 6–7
5 EXPERIMENT
The experiments evaluate CVIF on two diagram-parsing datasets, then test diagram-level accuracy, ablate its components, examine representative parses, and assess downstream geometry reasoning. CVIF improves Overall F1 on both parsing datasets, while the downstream results vary by representation and reasoning model.
5.1 EXPERIMENTAL SETUP
- Evaluation uses 2,650 image–formal language pairs from each dataset, converts PGDP5K annotations to PGPS format, and applies Qwen3-VL-32B-Instruct as the intervention backbone.Precision, recall, and F1 are reported for structural, semantic, and overall predicates.Source: §5.1 · p. 7
5.2 DIAGRAM PARSING RESULTS
- CVIF raises Overall F1 over Qwen3-VL-32B-Instruct from 77.85 to 85.58 on PGPS9K and from 75.23 to 82.84 on PGDP5K.Overall precision and recall also rise by 5.35 and 9.95 points on PGPS9K, and by 4.84 and 10.15 points on PGDP5K.Source: Table 1 · p. 7, §5.2 · pp. 7–8, Table 2 · p. 8
5.3 DIAGRAM-LEVEL EXACT MATCH EVALUATION
- CVIF leads evaluated open-source configurations on all Table 3 metrics, raising PPR from 43.47 to 59.47 on PGPS9K and from 40.11 to 55.77 on PGDP5K.Sample Accuracy requires all predicates within a category to match; PPR requires all predicates across a diagram to be correct.Source: Table 3 · p. 8, §5.3 · p. 8
5.4 ABLATION STUDY
- The two-stage pipeline reaches 79.45 Overall F1 versus CVIF’s 85.58; fixed-layer intervention reaches 79.86, Top-K selection 81.28, and direct attention amplification 80.52.The ablations replace criticality-guided placement, GCLR, or AVSO, respectively.Source: Table 4 · p. 8, §5.4 · p. 8
5.5 CASE ANALYSIS
- Across representative diagrams, CVIF recovers vertex-centered topology, including angle assignments, line connectivity, collinearity, and perpendicular relations, where the baseline and two-stage pipeline often err or miss relations.The examples also show incorrect entity attributes in the baseline and two-stage pipeline.Source: Figure 6 · p. 9, §5.5 · p. 9
5.6 DOWNSTREAM GEOMETRY REASONING
- On MathVerse’s Vision Dominant Plane Geometry split, natural-language descriptions improve accuracy for all five reasoning models by 1.76 to 5.88 points, while formal-language representations improve four models.The preferred representation varies by backbone.Source: Table 5 · p. 9, §5.6 · p. 9
6 CONCLUSION
The conclusion presents CVIF as a training-free, inference-time intervention for textual-prior bias in geometric diagram understanding. It combines critical-layer selection with vertex-guided visual evidence and reports improved diagram parsing.
6 CONCLUSION
- CVIF is a training-free framework proposed to mitigate textual-prior bias in MLLMs’ geometric diagram understanding.The conclusion characterizes it as a novel inference-time visual intervention paradigm for mitigating visual hallucinations.Source: §6 · p. 9
- An energy-landscape perspective motivates reinforcing local visual evidence at representational critical states, before competing interpretations become stable.Layer-wise prediction diagnostics select candidate critical layers for intervention.Source: §6 · p. 9
- GCLR uses geometric vertices to reconstruct spatial-semantic mappings, while AVSO reshapes attention toward vertex-guided visual evidence.The reported experiments show improved diagram-parsing performance with the intervention.Source: §6 · p. 9
V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale
The supplied passages are bibliography fragments rather than narrative exposition. They list works on multimodal models, geometry parsing, and diagram-based reasoning.
V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale
- A listed work is GLM-4.5v and GLM-4.1v-thinking, titled “Towards versatile multimodal reasoning with scalable reinforcement learning.”The entry identifies it as a 2025 arXiv preprint.Source: V Team, Wenyi Hong, Wenmeng Yu,… · p. 11
- The bibliography includes GeoParse, a unified formal-language approach to diagram parsing for plane and solid geometry.The entry is listed in Findings of ACL 2026.Source: V Team, Wenyi Hong, Wenmeng Yu,… · p. 12
- MathVerse is listed as a study asking whether multimodal LLMs truly see diagrams in visual math problems.The entry identifies it as an ECCV 2024 paper.Source: V Team, Wenyi Hong, Wenmeng Yu,… · p. 12