Source-linked AI summary
See Less, See Right: Bi-directional Perceptual Shaping For Multimodal Reasoning
Shuoshuo Zhang, Yizhen Zhang, Jingjing Fu, Lei Song, Jiang Bian, Yujiu Yang, Rui Wang
TL;DR
VLMs still struggle to identify and read fine-grained visual evidence, while visual-cue approaches can be shape-rigid, domain-specific, and costly at inference. BiPS uses question-conditioned evidence-preserving and evidence-ablated views to shape perception during training. With added math-focused GRPO training, it reports an 8.2% average improvement over Qwen2.5-VL-7B across eight benchmarks.
Problem
VLMs have difficulty identifying, localizing, and accurately reading fine-grained visual cues, while existing visual-cue approaches can be shape-rigid, domain-specific, and costly at inference.
Source: §1 · p. 1Method
BiPS trains VLMs with KL consistency toward question-conditioned evidence-preserving views and KL separation from evidence-ablated views.
Source: §1 · p. 2Results
+8.2% average improvement over Qwen2.5-VL-7B across eight chart and general VQA benchmarks is reported after adding 39K math-focused samples with GRPO.
Source: §1 · p. 2Takeaways & Limitations
BiPS can generalize from chart-derived training cues to real-world figures and out-of-domain general VQA benchmarks.
Source: §1 · p. 2, §5 · p. 9Because BiPS is trained exclusively on chart data, transferring fine-grained perceptual supervision to diverse visual domains is harder, and its gains are bounded when target domains differ.
Source: §7.2 · p. 13
Abstract
from arXiv · showhide
Large vision-language models (VLMs) often benefit from intermediate visual cues, either injected via external tools or generated as latent visual tokens during reasoning, but these mechanisms still overlook fine-grained visual evidence (e.g., polylines in charts), generalize poorly across domains, and incur high inference-time cost. In this paper, we propose Bi-directional Perceptual Shaping (BiPS), which transforms question-conditioned masked views into bidirectional where-to-look signals that shape perception during training. BiPS first applies a KL-consistency constraint between the original image and an evidence-preserving view that keeps only question-relevant regions, encouraging coarse but complete coverage of supporting pixels. It then applies a KL-separation constraint between the original and an evidence-ablated view where critical pixels are masked so the image no longer supports the original answer, discouraging text-only shortcuts (i.e., answering from text alone) and enforcing fine-grained visual reliance. Across eight benchmarks, BiPS boosts Qwen2.5-VL-7B by 8.2% on average and shows strong out-of-domain generalization to unseen datasets and image types.
1. Introduction
Fine-grained visual perception remains a VLM bottleneck, while prior visual-cue approaches face limits in evidence shape, domain transfer, and inference cost. BiPS instead shapes perception during training with paired visual views, and its reported gains extend across chart and general-VQA benchmarks.
1. Introduction
- Fine-grained visual cues remain a VLM bottleneck, and perceptual slips can yield plausible answers mismatched with the visual evidence.Prior tools and visual-hint methods are also limited by rigid shapes, domain-specific design, and added inference-time computation.Source: §1 · pp. 1–2
- BiPS applies KL consistency between the original image and an evidence-preserving view, then KL separation from an evidence-ablated view to promote visual grounding.The first view retains question-relevant regions; the second removes critical pixels, discouraging text-only shortcuts and encouraging fine-grained visual reliance.Source: §1 · p. 2
- Rather than teaching models to emit visual cues at inference, BiPS uses question-conditioned cues as training signals that shape the model’s internal policy.The approach is described as requiring no extra annotations or customized parsers at inference.Source: §1 · p. 2
- Executable chart scripts in ECD provide object provenance for precise edits that create evidence-preserving and evidence-ablated training views.The pipeline yields 13K synthetic training examples.Source: §1 · p. 2
- +8.2% average improvement over Qwen2.5-VL-7B is reported across eight chart and general VQA benchmarks after adding 39K math-focused samples with GRPO.With 13K chart samples alone, the reported average improvement is 7.3%.Source: §1 · p. 2
2. Related Works
Prior multimodal reasoning work adds or predicts visual cues, while chart research studies structured quantitative graphics and reasoning over their visual data. The passages highlight limitations in fine-detail handling, generalization, and chart-to-code accuracy.
2.1. Multimodal Reasoning
- Inference-time boxes, crops, and masks, along with trained evidence-prediction methods, can struggle with irregular details, domain transfer, and extra inference steps.Examples include thin polylines and intersections in charts, lesion contours, and nonconvex polygons.Source: §2.1 · p. 3
- Latent visual reasoning remains task-specific, while noise and random-mask perturbations can overlook informative details within figures.The cited approaches include ChiP [10] and PAPO [44].Source: §2.1 · p. 3
2.2. Chart Understanding And Reasoning
- Chart understanding requires precise perception of numerical relations encoded by axes and marks, and recent benchmarks emphasize reasoning beyond element detection and text extraction.Chart reasoning approaches include symbolic programs or code, multimodal feedback, reflective learning, and reinforcement-based optimization.Source: §2.2 · p. 3
- Chart reasoning still faces limited chart-to-code accuracy, weak generalization across layouts, and difficulty with high-frequency curve changes and subtle oscillations.Figure 3 depicts a pipeline that edits chart source code to create paired evidence-preserving and evidence-ablated views.Source: §2.2 · pp. 3–4, Figure 3 · p. 4
3. Method
BiPS shapes VLM perception through paired question-conditioned views and bidirectional KL objectives, with a coarse-to-fine curriculum that first aligns predictions to preserved evidence and then separates them from ablated evidence.
3. Method
- BiPS uses a two-stage GRPO training curriculum to keep predictions consistent with evidence-preserving views and divergent from evidence-ablated views.The Consistency Stage retains question-relevant regions; the Separation Stage removes critical regions.Source: §3 · p. 4
3.1. Programmatic Data Construction Pipeline
- Code-domain editing of ECD charts creates question-specific preserved and ablated views with semantically precise control, yielding 13K training samples.Charts’ marks, axes, and legends are linked to executable rendering code, enabling edits beyond coarse pixel masks or rectangular crops.Source: §3.1 · p. 4
3.2. Bi-directional KL Constraints
- The paired-view objectives provide positive guidance for evidence localization and negative-space regularization for robust visual grounding.The consistency constraint supplies the positive signal; separation supplies the complementary regularizer.Source: §3.2 · p. 5
3.2.1. Focusing via Consistency
- The consistency constraint minimizes KL divergence between full-image and evidence-preserving-view answer distributions, guiding focus toward relevant evidence.The evidence-preserving branch serves as a fixed target, and supervision is restricted to validated correct samples.Source: §3.2.1 · p. 5
3.2.2. Robustness via Separation
- Consistency alone can permit shortcut learning from surrounding text or language priors without attending to fine-grained visual details.The separation constraint instead maximizes divergence between policies for the full image and evidence-ablated view, penalizing similarity until it exceeds a clipping threshold.Source: §3.2.2 · p. 5
3.3. A Coarse-to-Fine Training Curriculum
- GRPO normalizes rewards within rollout groups; BiPS first adds consistency for coarse focus, then separation from a Stage 1 checkpoint to strengthen visual grounding.The two-stage curriculum decouples objectives whose simultaneous optimization can produce conflicting gradients.Source: §3.3 · p. 5
4. Experiments
Experiments evaluate BiPS models across chart and general-reasoning benchmarks, then test the contributions of the constraints, training curriculum, counterpart generation, and coefficient choices.
4.2. Main Results
- BiPS-Chart-7B improves the average score by +7.3 points (44.3 to 51.6), and BiPS-General-7B by +8.2 points (to 52.5) over Qwen2.5-VL-7B.The evaluation spans chart-understanding and general perception and reasoning benchmarks.Source: Table 1 · p. 6, §4.2 · p. 6
- With only 13K chart samples, BiPS-Chart-7B raises ChartXiv from 42.5 to 49.4 (+6.9) and Evochart from 52.0 to 68.2 (+16.2), while improving unseen MathVista by +5.3 and MMStar by +2.8.The authors report that these improvements surpass chart-specialized models trained on hundreds of thousands to millions of chart examples.Source: §4.2 · pp. 6–7
4.3. Ablation Study
- Adding either KL constraint improves performance, while combining them achieves 39.9% on ECD-Bench and 49.4% on CharXiv.Consistency yields +2.9% on CharXiv; separation yields +2.7% on ECD-Bench and +3.4% on CharXiv.Source: Table 2 · p. 7, §4.3 · p. 7
4.2. Main Results
- Using 39K math-focused samples, BiPS-General-7B reaches 50.6 on ChartXiv, 68.7 on Evochart, 75.0 on MathVista, and 65.7 on MMStar, surpassing both the baseline and BiPS-Chart-7B.The authors attribute further gains in abstract and numerical reasoning to adding math-specific data.Source: §4.2 · p. 7
- On CharXiv, BiPS-General-7B exceeds standard GRPO trained on the same combined data by +5.2.The standard-GRPO baseline treats the data uniformly without the two-stage perceptual-shaping curriculum.Source: §4.2 · p. 7
4.3. Ablation Study
- The coarse-to-fine curriculum outperforms joint training by +3.0% on CharXiv and +3.2% on ECD-Bench, and reversed ordering scores 46.8% versus 49.4% on CharXiv.The authors suggest that simultaneously optimizing the objectives can create competing gradients and higher update variance.Source: Table 3 · p. 7, §4.3 · pp. 7–8
- Programmatic code editing significantly outperforms random masking across benchmarks by producing semantically controlled evidence-preserving and evidence-ablated views.The random-masking baseline removes 60% of image patches and can leave either an easier image or a mixture of missing evidence and residual distractions.Source: Table 4 · p. 7, §4.3 · p. 8
- On CharXiv, moderate coefficients α = 0.01 and β = 0.02 yield peak accuracy, while large coefficients such as 0.08 degrade performance.In the case study, BiPS gives visually grounded answers where Qwen2.5-VL-7B relies on textual or statistical cues and answers incorrectly.Source: Figure 4 · p. 7, §4.3 · pp. 8–9, Figure 5 · p. 8
5. Conclusion
BiPS uses paired question-conditioned views to shape visual perception during training. The paper reports gains in generalization, fine-grained perception, and inference efficiency.
5. Conclusion
- BiPS applies complementary KL constraints to pull predictions toward evidence-preserving views and away from evidence-ablated views.The constraints encourage coarse coverage of supporting regions and fine-grained reliance on visual evidence without generating visual cues at inference.Source: §5 · p. 9
- Training on 13K chart samples yields gains on chart-specific benchmarks and general VQA.The paper describes these gains as significant and wide-ranging.Source: §5 · p. 9
- BiPS improves fine-grained perception on chart benchmarks and adds no test-time overhead.The paper points to complex layouts and thin polylines as demands for precise visual interpretation.Source: §5 · p. 9
6. Implementation Details
The implementation constructs chart training data through staged reformulation, difficulty filtering, and view synthesis, then applies reinforcement-learning training with specified optimizer and reward settings.
6.1. Data Statistics
- From 50,000 raw chart-code pairs, reformulation and validation retained approximately 30K samples, and difficulty filtering left roughly 20K challenging samples.About 10K samples answered correctly in all eight base-model rollouts and were discarded.Source: §6.1 · p. 12, Table 5 · p. 12
- View synthesis produced evidence-ablated views for 13K samples and evidence-preserving views for approximately 7K, while the final BiPS training dataset comprised 13K samples.The preserved-view count is a subset of the 13K samples with ablated views.Source: §6.1 · p. 12
6.2. Training Details
- RL training uses AdamW at a learning rate of 1 × 10−6, keeps the vision tower unfrozen, and assigns reward weights of 0.1 for formatting and 0.9 for prediction.Table 6 is identified as the source of the RL hyperparameter configuration.Source: §6.2 · p. 12, Table 6 · p. 12
7. Additional Results
The additional results examine visual counting, paired-view construction, and BiPS on a stronger thinking-mode model. They report cross-domain transfer alongside qualifications about OCR and the limits of chart-only supervision.
7.1. Case Study
- On visual counting, BiPS tracks and subtracts objects correctly where the baseline fails from incomplete object reasoning.The authors cite this case as evidence of stronger cross-domain generalization.Source: §7.1 · p. 12, Figure 6 · p. 13
- The paired-view example shows that cropping can miss fine-grained evidence and sparse metadata, while preserving all question-related visual elements.The evidence-ablated view removes only the key visual element, such as a target line.Source: §7.1 · p. 12, Figure 7 · p. 13
7.2. Qwen3-VL-Thinking Results and Discussion
- Fine-tuning Qwen3-VL-8B-Thinking on 13K chart samples yields significant chart improvements and generalization to non-chart domains.The authors describe these perceptual improvements as complementary to the model’s reasoning capabilities.Source: §7.2 · p. 12
- Chart-sample RL fine-tuning negatively affects OCR, an effect partially alleviated by an OCR-priority system prompt used for MathVerse.The prompt is applied where some questions appear in the image and is shared across models for fair comparison.Source: §7.2 · p. 13
- On Qwen3-VL-8B-Thinking, BiPS brings substantial in-domain gains, largely maintains cross-domain performance, and gives moderate out-of-domain gains on most benchmarks.The discussion notes that transfer magnitude is naturally bounded when target domains differ substantially from chart training data.Source: §7.2 · p. 13
- The discussion proposes heterogeneous or multi-domain BiPS training and leaves systematic study of these settings to future work.Suggested view-construction routes include semantic masks for natural images and counterfactual views for procedural domains such as mazes.Source: §7.2 · p. 14
8. Prompts
The prompt materials describe chart-question reformulation and two chart-editing procedures: one preserves relevant evidence while removing distractors, and the other removes decisive evidence while preserving layout.
8. Prompts
- Question reformulation checks chart question-answer correctness and can rewrite valid items as multiple-choice questions with three or four options.The instructions call for plausible distractors and clear, unambiguous questions.Source: §8 · p. 14
- The evidence-preserving editing prompt removes irrelevant chart elements while retaining referenced content and the layout, axes, legends, and series positions.It directs the editor not to modify global-comparison or vague questions and to avoid editing when uncertain.Source: §8 · p. 14
- The evidence-ablating prompt blanks decisive chart evidence to make the question unanswerable while preserving layout and legend structure.Its instructions include blanking compared candidates, target geometry, annotations, and other cues that reveal an answer.Source: §8 · p. 15