PaperlayerView exact revision on arXiv

Source-linked AI summary

Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answering

Yanjie Zhang, Yafei Li, Rui Sheng, Zixin Chen, Yanna Lin, Huamin Qu, Lei Chen, Yushi Sun

arXiv:2603.28583v2cs.CVcs.AIcs.MM

TL;DR

Misleading charts challenge VLMs because deceptive structures can conflict with the underlying numerical data. ChartCynics separates structural diagnosis from OCR-based verification and achieves state-of-the-art 74.43% accuracy on the Qwen3-VL-8B backbone.

  • Problem

    Misleading ChartQA requires VLMs to detect visual manipulations and resolve cross-modal contradictions, yet state-of-the-art models often score below 50% accuracy.

  • Method

    ChartCynics combines diagnostic visual cropping, OCR-based numerical grounding, and jointly optimized summarization to resolve conflicts between visual cues and literal data.

  • Results

    ChartCynics achieves state-of-the-art accuracy across challenging chart benchmarks, including 74.43% on the Qwen3-VL-8B backbone.

  • Takeaways & Limitations

    The results support specialized agentic workflows as a generalized approach for robust interpretation of both misleading and standard charts.

  • Takeaways & Limitations

    Holistic end-to-end VLMs remain vulnerable to malicious chart encodings because they favor macro-level visual heuristics over fine-grained structural cues.

Abstract

from arXiv · show

Despite the success of Vision-Language Models (VLMs), misleading charts remain a significant challenge due to their deceptive visual structures and distorted data representations. We present ChartCynics, an agentic dual-path framework designed to unmask visual deception via a "skeptical" reasoning paradigm. Unlike holistic models, ChartCynics decouples perception from verification: a Diagnostic Vision Path captures structural anomalies (e.g., inverted axes) through strategic ROI cropping, while an OCR-Driven Data Path ensures numerical grounding. To resolve cross-modal conflicts, we introduce an Agentic Summarizer optimized via a two-stage protocol: Oracle-Informed SFT for reasoning distillation and Deception-Aware GRPO for adversarial alignment. This pipeline effectively penalizes visual traps and enforces logical consistency. Evaluations on two benchmarks show that ChartCynics achieves 74.43% and 64.55% accuracy, providing an absolute performance boost of ~29% over the Qwen3-VL-8B backbone, outperforming state-of-the-art proprietary models. Our results demonstrate that specialized agentic workflows can grant smaller open-source models superior robustness, establishing a new foundation for trustworthy chart interpretation.

1 Introduction

Misleading Chart Question Answering requires VLMs to maintain logical consistency against deceptive visual cues rather than merely extract explicit chart facts. ChartCynics addresses this challenge with a skeptical dual-path architecture and two-stage optimization that combine structural diagnosis, numerical grounding, and deception-aware reasoning.

  • Motivation: Misleading Chart Question Answering shifts chart understanding from extracting explicit visual facts to critical reasoning that resists deceptive cues.Misleading charts can manipulate perception through axis manipulation, cherry-picking, or disproportionate encoding.
  • Existing Limitations: End-to-end VLMs favor macro-level visual heuristics, while standalone OCR loses spatial semantics and causes catastrophic entity misalignment.VLMs may overlook exact tick values; OCR may confuse axis ticks with data points.
  • ChartCynics Framework: ChartCynics decouples structural perception from literal extraction through strategic diagnostic cropping and an OCR-Driven Data Path.The Diagnostic Vision Path detects anomalies such as inverted scales and supplies spatial semantics for data-driven reasoning.
  • Optimization: ChartCynics uses SFT to establish a 5-step verification framework and deception-aware GRPO to calibrate reasoning by penalizing misleading visual choices.The two-stage protocol equips the Summarizer with detective Chain-of-Thought and alignment against visual traps.
  • Results: 74.43% accuracy is achieved by ChartCynics versus 45.57% for the Qwen3-VL-8B baseline on deceptive benchmarks, with SFT providing a 22.95 percentage point boost.GRPO further optimizes reasoning trajectories for complex cross-modal contradictions.

2 Related Work

Related work progresses from factual chart extraction to VLM-based reasoning and specialized alignment, yet misleading encodings, weak element grounding, and visual–data contradictions remain unresolved. Recent benchmarks and optimization methods motivate skepticism-focused approaches for robust chart question answering.

  • Traditional Chart Comprehension: Early chart comprehension datasets primarily targeted factual extraction from natural visualizations, including FigureQA, DVQA, PlotQA, and Chart-HQA.
  • Misleading Charts: Misleading ChartQA, LEAF-QA, and CHARTOM show that advanced VLMs remain vulnerable to deceptive encodings and Theory-of-Mind visual traps.These deceptions include axis truncation and inverted axes, exploiting human cognitive biases.
  • VLM-Based ChartQA: VLM advances use chart-specific pre-training, math reasoning, and plot-to-table translation, but holistic processing can miss fine-grained anomalies such as non-uniform axis scales.
  • Open Challenges: VLMs still exhibit weak grounding in chart elements, while resolving contradictions between visual representations and raw data remains an open challenge.
  • Reasoning Alignment: Specialized reasoning alignment combines SFT for domain-specific skeptical CoT with GRPO, MCTS-guided sampling, and targeted reinforcement learning for difficult or misleading cases.These methods address misleading distractors and humancentric Theory-of-Mind questions.

3 Methodology · 3.1 Concept and Problem Definition

This section formalizes ChartQA and Misleading ChartQA by modeling deceptive charts as conflicts between perceived visual trends and actual numerical relationships. ChartCynics addresses this conflict through inconsistency-aware fusion of visual and data-path information guided by expert taxonomy.

  • 3.1 Concept and Problem Definition: ChartQA predicts the correct answer a* from candidate options O given a chart image I and natural-language question Q.Traditional VLM approaches model this as a direct mapping P(a| I, Q).
  • 3.1 Concept and Problem Definition: A misleading chart applies a deceptive manipulation function to raw data so its perceived visual trend contradicts the actual numerical relationship.The manipulation is represented as f_deceptive(D) → I, contrasting V_trend with D_rel.
  • 3.1 Concept and Problem Definition: Misleading ChartQA explicitly includes at least one trap answer derived from the deceptive manipulation function.The trap answer belongs to the candidate option set O.
  • 3.1 Concept and Problem Definition: Misleading ChartQA requires complementary reasoning: systems must interpret the user’s perceptual context while simultaneously using the actual numerical relationship.This requirement arises because visual cues and numerical data can diverge in misleading charts.
  • 3.1 Concept and Problem Definition: ChartCynics introduces a Misleading Taxonomy T as expert prior knowledge to guide dual-path arbitration through Inconsistency-Aware Fusion.The objective avoids simply prioritizing one reasoning path over the other.
  • 3.1 Concept and Problem Definition: The Visual Path captures semantic context, whereas the Data Path ensures numerical precision during fusion.These roles are represented by P_v and P_d, respectively.

3.2 Solution Overview

ChartCynics uses a skeptical dual-path architecture that separates visual heuristics from OCR-grounded numerical facts before joint summarization. Its two-stage optimization distills investigative reasoning and aligns the model against deception.

  • Skeptical Dual-Path Philosophy: ChartCynics decouples visual intuition from final decisions to identify cross-modal deviations and calibrate perceptions through data verification.The framework also uses RL-based optimization to learn balancing weights between Pᵥ and P_d across deception types.
  • Dual-Path Architecture: The pipeline separates visual heuristics from OCR-based numerical facts and feeds both into a Summarizer Joint Inference module.This architecture operationalizes the framework’s skeptical reasoning philosophy.
  • Two-Stage Optimization: Oracle-Informed SFT distills reasoning, while Deception-Aware GRPO provides adversarial alignment in a two-stage optimization strategy.The strategy is designed to internalize the framework’s investigative logic.

3.3 Vision Path: Diagnostic-Augmented Investigation

The Vision Path separates visual diagnosis from reasoning through two agents, using targeted ROI inspection to expose deceptive structural cues before final inference. A blind diagnostic report and mandatory directive anchoring reduce confirmation bias and shift reasoning toward evidence-grounded conclusions.

  • Architecture: ChartCynics decouples perception from reasoning with Diagnostic and Reasoning Agents that produce a visual report identifying chart traps before fusion.The path operationalizes the visual component of the agentic objective through a two-agent architecture.
  • ROI Extraction: Automated graphic-element detection localizes title, legend, x-axis, and y-axis regions for fine-grained inspection of deceptive chart cues.The ROI set is defined as C_roi = {title, legend, x-axis, y-axis}.
  • ROI Extraction: Padding expands legend and axis crops to include markers and full tick labels, enabling literal scale reading instead of visual estimation.Spatial spread determines the padding applied to each detected chart component.
  • Diagnostic Investigation: Functional isolation and a blind test withhold the question and candidate answer from the Diagnostic Agent, preventing confirmation bias and self-prompt pollution.The agent receives only the chart image and high-resolution ROI crops, then acts as a neutral auditor.
  • Directive-Anchored Reasoning: The Diagnostic Agent reports a factual Diagnosis and prescriptive Action Directive, which the Reasoning Agent must reiterate first to anchor its Chain-of-Thought.This converts deceptive visual trends into detectable anomalies by fusing visual evidence, relevant data, and trap definitions.

3.4 Data Path: OCR-Driven Serialization

The OCR-Driven Data Path reconstructs chart relationships from literal textual and numerical evidence, providing an orthogonal check against deceptive visual encodings. It serializes OCR outputs into structured Markdown and dynamically calibrates their trust during reasoning.

  • Data Path purpose: The Data Path bypasses manipulated areas, lengths, and angles by reconstructing underlying numerical relationships as a literal backbone.It serves as an orthogonal verification layer against visual trend illusions.
  • OCR serialization: A multimodal OCR module extracts axis scales, data labels, and legends from chart images, then serializes them into unified structured Markdown.This first stage directly exposes graphical attributes without relying on visual rendering.
  • OCR serialization: The Markdown representation recovers numerical data from parsed literal characters without contamination from deceptive visual manipulation.Because it disregards spatial positioning, the representation provides objective numerical evidence.
  • Trust calibration: Calibration Directives instruct the Reasoning Agent to dynamically evaluate OCR outputs because extraction may capture arbitrary text or hallucinate absent labels.This evaluation occurs during Chain-of-Thought reasoning rather than through a hard-coded preprocessing filter.
  • Trust calibration: The framework assigns high trust to explicit labels and low trust to axis-only values or elements exceeding the canvas boundary.Explicitly plotted data labels are treated as immutable ground truth, while other cases trigger skepticism.

3.5 Agentic Fusion with Detective Chain-of-Thought

The Agentic Fusion Summarizer integrates diagnostic visual evidence with calibrated OCR data to neutralize deceptive chart mechanisms. Its five-step Detective Chain-of-Thought prioritizes grounded reconstruction, deception classification, evidence sufficiency, and explicit trap rejection.

  • Agentic Fusion: The Summarizer combines the Diagnostic Report R_diag and calibrated OCR data M_ocr to maximize the posterior while identifying and neutralizing the deceptive function f_deceptive.This module integrates outputs from the Vision and Data Paths as the final reasoning component.
  • Golden Rules: Golden Rules resolve conflicts by recontextualizing visual heuristics with structural anomalies and dynamically conditioning M_ocr’s epistemic weight on trust labels.High-Trust OCR labels serve as immutable ground truth for D_rel, while scale violations can affect interpretation when labels are absent.
  • Detective Chain-of-Thought: The D-CoT first audits perception and reconstructs the numerical relationship by mapping OCR categories and values to the entities in the question.These steps recalibrate visual priors and distinguish the actual relationship from a partial visual subset.
  • Detective Chain-of-Thought: The agent then classifies the manipulation subtype, applies the corresponding correction, and checks whether combined evidence is sufficient for a reliable conclusion.When trust levels conflict or evidence is inadequate, it may output “Cannot be Inferred” to avoid hallucination.
  • Detective Chain-of-Thought: Finally, the agent proves why the visual Trap Answer is mathematically fallacious, selecting the answer through critical elimination rather than pattern matching.The framework integrates the Diagnostic Report and OCR Markdown through this five-step process for adversarially robust, evidence-based deduction.

3.6 Optimization: SFT and GRPO

ChartCynics uses a two-stage optimization pipeline that distills reasoning from a larger model and then aligns generation with adversarial skepticism. GRPO combines numerical, semantic, and structural rewards to improve factual consistency without a dedicated Critic network.

  • Two-stage optimization: The optimization pipeline first applies Oracle-Informed Distillation and then Deception-Aware Group Relative Policy Optimization (GRPO) to internalize heuristics and resolve multimodal conflicts.This two-stage sequence is explicitly designed to combine reasoning distillation with adversarial alignment.
  • Supervised Fine-Tuning: Supervised Fine-Tuning (SFT) uses data generated by Qwen3-VL-32B, with ground-truth CSV data and trap labels supporting factual accuracy during generation.The target model processes visual pixels, while the source model has access to raw CSV data, creating an information disparity that the prompting strategy addresses.
  • Deception-aware GRPO: GRPO uses a multi-dimensional continuous reward function R_total over sampled groups, unlike DPO’s binary preference formulation.The reward evaluates the quality of generated reasoning traces while aligning the reasoning process with adversarial skepticism.
  • Deception-aware GRPO: Numerical Grounding R_fact rewards Spearman rank correlation between OCR-extracted numeric tokens and Oracle CSV values, prioritizing relative trends over absolute pixel-to-value accuracy.This design aims to improve robustness against OCR noise by rewarding relationships among values rather than exact visual measurements.
  • Deception-aware GRPO: Semantic Contradiction R_contra combines keyword hit rates with semantic overlap against expert explanations, while Structural Enforcement R_fmt penalizes shortcut generations.The semantic reward targets logical consistency across the 5-step D-CoT trajectory.
  • Optimization outcome: Optimizing group-relative advantages over fine-grained rewards helps the model balance visual intuition with factual evidence without a dedicated Critic network.The result passage attributes this behavior to group-relative reward optimization.

4 Experiments

ChartCynics is evaluated across deceptive and mixed chart benchmarks using accuracy and deception-specific error metrics. Results show that its dual-path architecture and two-stage alignment substantially improve Qwen3-VL-8B performance while reducing visually induced errors without harming standard-chart accuracy.

  • Experimental Setup: ChartCynics is evaluated on Misleading ChartQA, CDCC, and the Mixed Standard and Misleading Benchmark using Accuracy, WM, and WO.Misleading ChartQA contains 2,619 training samples and 305 test samples, while CDCC contains 110 expert-validated deceptive visualizations.
  • Main Results: 35.45% to 64.55%: optimized Qwen3-VL-8B nearly doubles its baseline accuracy on CDCC.The improvement demonstrates transfer of the optimized model’s robustness to the secondary deceptive benchmark.
  • Mechanistic Analysis: 34.43% to 18.69%: training-free o4-mini’s WM rate drops nearly two-fold without a compensatory spike in WO.The result supports grounding reasoning in structurally grounded OCR evidence rather than blindly avoiding traps.
  • Mixed-Benchmark Evaluation: 68.03% versus 31.97%: ChartCynics doubles ChartMoE’s accuracy on visual deceptions and also reaches 94.26% versus 88.52% on standard charts.The comparison is conducted on the Mixed Standard and Misleading Benchmark.
  • Ablation Study: 74.43% accuracy follows GRPO alignment, improving on SFT alone at 68.52% while the full architecture minimizes both WM and WO.OCR Only and VLM + Crop ablations show that complementary visual-structural and OCR information is necessary; VLM + Crop reduces WO to 15.41%.

5 Conclusion

ChartCynics strengthens VLMs against misleading visualizations by separating intuitive perception from rigorous verification through an agentic dual-path design. Its Diagnostic Vision and OCR-Driven Data Paths support skeptical reasoning, achieving 74.43% accuracy on Qwen3-VL-8B.

  • Framework: ChartCynics decouples intuitive perception from rigorous verification to fortify VLMs against misleading visualizations.The framework uses an agentic dual-path design and a skeptical reasoning paradigm.
  • Framework: The Diagnostic Vision Path and OCR-Driven Data Path systematically unmask deceptive chart structures.Their synergy operationalizes the framework’s skeptical reasoning paradigm.
  • Results: 74.43% accuracy was achieved by ChartCynics on the Qwen3-VL-8B backbone.The passage reports this as a state-of-the-art result.
Loading 2603.28583v2…