PaperlayerView exact revision on arXiv

Source-linked AI summary

Advancing Chart Question Answering with Robust Chart Component Recognition

Hanwen Zheng, Sijia Wang, Chris Thomas, Lifu Huang

arXiv:2407.21038v1cs.CLcs.AIcs.IR

TL;DR

ChartQA requires fine-grained interpretation of textual and visual chart elements, while existing methods can struggle with complex graphics and question-guided attention. The paper combines unified chart-component recognition with question-guided deformable co-attention; it reports gains over baseline models on both recognition and question answering.

  • Problem

    ChartQA requires fine-grained interpretation of chart text and visuals, while existing approaches can struggle with complex graphics and question-relevant attention.

    Source: §1 · p. 1

  • Method

    CHARTFORMER recognizes chart components, and QDCHART uses Question-guided Deformable Co-Attention to fuse chart features with question information.

    Source: §1 · p. 2

  • Results

    3.2% in mAP for CHARTFORMER and 15.4% in accuracy for QDCHART are the reported improvements over baseline models on chart component recognition and ChartQA, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    QDCHART’s question-guided fusion enables the model to focus on chart components related to the question.

    Source: §1 · p. 2

    Minimal pre-training on ExcelChart400K may not capture the same level of detail and complexity, resulting in relatively lower performance.

    Source: §5.2 · p. 7

Abstract

from arXiv · show

Chart comprehension presents significant challenges for machine learning models due to the diverse and intricate shapes of charts. Existing multimodal methods often overlook these visual features or fail to integrate them effectively for chart question answering (ChartQA). To address this, we introduce Chartformer, a unified framework that enhances chart component recognition by accurately identifying and classifying components such as bars, lines, pies, titles, legends, and axes. Additionally, we propose a novel Question-guided Deformable Co-Attention (QDCAt) mechanism, which fuses chart features encoded by Chartformer with the given question, leveraging the question's guidance to ground the correct answer. Extensive experiments demonstrate that the proposed approaches significantly outperform baseline models in chart component recognition and ChartQA tasks, achieving improvements of 3.2% in mAP and 15.4% in accuracy, respectively. These results underscore the robustness of our solution for detailed visual data interpretation across various applications.

1. Introduction

ChartQA requires fine-grained interpretation of both textual and visual chart elements, while existing approaches can struggle with complex graphics and question-relevant attention. The paper introduces CHARTFORMER for unified component recognition and QDCHART for question-guided chart answering, reporting gains over baseline models on both tasks.

1. Introduction

  • ChartQA demands fine-grained interpretation of textual and visual elements, including linking chart labels and visual marks to answer natural-language questions.The example describes locating category and legend text, then matching a bar to its value.Source: §1 · p. 1
  • Existing recognition pipelines can struggle with stacked boxes, overlapping labels, and crossed lines, while OCR or extracted tables may introduce noise and confusion.The introduction also identifies late fusion of question embeddings as limiting question-guided attention to relevant chart regions.Source: §1 · p. 1
  • QDCHART uses Question-guided Deformable Co-Attention to fuse CHARTFORMER chart features with question information and focus on components related to the question.A Question-guided Offset Network and deformable co-attention module produce features that a text decoder uses to generate the answer.Source: §1 · p. 2
  • CHARTFORMER is an end-to-end unified model for identifying diverse chart components, including objects with distinct visual structures and chart semantics.The paper also introduces an automatically annotated instance-segmentation dataset spanning several prominent chart types.Source: §1 · p. 2
  • 3.2% in mAP for CHARTFORMER and 15.4% in accuracy for QDCHART over baseline models are the reported gains on chart component recognition and ChartQA, respectively.The introduction also reports that CHARTFORMER outperforms DAT by 11.4% and Mask2former by 3.2% in mAP, while QDCHART outperforms Pix2Struct by 1.2% accuracy.Source: §1 · p. 2

2. Related Work

Prior chart-comprehension work includes component detectors and varied multimodal question-answering systems. The paper positions CHARTFORMER as end-to-end instance segmentation and QDCHART as a unified approach that directs visual reasoning toward question-relevant components.

2. Related Work

  • Chart component recognition has used bounding-box detection, keypoint detection and grouping, or line-graph detection; CHARTFORMER instead uses end-to-end instance segmentation.Examples include ChartOCR’s keypoint association, ChartDETR’s unified keypoint-group detection, and Lineformer’s instance segmentation for line charts.Source: §2 · p. 2
  • Object detection and segmentation methods span CNNs and transformers, with approaches such as ViT, Swin Transformer, DCN, Mask2Former, and DAT using differing attention mechanisms.The paper draws on instance-segmentation frameworks for small and varied chart objects.Source: §2 · p. 2
  • ChartQA systems include visual encoder-decoder models, chart-focused pretraining, and component-aware approaches; QDCHART combines unified component detection with question-relevant visual attention.The cited examples include Donut, Pix2Struct, MatCha, and UniChart.Source: §2 · pp. 2–3

3. Method

The method combines CHARTFORMER’s unified chart-component recognition with QDCHART, which fuses chart and question information to generate answers. CHARTFORMER uses attention and mask decoding for component recognition, while QDCHART uses question-guided offsets and deformable co-attention to process chart features.

3.1. CHARTFORMER

  • CHARTFORMER identifies and classifies visual and textual chart components, including lines, bars, pie slices, axes, legends, and titles.It is presented as an end-to-end network for diverse chart types.Source: §3.1 · p. 3
  • Neighborhood and deformable attention give CHARTFORMER mechanisms for learning chart-image features, with deformable attention focusing on visual connections.The vision encoder follows a CNN feature-map stage; sampled features provide the key and value vectors for attention.Source: §3.1 · pp. 3–4
  • The pixel decoder upsamples features into a per-pixel embedding, which the mask transformer decoder combines with object queries to produce class and binary mask predictions.Bounding boxes are obtained from the smallest boxes enclosing predicted masks; mask and classification predictions use focal-plus-dice and classification losses, respectively.Source: §3.1 · p. 4

3.2. QDCHART

  • QDCHART combines a CHARTFORMER chart encoder, whose weights are frozen during ChartQA fine-tuning, with a Swin Transformer vision encoder and a text decoder.The vision encoder supplies complementary visual information, and the text decoder generates answers from processed features and task-specific prompts.Source: §3.2 · pp. 4–5
  • The question-guided offset network conditions sampling offsets on the question embedding so sampled points can align with locations semantically relevant to that question.The network uses combined inputs, rather than locations of overall significance, to guide sampling.Source: §3.2 · pp. 4–5
  • QDCAt combines co-attention across modalities with deformable attention over semantically relevant features, integrating chart features from the complementary encoders.Its output is processed with residual connections, normalization, and a feed-forward network before projection to the final image feature.Source: §3.2 · p. 5

4. Experimental Setup

The experiments evaluate CHARTFORMER on chart component recognition and QDCHART on ChartQA, using chart-specific datasets, established baselines, and task-appropriate metrics.

4. Experimental Setup

  • The evaluation covers chart component recognition and ChartQA, with Relaxed Accuracy allowing 5% numeric-answer error and requiring case-insensitive exact matches for non-numeric answers.ChartQA includes human-written and machine-generated questions across bar, line, and pie charts.Source: §4 · p. 5, §4.2 · p. 6

4.2. Chart Question Answering

  • CHARTFORMER is compared with four chart-recognition baselines, while QDCHART is benchmarked against twelve established ChartQA baselines.Component recognition uses mAP, mAP50, and mAP75; the supplied Table 2 caption identifies a comparison of chart-component recognition baselines.Source: §4.1 · pp. 5–6, §4.2 · p. 6, Table 2 · p. 7

4.1. Chart Component Recognition

  • ExcelChart400K annotations are converted from keypoints into instance-segmentation masks for seven chart-component categories.The categories are bar, line, pie, legend, value axis title, chart title, and category axis title.Source: §4.1 · p. 6, Table 1 · p. 6

5. Results

CHARTFORMER performs strongly across chart-component recognition, while QDCHART improves on many ChartQA baselines but trails extensively pretrained models. Ablations and visualizations probe the fusion components and question-guided sampling.

5.1. Chart Component Recognition

  • CHARTFORMER handles narrow lines and complex stacked-bar or overlapping-line charts more accurately than competing models, including Mask2Former on bar segments.The reported bar-segment accuracy increase over Mask2Former is 10.0%.Source: §5.1 · p. 7

5.2. Chart Question Answering

  • With minimal ExcelChart400K pretraining, QDCHART trails extensively pretrained models; its sampled points cluster around question-relevant chart elements and answer regions.The qualitative examples span bar, line, and pie charts, with points sparse or uniformly spaced in blank areas.Source: §5.2 · p. 7, Figure 5 · p. 8

5.1. Chart Component Recognition; 5.2. Chart Question Answering

  • CHARTFORMER leads across chart-recognition mAP metrics, while QDCHART surpasses non-pretraining baselines and some pretrained and LLM baselines but trails MatCha and UniChart.The supplied tables report chart-recognition category mAP results and ChartQA baseline comparisons.Source: §5.1 · p. 7, §5.2 · p. 7, Table 3 · p. 8

5.3. Ablation Studies

  • CNN fusion (QDCHARTCNN) decreases performance by 5.52%; simple concatenation yields 49.92% accuracy, and removing QON decreases accuracy by 1.64%.The QON effect is especially influential for human-written questions; the paper also reports that early question fusion enhances answers to more complex questions.Source: §5.3 · pp. 7–8, Table 5 · p. 8

6. Conclusion

The conclusion presents CHARTFORMER as a unified end-to-end approach to chart-component recognition and QDCHART as a question-aware fusion approach for ChartQA.

6. Conclusion

  • CHARTFORMER handles multiple chart-comprehension tasks across chart types, while QDCHART combines its question-aware chart features with general-purpose multimodal encoder features.The conclusion describes QDCAt as the fusion block and frames the approach as addressing ChartQA challenges.Source: §6 · p. 8

A. Segmentation Annotation

The appendix describes how chart keypoints are converted into polygon segmentation annotations for pies, lines, and other components.

A. Segmentation Annotation

  • Pie-slice annotations interpolate intermediate points from the angles and radii of the slice-edge vertices relative to the center.The procedure uses roughly five intermediate points per radian to approximate the circular arc.Source: A. Segmentation Annotation · p. 11
  • Line annotations use a width equal to 1% of image height because centerline keypoints do not encode thickness.This width is generally accurate but need not match the ground-truth line thickness; acute bends are adjusted to avoid elongated mask spikes.Source: A. Segmentation Annotation · p. 11
  • Other chart components receive rectangular annotations whose four vertices are computed from the supplied parameters.Source: A. Segmentation Annotation · p. 11

B. Supplementary materials

The supplementary materials include a ChartQA dataset-distribution table and a table comparing baseline results. Figures cover the question-guided offset flow and deformed points, alongside performance case studies for line, pie, and bar charts.

B. Supplementary materials

Related papers

Loading 2407.21038v1…