PaperlayerView exact revision on arXiv

Source-linked AI summary

TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning

Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yichen Xu, Qin Jin, Ji Zhang, Fei Huang

arXiv:2404.16635v1cs.CV

TL;DR

Chart MLLMs face constraints from model size, numerical reasoning, and lengthy high-resolution visual sequences. TinyChart combines visual token merging with Program-of-Thoughts learning in a 3B model. It achieves state-of-the-art results across chart benchmarks, including 83.60 accuracy on ChartQA.

  • Problem

    Chart MLLMs face deployment constraints from large parameter counts, numerical-answer errors, and lengthy feature sequences for high-resolution images.

    Source: §1 · pp. 1–2

  • Method

    TinyChart uses visual token merging to shorten vision-feature sequences and trains the model to generate Python programs for numerical questions.

    Source: §1 · p. 2

  • Results

    TinyChart achieves state-of-the-art performance across chart benchmarks, surpasses existing 13B chart MLLMs, and reaches 83.60 accuracy on ChartQA, surpassing GPT-4V and other listed models.

    Source: §4.3 · p. 6, §5 · p. 11

  • Takeaways & Limitations

    Visual token merging enables efficient encoding of high-resolution chart images with constrained computing resources.

    Source: §1 · p. 2

    In chart-to-text cases, TinyChart sometimes produces content that does not match the chart.

    Source: §4.6 · p. 11

Abstract

from arXiv · show

Charts are important for presenting and explaining complex data relationships. Recently, multimodal large language models (MLLMs) have shown remarkable capabilities in various chart understanding tasks. However, the sheer size of these models in terms of parameters and computational requirements limits their use in resource-constrained environments. In this paper, we present TinyChart, an efficient MLLM for chart understanding with only 3B parameters. TinyChart overcomes two key challenges in efficient chart understanding: (1) reduce the burden of learning numerical computations through a Program-of-Thoughts (PoT) learning strategy, which trains the model to generate Python programs for numerical calculations, and (2) reduce lengthy vision feature sequences produced by the vision transformer for high-resolution images through a Vision Token Merging module, which gradually merges most similar vision tokens. Extensive experiments demonstrate that our 3B TinyChart achieves SOTA performance on a variety of chart understanding benchmarks including ChartQA, Chart-to-Text, Chart-to-Table, OpenCQA, and ChartX. It outperforms several chart understanding MLLM with up to 13B parameters such as ChartLlama and ChartAst, and close-sourced general-purpose MLLM GPT-4V on ChartQA. It also demonstrates its superior efficiency with higher throughput during inference due to a smaller model scale and more efficient vision encoding. Our code and model are available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/TinyChart.

1 INTRODUCTION

TinyChart targets the size, numerical-reasoning, and high-resolution encoding challenges of chart understanding with a 3B model, PoT learning, and visual token merging. The paper reports state-of-the-art performance across chart benchmarks and faster inference than several larger models.

1 INTRODUCTION

  • Existing chart models face large parameter counts, numerical-answer errors, and lengthy visual feature sequences for high-resolution images.The introduction gives ChartLlama's 13B parameter scale as an example of deployment difficulty.Source: §1 · pp. 1–2
  • Visual Token Merging aggregates similar tokens within vision-transformer layers, shortening feature sequences while retaining high-resolution chart input with controlled computation.The method targets charts' large areas of color and whitespace.Source: §1 · p. 2
  • Program-of-Thoughts learning trains TinyChart to generate step-by-step Python programs, which an interpreter executes to answer numerical questions.The authors report that PoT particularly improves numerical question answering and construct ChartQA-PoT to support this training.Source: §1 · p. 2
  • TinyChart achieves state-of-the-art performance on chart benchmarks with 3B parameters, outperforming several 13B MLLMs while delivering faster inference.The authors also report that it outperforms GPT-4V on ChartQA.Source: §1 · p. 2

2 RELATED WORK

Related work describes a shift from chart-processing pipelines to end-to-end MLLMs, while identifying constraints in numerical reasoning, model size, and high-resolution visual encoding.

2.1 Chart Understanding

  • Chart tasks range from data extraction to question answering, summarization, and re-rendering, requiring OCR-text recognition and numerical reasoning.The passage notes that earlier pipeline approaches were limited by error accumulation.Source: §2.1 · p. 2
  • Earlier chart systems converted charts into tables or text before language-model processing, while end-to-end MLLMs improved performance but remained difficult to train or deploy in resource-constrained settings.Template-based executable commands can be narrow in coverage, motivating the paper's combination of template- and GPT-generated programs.Source: §2.1 · p. 2

2.2 Multimodal Large Language Model

  • High-resolution MLLM approaches can improve fine-grained understanding, but lengthy visual sequences reduce efficiency; visual token merging reduces sequence length and computational requirements.The passage lists increased resolution, image segmentation, frequency-domain signals, and additional high-resolution encoders as prior approaches.Source: §2.2 · p. 2

3.1 Model Architecture

TinyChart follows an MLLM architecture and inserts visual token merging into the vision transformer to encode high-resolution chart images efficiently.

3.1 Model Architecture

  • TinyChart combines a vision-transformer encoder, a vision-language connector, and a large language model, with visual token merging inside each encoder layer.The vision transformer resizes images and processes them as patches represented by vision tokens.Source: §3.1 · p. 3, §3.1.1 · p. 3

3.1.1 Vision Transformer Encoder.

  • Standard vision-transformer sequences have length (N//P)^2 for image resolution N×N and patch size P×P, and do not shrink across layers.Long sequences become inefficient for the language model when image resolution is large.Source: §3.1.1 · p. 3
  • Visual Token Merging reduces each layer's feature sequence by merging the r most similar token pairs, selected by key-vector cosine similarity and bipartite matching.The selected pair features are combined by average pooling, and similar tokens need not be spatially adjacent.Source: §3.1.1 · pp. 3–4
  • Proportional attention adds log s inside softmax so a merged token representing s patches retains their combined representation in attention.The passage defines Q, K, and V as self-attention projections from hidden states.Source: §3.1.1 · p. 4

3.1.2 Vision-Language Connector.

  • The vision-language connector projects visual features into the language-model embedding space using a one-hidden-layer MLP with GeLU activation..Source: §3.1.2 · p. 4

3.1.3 Large Language Model.

  • The language model is a causal-attention transformer decoder that generates responses from visual features and instructions, with loss computed over response tokens.The stated objective is language modeling.Source: §3.1.3 · p. 4

3.2 Program-of-Thoughts Learning

TinyChart's Program-of-Thoughts training uses executable Python code as answers and a ChartQA-derived dataset built through template-based and GPT-based construction.

3.2 Program-of-Thoughts Learning

  • Program-of-Thoughts learning trains models to produce executable Python code for a question, with an interpreter returning the final answer.Code can include comments and multistep reasoning rather than only the calculated value.Source: §3.2 · p. 4
  • ChartQA-PoT contains 140,584 question-and-PoT-answer pairs, with answers written as commented, multiline Python code.The dataset is built from the ChartQA training split using template-based and GPT-based construction.Source: §3.2 · p. 5

3.2.1 Template-based PoT.

  • Template-based construction fills human-written question and code templates with chart data and yields 119,281 pairs across 17,498 images.Its templates cover common operations such as sum, average, minimum, and maximum, using 40 question templates.Source: §3.2.1 · p. 5
  • GPT-based construction uses gpt-3.5-turbo with chart tables and in-context examples, retaining generated programs only when execution matches the annotated answer.This approach adds diversity beyond fixed templates and produces 21,303 pairs on 15,521 chart images.Source: §3.2.2 · p. 5

3.3 Multitask Learning

TinyChart is trained jointly on 1.36M samples spanning several chart-understanding tasks, with task-specific instructions to distinguish them. For ablations, training uses only benchmark datasets because of limited computational resources.

3.3 Multitask Learning

  • The supervised fine-tuning dataset contains 1.36M samples covering chart question answering, chart-to-text, chart-to-table, and chart instruction following.Table 1 presents the training-dataset collection; its benchmark datasets cover basic chart QA, summary, and chart-to-table generation.Source: §3.3 · p. 5, Table 1 · p. 5
  • Data from different tasks are mixed for joint training, and task-specified instructions help the model differentiate between them.Source: §3.3 · p. 5
  • The training objective is language modeling on response tokens.Source: §3.3 · p. 5
  • Ablation studies use only benchmark datasets for training because of limited computational resources.Source: §3.3 · p. 5, Table 1 · p. 5

4 EXPERIMENT

TinyChart’s experiments report leading benchmark performance alongside faster inference, with ablations showing how Program-of-Thoughts learning and visual token merging contribute to chart understanding. Case studies also show specific limits in reading values without OCR text and handling hallucinations or unseen chart types.

4.1 Implementation Details

  • TinyChart is initialized from TinyLlava and trained for 3 epochs; training took 3 days on 32 Tesla V100 GPUs.The model extends vision input resolution to 512×512 and 768×768.Source: §4.1 · p. 5

4.2 Evaluation Benchmarks

  • Evaluation spans ChartQA, chart summarization and table extraction, OpenCQA, and ChartX cognition tasks, using accuracy, BLEU4, RMSF1, GPT-Accuracy, and GPTscore metrics.ChartQA is evaluated by default with Combine, which uses a keyword detector to select direct answers or Python-based PoT answers.Source: §4.2 · pp. 5–6

4.3 Main Results

  • At 768×768, TinyChart reaches 83.60 on ChartQA and is reported to lead four chart benchmarks while exceeding several closed-source models on ChartQA.The comparison includes GPT-4V; the paper also reports larger inference throughput than previous models.Source: Table 2 · p. 6, §4.3 · pp. 6–7
  • ChartQA-human reaches 73.34%, 7.44% above ChartAst; PoT improves calculative accuracy (78.98 vs. 56.64), while ChartX QA beats GPT-4V but its other tasks trail.On ChartX, inference uses TinyChart@768 without additional fine-tuning; it outperforms open-source Chart MLLMs on summary, description, and redrawing.Source: §4.3 · pp. 6–7, Table 4 · p. 7, Table 5 · p. 7

4.4 Ablation Studies

  • GPT-based PoT training raises combined ChartQA performance to 79.48 vs. 73.00 with template-based PoT training.With template-only training, PoT answers score 55.44 vs. 71.12 for direct answers, which the authors attribute possibly to limited template coverage.Source: §4.4 · p. 7
  • At 512×512, r=20 yields 3.65 it/s vs. 3.73 it/s at 384×384; at 768×768, r=84 reduces sequence length from 2,916 to 732.At 768×768, performance is 81.04 vs. 80.76 on ChartQA and 88.90 vs. 87.81 on Chart-to-Table; without merging, the sequence could not be trained on a 32GB V100.Source: §4.4 · p. 8, Table 6 · p. 8

4.5 Visualization

  • The largest merged token groups typically correspond to blank or colored chart areas, which the module compresses to single tokens.Figure 5 outlines the ten groups with the most merged tokens in different colors.Source: §4.5 · p. 8, Figure 5 · p. 8

4.6 Case study

  • Case studies show QA gains from high-resolution input and PoT, but value estimation without OCR words, chart-to-text hallucinations, and redraws of unseen chart types remain difficult.The authors note difficulty estimating values from coordinate axes without OCR words and redraw difficulty for unseen types such as 3D bar charts.Source: §4.6 · pp. 8, 10, 11

5 CONCLUSION

TinyChart is a 3B-parameter chart-understanding MLLM that combines visual token merging with Program-of-Thoughts learning. It reports state-of-the-art performance across chart benchmarks, surpasses larger chart MLLMs, and outperforms GPT-4V on ChartQA.

5 CONCLUSION

  • TinyChart reports state-of-the-art performance on multiple chart benchmarks, surpasses existing 13B chart MLLMs, and outperforms GPT-4V on ChartQA.The paper also reports that ablation studies confirm the effectiveness of its methods.Source: §5 · p. 11
  • The 3B-parameter model uses visual token merging to efficiently encode high-resolution images and Program-of-Thoughts learning to generate Python programs for answering questions.Source: §5 · p. 11
  • TinyChart is presented as an efficient chart-understanding MLLM, with code and model released by the authors.Source: §5 · p. 11

A.1 Dataset Statistic

ChartQA-PoT is built from ChartQA training-split images and questions and contains template-based and GPT-based subsets. The paper compares their answer lengths and question-type distributions.

A.1 Dataset Statistic

  • ChartQA-PoT has template-based and GPT-based subsets built from images and questions in the ChartQA training split.Source: §A.1 · p. 12
  • GPT-based PoT answers are longer than template-based answers because they cover more diverse scenarios.Source: §A.1 · p. 12
  • After stop-word removal, GPT-PoT covers more diverse “what” questions, while template-based questions are more evenly distributed across question types.Source: §A.1 · pp. 12–13, Figure 11 · p. 13

A.2 Instructions for GPT-based PoT

GPT-based PoT answers are generated by prompting GPT-3.5-turbo for Python code. The instructions constrain the code form and supply chart metadata when it may help answer a question.

A.2 Instructions for GPT-based PoT

  • The instructions prompt GPT-3.5-turbo to produce Python code for GPT-based PoT answers.Source: §A.2 · p. 13
  • The generated code is restricted to assignment statements, avoiding loops and judgment statements to simplify programs and reduce syntax errors.Source: §A.2 · p. 13
  • Prompts include chart title, type, and colors because some questions rely on that information.Source: §A.2 · p. 13
Loading 2404.16635v1…