PaperlayerView exact revision on arXiv

Source-linked AI summary

Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots

Chengyue Wu, Yixiao Ge, Qiushan Guo, Jiahao Wang, Zhixuan Liang, Zeyu Lu, Ying Shan, Ping Luo

arXiv:2405.07990v1cs.CLcs.CV

TL;DR

Multimodal code generation from scientific plots has not been thoroughly evaluated, despite the need to assess models’ interpretation of visual elements and textual context. Plot2Code benchmarks this task with plot-code pairs, varied input settings, and code and image evaluation metrics. Its evaluations reveal substantial challenges, including difficulty reproducing text and fine-grained plot details.

  • Problem

    Multimodal code generation from plots lacks thorough evaluation, and existing benchmarks have limited coverage of multimodal inputs and image fidelity.

    Source: §1 · p. 2

  • Method

    Plot2Code pairs 132 plots across six plot types with code and GPT-4-generated descriptions, using varied input settings and metrics for code execution and rendered-image fidelity.

    Source: §1 · p. 3, §3.2 · p. 5, §3.4 · p. 6

  • Results

    Current models struggle to reproduce text elements and fine-grained details, and GPT-4V achieves an overall score of 7.68/10.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The benchmark is intended to support further exploration of multimodal reasoning, text-dense image understanding, and complex code generation in MLLMs.

    Source: §6 · p. 10

    Further work is identified for multimodal reasoning prompt strategies and vision encoders compatible with text-dense images.

    Source: §6 · p. 11

Abstract

from arXiv · show

The remarkable progress of Multi-modal Large Language Models (MLLMs) has attracted significant attention due to their superior performance in visual contexts. However, their capabilities in turning visual figure to executable code, have not been evaluated thoroughly. To address this, we introduce Plot2Code, a comprehensive visual coding benchmark designed for a fair and in-depth assessment of MLLMs. We carefully collect 132 manually selected high-quality matplotlib plots across six plot types from publicly available matplotlib galleries. For each plot, we carefully offer its source code, and an descriptive instruction summarized by GPT-4. This approach enables Plot2Code to extensively evaluate MLLMs' code capabilities across various input modalities. Furthermore, we propose three automatic evaluation metrics, including code pass rate, text-match ratio, and GPT-4V overall rating, for a fine-grained assessment of the output code and rendered images. Instead of simply judging pass or fail, we employ GPT-4V to make an overall judgement between the generated and reference images, which has been shown to be consistent with human evaluation. The evaluation results, which include analyses of 14 MLLMs such as the proprietary GPT-4V, Gemini-Pro, and the open-sourced Mini-Gemini, highlight the substantial challenges presented by Plot2Code. With Plot2Code, we reveal that most existing MLLMs struggle with visual coding for text-dense plots, heavily relying on textual instruction. We hope that the evaluation results from Plot2Code on visual coding will guide the future development of MLLMs. All data involved with Plot2Code are available at https://huggingface.co/datasets/TencentARC/Plot2Code.

1 Introduction

Plot2Code addresses the limited evaluation of multimodal code generation from plots with a benchmark designed to assess models across input modalities and code and image outputs. Evaluations show that current models still struggle with this task.

1 Introduction

  • Existing benchmarks leave a gap in evaluating diagrams within text-dense images, while unimodal code and image benchmarks offer limited coverage of multimodal inputs.The introduction argues that unit tests alone miss image fidelity and that multiple-choice formats are inadequate for realistic code-task evaluation.Source: §1 · p. 2
  • Plot2Code pairs 132 matplotlib plots across six plot types with source code and GPT-4-generated descriptions, supporting text, image, or combined inputs and code-pass-rate and GPT-4V image assessments.It includes Direct Asking and Conditional Asking settings for evaluating text and image outputs.Source: §1 · p. 3
  • GPT-4V achieves an overall score of 7.68/10, while evaluations of publicly available MLLMs show substantial challenges in visual coding.The introduction also characterizes GPT-4V, Gemini-Pro, and Claude-3 as demonstrating modest performance.Source: §1 · p. 3
  • The benchmark is intended to support research into MLLMs’ multimodal understanding and reasoning through evaluation of their coding capabilities.The paper presents Plot2Code as a specialized multimodal code benchmark.Source: §1 · p. 3

2 Related Work

Related work covers multimodal language models and code benchmarks, positioning Plot2Code as an evaluation of code generation from reference plots across unimodal and multimodal inputs.

2.1 Multi-modal Large Language Models

  • Research on multimodal LLMs develops encoders for processing multimodal inputs, including high-resolution vision encoders for text-dense documents and charts.Plot2Code assesses their ability to generate code from reference plots as a measure of visual coding proficiency.Source: §2.1 · p. 3

2.2 Multi-modal Code Benchmark

  • Plot2Code extends code benchmarks with varied unimodal and multimodal inputs and metrics for code pass rate and generated-plot similarity.The passage contrasts it with HumanEval and MBPP’s unit-test-based Pass@k, MMCode’s image inputs, and Design2Code’s generated HTML evaluation.Source: §2.2 · p. 3

3 Dataset Collection

Plot2Code curates manually filtered Matplotlib plot-code examples and evaluates direct and instruction-conditioned code generation using execution, text-match, and visual-similarity measures.

3.1 Test Set Curation

  • 841 scraped code blocks yielded 529 plot-code pairs after generation filtering and 132 manually curated test examples.Source: §3 · p. 4, §3.1 · pp. 4–5
  • The test set assumes static plots, filters animation, widget, and event-handling examples, and retains plots without external file dependencies across varied sizes, text, colors, types, and difficulty levels.Source: §3.1 · pp. 4–5

3.2 Evaluation Setting

  • Direct asking supplies an image, while conditional asking adds instructions; LLMs receive instructions alone in the conditional setting.GPT-4 generates instructions from ground-truth code while retaining essential reproduction information and avoiding implementation details.Source: §3.2 · p. 5

3.3 Data Statistics

  • The test set contains 293 subplots, averages 409 ground-truth code tokens, and averages 23 plot text elements.Subplots range from 1 to 18; code-token standard deviation is 291, and plot-text standard deviation is 13.Source: §3.3 · pp. 5–6

3.4 Evaluation Metrics

  • Evaluation combines code pass rate, GPT-4V ratings of visual similarity, and text-match ratio for plot text precision and extra text.The text-match ratio checks whether ground-truth text is reproduced accurately without extra text in generated images.Source: §3.4 · p. 6

3.5 Comparison with Other Datasets

  • Plot2Code covers the most extensive range of evaluation settings and metrics among the compared unimodal and multimodal code benchmarks.Source: §3.5 · p. 6, Table 2 · p. 6

4 Experiments

The experiments evaluate 14 closed- and open-source models using multiple settings and metrics, and find that Plot2Code remains challenging. Results vary with backbone, instructions, prompt strategy, and OCR support; pairwise evaluation also favors GPT-4V over GPT-4 and finds no clear advantage from Chain-of-Thought prompting.

4.1 Evaluation Details

  • The evaluation covers 14 representative closed- and open-source models with varied parameters, resolutions, and backbone LLMs, using direct- and conditional-asking settings and three reported metrics.The metrics are code pass rate, text-match ratio, and GPT-4V overall rating.Source: §4.1 · p. 7, §4.2 · p. 7

4.2 Overall Evaluation

  • 7.68, 7.10, and 7.68 are the conditional-asking overall scores for Claude-3-Opus, Gemini-Pro, and GPT-4V, respectively, while Gemini-Pro’s pass rate falls from 68.2% to 55.3% with instructions.The results indicate substantial room for improvement; the added instruction imposes requirements that make code generation harder.Source: §4.2 · pp. 7–8
  • Mini-Gemini-8x7B-HD leads the open-source models with a GPT-4V judgement score of 6.08 and a 58.4% code pass rate, but trails commercial closed-source models.The authors describe the open-source/closed-source performance gap as considerable.Source: §4.2 · p. 8

4.3 Influence of difference settings.

  • Across Mini-Gemini and LLaVA, performance strongly correlates with the backbone LLM, while conditional asking generally lowers pass rates and raises similarity versus direct asking.The authors attribute this pattern to stricter code requirements from instructions, which can also make generated images more similar to references.Source: §4.3 · p. 8
  • Chain-of-Thought and Plan-and-Solve prompts show no clear advantage over the default prompt, while adding PaddleOCR tokens yields improvements similar to high-resolution settings.The OCR ablation uses Mini-Gemini-8x7B-HD and PaddleOCR; the authors suggest current MLLMs may need more powerful vision encoders to capture image details.Source: §4.3 · pp. 8–9, Figure 5 · p. 8

4.4 Pairwise Model Comparison

  • In conditional-asking pairwise comparisons, GPT-4V judges which generated image is more similar to the reference, with GPT-4V without prompt strategies serving as the baseline.Images are swapped for a second evaluation; a model wins only if it wins both rounds, otherwise the result is a tie. GPT-4V with image input produces higher-quality plots than GPT-4, while Chain-of-Thought has no additional advantage.Source: Figure 4 · p. 8, §4.4 · p. 9

5 Statistical Analysis

The statistical analyses test whether image-similarity metrics distinguish groups with different visual similarity and whether GPT-4V judgements align with human evaluations. MSE and SSIM do not distinguish the tested groups at the chosen significance level, whereas GPT-4V judgement and text-match ratio do; GPT-4V judgements show moderate positive correlations with human ratings.

Hypothesis tests for image similarity metrics

  • For MSE and SSIM, p-values were 0.22 and 0.21, respectively, both exceeding the significance level; thus, the null hypothesis could not be rejected, suggesting these metrics might not distinguish the groups.The groups comprised GPT-4V images with high visual similarity and Mini-Gemini-2B images with considerably lower similarity.Source: §5 · pp. 9–10
  • GPT-4V-Judgement and Text-Match Ratio had p-values of 1.22 × 10−11 and 9.62 × 10−7, respectively, both below the significance level and CLIP-score’s 1.24 × 10−4, suggesting they distinguish the two image groups.Hypothesis testing provides statistical evidence based on the collected data, but does not prove whether either hypothesis is absolutely true.Source: §5 · p. 10

Correlative analysis between GPT-4V judgement and human evaluation

  • For 920 samples, Kendall’s Tau, Pearson, and Spearman coefficients each indicate a statistically significant moderate positive relationship between GPT-4V judgement and human evaluation.The authors interpret this as general agreement between GPT-4V and human evaluation when assessing image similarity.Source: §5 · p. 10

6 Conclusion

Plot2Code combines varied scientific plots with code pass rate, text-match ratio, and GPT-4V judgment to evaluate model code generation. The evaluation finds that reproducing text and fine-grained plot details remains challenging, while pointing to future research directions.

6 Conclusion

  • The benchmark covers varied scientific plot complexities and types, and evaluates models with code pass rate, text-match ratio, and GPT-4V judgment score.Together, the metrics are intended to provide a holistic evaluation of model performance.Source: §6 · p. 10
  • Models differ substantially on Plot2Code, and reproducing all text elements and fine-grained plot details remains challenging.Some models can generate executable code and plots similar to the reference image.Source: §6 · p. 10
  • Future work could explore multimodal reasoning prompt strategies and vision encoders compatible with text-dense images.The authors hope further research can reduce the gap between open-source MLLMs and closed-source commercial APIs.Source: §6 · pp. 10–11

A Prompt Template

The appendix gives prompts for generating matplotlib code from plots, generating textual plot instructions, and evaluating generated images. It also describes alternative reasoning prompts, including CoT and a modified Plan-and-Solve strategy.

A.1 Prompt for Code Generation

  • Direct Asking prompts the model to generate matplotlib code resembling the supplied image, while Conditional Asking prepends an instruction describing the reference plot.The prompts request Python code in a fenced code block and provide image tokens as input.Source: §A.1 · p. 14

A.2 Prompt for Instruction Generation

  • GPT-4 is prompted to inspect each plot’s code and describe the type, text, and other details needed for recreation without implementation details.The prompt requests everyday language and asks for random seeds and data when relevant.Source: §A.2 · p. 14

A.3 Prompt for Evaluation

  • The overall-rating prompt compares a ground-truth image with an assistant-generated image and requests an objective similarity rating from 1 to 10.The prompt asks the evaluator to consider appearance, colors, shapes, positions, and other visual elements.Source: §A.3 · pp. 14–15
  • The pairwise prompt asks GPT-4V to choose which assistant’s generated image is more similar to the ground truth, considering visual appearance, colors, shapes, and positions.The instructions also ask the evaluator to avoid bias from the presentation order.Source: §A.3 · p. 15

A.4 Prompt Strategy

  • Chain-of-Thought adds a request to think step by step at the start of the assistant’s response.The appendix presents this as an alternative prompt strategy.Source: §A.4 · p. 15
  • The modified Plan-and-Solve strategy, called PS+, first asks the assistant to describe the plot and make a detailed plan.If code is not produced in the first step, a second step requests the final answer based on that plan.Source: §A.4 · p. 16
  • The appendix wraps generated code in a Python code block to make it easier to extract with regular expressions.The second PS+ step asks the assistant to produce code based on its plan.Source: §A.4 · p. 16

B Case Study

The case study presents GPT-4V examples from direct asking, conditional asking, and pair evaluation against Gemini-Pro, using the default prompt strategy.

B Case Study

  • Examples show GPT-4V outputs in direct asking, conditional asking, and pair evaluation against Gemini-Pro, all with the default prompt strategy.The examples correspond to Figures 7, 8, and 9, respectively.Source: B Case Study · p. 16

C Correlation Analysis

The correlation analysis compares GPT-4V with Gemini-Pro and Claude-3-Opus on conditional-asking examples, then collects human judgments through an online questionnaire. The study uses 920 judgments from 46 completed questionnaires across 20 comparison samples.

C Correlation Analysis

  • The analysis uses 20 conditional-asking pair evaluations, with GPT-4V compared against Gemini-Pro and Claude-3-Opus in 10 samples each.The questionnaire presents the ground truth and both assistants’ generated images for each question.Source: C Correlation Analysis · p. 16
  • 46 completed questionnaires yield 920 judgments across the 20 samples, with participants choosing which image is closer or whether similarity is close.The figure caption indicates that the response order is swapped and evaluation is conducted twice for each pair.Source: C Correlation Analysis · p. 16, Figure 9 · p. 19

Related papers

Loading 2405.07990v1…