PaperlayerView exact revision on arXiv

Source-linked AI summary

From Pixel to Coding: Evaluating the Figure Reproduction Capabilities of MLLMs

Zijian Chen, Zhengyu Chen, Bohan Liang, Lirong Deng, Yushuo Zheng, Yanwei Jiang, Qi Jia, Kaiwei Zhang, Wenjun Zhang, Guangtao Zhai

arXiv:2610.10066v1cs.CVcs.AI

TL;DR

Existing benchmarks often assess visual understanding and code generation separately, and visual-code tasks commonly use simplified figures or limited programming settings. FigCodeBench evaluates figure reproduction across languages and categories with multidimensional metrics; experiments find broad performance cliffs across models and a particular bottleneck in LaTeX rendering.

  • Problem

    Existing benchmarks generally assess visual understanding and code generation separately, while visual-code benchmarks often use simplified layouts and limited programming settings.

    Source: §1 · p. 2

  • Method

    FigCodeBench evaluates figure reproduction with 6,194 figure-code pairs spanning four programming languages and seven categories, using multidimensional visual- and code-oriented metrics.

    Source: §1 · pp. 3–4

  • Results

    Experiments find widespread, non-linear performance cliffs across models and scenarios, with descriptive LaTeX a bottleneck across evaluated models.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The benchmark exposes weaknesses in MLLMs’ figure reproduction, including limits in 2D spatial reasoning and fine-grained aesthetic rendering.

    Source: §1 · p. 4, §5 · p. 35

    Because FigCodeBench targets plotting code generation, its findings may not generalize to other programming tasks, and MLLM judges may miss visually subtle but scientifically consequential errors in specialized figures.

    Source: §5 · p. 35

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in both visual understanding and code generation. However, existing benchmarks typically evaluate these two modalities in isolation, lacking a dedicated assessment of their unification, i.e., how a model can perceive complex visual structures and synthesize them into precise, executable code. Moreover, current visual code generation benchmarks often rely on simplified layouts within single programming environments, falling short of evaluating true unified multimodal reasoning. To bridge this gap, we propose FigCodeBench, a comprehensive framework for rigorously evaluating MLLMs on figure reproduction, integrating multimodal comprehension and generation. We first design a systematic dataset construction pipeline, resulting in a total of 6,194 instances that cover 7 functional categories and 4 types of programming languages. We further categorize figure reproduction into three tiers with visual and code complexity modeling, specifically targeting complex structural reasoning, varying aspect ratios, and dense geometric constraints. We introduce a multi-dimensional evaluation protocol, encompassing visual fidelity and syntactic isomorphism, that aligns highly with the Mean Machine Opinion Score (MMOS) and human preferences. Based on our framework, we conducted extensive experiments on 24 widely used proprietary and open-source MLLMs (e.g., Gemini 3.1 Pro, GPT-5.4, and Kimi-K2.5), where we observed a universal, non-linear performance cliff across different programming languages and difficulty scenarios for all models, and gained several insights, such as the significant metric decline in rigid declarative languages.

1 Introduction

Existing benchmarks largely separate visual comprehension from code generation, while visual-code benchmarks often simplify figures and programming settings. FigCodeBench evaluates MLLMs on reproducing complex scientific figures across languages, and experiments reveal substantial challenges and performance cliffs.

1 Introduction

  • Existing evaluations separate code generation from visual comprehension, leaving MLLMs’ ability to turn visual understanding into complex executable outputs untested.Code benchmarks commonly use text specifications, while multimodal evaluations often stop at VQA or multiple choice; CharXiv lacks executable validation and active figure reproduction.Source: §1 · p. 2
  • Across evaluated models, performance differs substantially by programming language, with descriptive LaTeX a bottleneck for all models.Fig. 1 compares MMOS against cost and shows performance across image categories; the experiments also identify weaknesses in 2D spatial reasoning, abstract set relationships, and fine-grained aesthetic rendering.Source: Figure 1 · p. 3, §1 · p. 4
  • Earlier visual-code benchmarks often use simplified plots or web layouts and single-language settings, with evaluation that misses structural fidelity across programming environments.The paper highlights missing mathematical rigor, dense geometric constraints, and multi-paradigm complexity in professional scientific figure reproduction.Source: §1 · p. 3
  • FigCodeBench evaluates reproduction of complex scientific figures across Python, Matlab, R, and LaTeX, covering seven function-focused categories and 6,194 figure-code pairs.The dataset uses human-in-the-loop augmentation to mitigate contamination and includes typographic layouts, geospatial projections, vector fields, flowcharts, and abstract diagrams.Source: §1 · p. 3
  • Its evaluation combines image- and code-oriented metrics, including a joint measure of visual quality relative to code length for reproduction efficiency.The metric system is reported to align with machine judgments and human evaluations.Source: §1 · p. 4
  • Experiments on 11 proprietary and 13 open-source MLLMs find widespread performance cliffs and divergent coding strategies, with few models balancing visual comprehension and reliable code generation.The experiments also report an impact of reasoning status and model size, and high correlation between the adopted metrics and human evaluation.Source: §1 · p. 4

2 Related Work

Related work traces MLLMs’ expansion across multimodal tasks, surveys visual-language evaluation, and identifies a gap in code benchmarks: they do not adequately test the synergy of visual grounding and code generation.

2.1 Multimodal Large Language Models

  • MLLMs have expanded from visual perception toward multimodal reasoning across images, video, and 3D, with applications spanning scientific, medical, and GUI tasks.Recent work also extends models to audio, code, action, and omni-modal capabilities.Source: §2.1 · pp. 4–5

2.2 Evaluation of MLLMs

  • Visual-language benchmarks expanded from VQA to OCR, chart understanding, abstract reasoning, video, and scientific tasks; FigCodeBench connects perception evaluation with code generation.The paper frames this connection as relevant to capabilities used in scientific analysis workflows.Source: §2.2 · p. 5

2.3 Code Generation Benchmarks

  • Code-generation benchmarks cover multiple languages, competition problems, code repair, execution, and software maintenance, but predominantly serve LLMs rather than testing MLLMs’ visual grounding with code generation.Examples include HumanEval, LiveCodeBench, and SWE-Bench.Source: §2.3 · pp. 5–6

3 Benchmark Curation

This section presents FigCodeBench as a benchmark for evaluating how MLLMs reproduce scientific visualization figures from images and task instructions using executable code. It describes its curation and complementary image- and code-oriented assessment, and compares its visual-property coverage and diversity with other benchmarks.

3.1 Design Philosophy

  • FigCodeBench targets cross-modal figure reproduction using authentic, in-the-wild references across Python, Matlab, R, and LaTeX.The four programming spaces have distinct syntax, rendering paradigms, and API logic; tasks include an image, target language, and auxiliary constraints or context.Source: §3.1 · pp. 6–7

3.2 Data Collection Pipeline

  • Curation combines source-code collection, attribute-guided perturbation, auxiliary-information extraction, and automated plus expert quality control.AST- and regex-based edits alter semantic or stylistic attributes, while five researchers also contributed preference-guided examples; two additional variants per seed yielded over 12,000 augmented pairs before filtering to the final collection.Source: §3.2 · pp. 8–9

3.3 Data Statistics and Division

  • 6,194 figure-code pairs span four programming languages and seven functional categories, with 2,548 unique instances treated as prototypes.The dataset does not enforce uniform language proportions; it reflects programming-language usage in specific scenarios, and user-generated samples have longer, more complex code sequences.Source: §3.3 · p. 10
  • Difficulty tiers combine visual and code complexity, with Easy, Medium, and Hard assigned by the 33rd and 67th percentiles of their equally weighted overall score.Visual complexity uses edge density, line-orientation and color diversity, foreground components, and spatial occupancy; code complexity uses effective token count, AST structure, and plotting-API calls.Source: §3.3 · pp. 10–12

3.4 Evaluation Protocol

  • Evaluation combines deterministic image- and code-oriented metrics with a three-MLLM jury that averages similarity ratings into MMOS.Image assessment includes PSNR, SSIM, LPIPS, and CIEDE2000, aggregated with execution-rate weighting; code assessment uses complementary semantic, lexical, AST, and plotting-API measures. Figure–Code Fidelity additionally rewards visual similarity relative to code length and requires successful execution.Source: §3.4 · pp. 12–14

3.5 Comparisons with Existing Benchmarks

  • Compared with existing benchmarks, FigCodeBench emphasizes real-world fidelity, broader language and figure-type coverage, and hybrid deterministic and MLLM-judge evaluation.Its data sources and LLM- and human-involved augmentation are intended to address limitations of synthetic or raw-crawled data, while its content-diversity analysis compares five low-level visual features across eight benchmarks.Source: §3.5 · pp. 15–17, Figure 4 · p. 15
  • FigCodeBench’s broadest sharpness and SI distributions, alongside its overall coverage, indicate greater diversity than the compared benchmarks.Plot2Code is concentrated at lower sharpness and SI values.Source: §3.5 · p. 17
  • FigCodeBench and ChartX show similar brightness and contrast styles and coverage, whereas ChartLlama, MMCode, Design2Code, and MatPlotBench are relatively simple and unevenly distributed.Source: §3.5 · p. 17

4 Experiments

The experiments evaluate 24 proprietary and open-source MLLMs and analyze how programming language affects generation length and evaluation cost. LaTeX has the highest resource demands, while R is the most economical language to evaluate.

4.1 Evaluation Setup

  • The evaluation included 24 proprietary and open-source MLLMs, with three MLLMs judging figure similarity across text, layout, type, data, and style.Each judge returned a 0–100 similarity score; failed executions received 0.Source: §4.1 · pp. 17–19

4.2 Analysis of Evaluation Resource Consumption

  • LaTeX had the highest AvgTok (median ≈831) and highest AvgCost across most models; R was the most economically efficient language.For Gemini 3.1 Pro, LaTeX cost $0.1739 per instance versus $0.1222 in Python, an approximate 42% increase.Source: Table 4 · p. 18, §4.2 · pp. 19–20
  • The resource analysis used Friedman tests to compare languages within models, addressing right-skewed token and cost distributions and cross-model differences.AvgTok analysis included all 24 models; AvgCost analysis included 18 API-used models.Source: §4.2 · p. 19
  • AvgTok differed significantly by language (χ2(3, N = 24) = 19.35, p < 0.001), with LaTeX significantly above R and Python.The reported pairwise tests give p < 0.01 for LaTeX versus R and p < 0.05 for LaTeX versus Python.Source: §4.2 · pp. 19–20
  • Ranking analysis found LaTeX had the highest evaluation cost across most models; Gemini 3.1 Pro averaged $0.1739 per instance in LaTeX, about 42% above Python ($0.1222).The AvgCost analysis covered 18 API-used models.Source: §4.2 · pp. 19–20

4.3 Benchmark Results

Across the evaluated models, figure-reproduction performance varies substantially by language and figure type, with LaTeX a recurring bottleneck. The results also show that execution, visual fidelity, code similarity, and human-aligned judgment capture distinct aspects of reproduction.

Performance Comparison on Code Executability

  • Python has the highest execution rates, followed by R, MATLAB, and LaTeX; the Gemini 3 series also gains nearly 40% in LaTeX execution rate over Gemini 2.5.The cross-language differences are tested on paired model-level measurements.Source: §4.3 · pp. 21–22, Figure 7 · p. 21

Performance Comparison on Image-oriented Metrics

  • Claude Opus 4.7’s high execution reliability does not ensure visual fidelity, while Kimi-K2.5 leads all 24 models on R image metrics.Kimi-K2.5 records an execution rate of 91.68%, PSNR of 12.610, SSIM of 0.699, and LPIPS of 0.459 in R.Source: §4.3 · p. 22

Performance Comparison on Code-oriented Metrics

  • Claude Opus 4.7 has near-bottom code-similarity scores despite high execution reliability, whereas Gemini 3.1 Pro leads Python SAST (0.741) and FAPI (0.646).CrystalBLEU peaks at only 0.306 for Gemini 3.1 Pro in R; the authors find SAST and FAPI more discriminative than exact lexical matching.Source: §4.3 · pp. 22, 24

Performance Comparison on Generation Efficiency

  • Gemini 3.1 Pro leads FCF in Python and Matlab, but Qwen2.5-VL-72B leads in R and LaTeX, reversing the efficiency hierarchy across languages.FCF relates visual quality to the volume of generated code; the authors speculate that concise macro commands contribute to the open-source models’ R and LaTeX results.Source: Figure 11 · p. 25, §4.3 · pp. 26–27

MLLM Judgment

  • Gemini 3 Flash (52.43) and Claude Opus 4.7 (51.81) achieve the highest reported MMOS values, but both barely exceed marginal acceptability.The MMOS results are rated by three MLLM judges.Source: Table 7 · p. 21, §4.3 · p. 25

Performance on Different Image Types

  • Statistical, temporal, and compositional figures are generally easier than geospatial figures; Gemini 3 Flash-TK scores 78.33, 79.60, and 74.59 in those categories, respectively.Geospatial scores include 65.57 for Gemini 3.1 Pro, 21.67 for GPT-5.2, and 4.40 for InternVL3.5-8B; conceptual and relational categories also expose spatial-reasoning weaknesses.Source: Table 8 · p. 26, §4.3 · p. 27

Comparison Between Reasoning Models and Their Non-reasoning Counterparts

  • Gemini 3 Flash reasoning raises MMOS in Python (76.69 to 78.51), Matlab (68.40 to 74.99), and R (69.76 to 73.01), despite lower pixel-level and code-similarity metrics.By contrast, enabling reasoning in Gemini 2.5 Flash-TK lowers execution rates and structural similarity across all four languages.Source: §4.3 · pp. 27–28

Model Size Analysis

  • Within-family parameter scaling generally raises MMOS for Qwen3-VL and InternVL3.5 in Python, Matlab, and R, while Qwen3.5 and GLM show inverse scaling.LaTeX is an exception: the authors report that a stronger language backbone can improve geometric reproducibility in this declarative, text-intensive setting.Source: §4.3 · pp. 28–29

4.4 Correlation Between Different Metrics

The evaluation compares image-oriented metrics with MMOS across four coding environments. PSNR and SSIM track similar low-level errors, while LPIPS and color difference capture other aspects of visual fidelity.

4.4 Correlation Between Different Metrics

  • PSNR and SSIM show consistently strong positive correlations, whereas LPIPS and ∆E trend negatively against them.The correlations are not perfectly linear; Kendall’s τ between ∆E and PSNR in R is -0.34.Source: §4.4 · p. 30
  • LPIPS aligns better with human preferences than the other image-oriented metrics, while all show moderate-to-strong correlations with MMOS across four languages.The experiment excludes code-oriented metrics because they quantify similarity rather than final graphical fidelity.Source: §4.4 · p. 30, Table 9 · p. 30
  • MMOS comparisons are highly consistent across languages, unlike image-oriented metric comparisons.The authors interpret the pronounced decay in cross-language image-metric correlations as evidence of language-specific syntactic priors in figure reproduction.Source: §4.4 · p. 30

4.5 Correlation Between FigCodeBench and Other Benchmarks

FigCodeBench performance relates more strongly to chart understanding than to general code generation. The authors connect this pattern to figure reproduction’s need for precise visual and geometric reasoning.

4.5 Correlation Between FigCodeBench and Other Benchmarks

  • FigCodeBench correlates more strongly with ChartXiv (r = 0.797) than with LiveCodeBench v6 (r = 0.537).ChartXiv represents chart understanding and LiveCodeBench v6 represents code generation.Source: §4.5 · p. 31
  • The comparison suggests that generic algorithmic programming proficiency does not directly translate to visual code generation.The authors identify deterministic coordinate reasoning and precise geometric mapping as requirements of figure reproduction.Source: §4.5 · p. 31
  • The authors describe accurate visual comprehension as an indispensable basis for cross-modal image reproduction.They present FigCodeBench as addressing the evaluation gap between cross-modal perception and generation.Source: §4.5 · p. 31

4.6 Human Evaluation

The authors compare multi-level MLLM judgments with expert ratings on sampled figure pairs. Aggregate MMOS aligns strongly with human judgments, while agreement varies across evaluation dimensions.

4.6 Human Evaluation

  • Human evaluation covered 1,000 generated-reference image pairs, rated by 12 experts on five dimensions using scores from 0 to 20.The experts had over three years of experience with related programming languages and scientific diagrams.Source: §4.6 · p. 32
  • Aggregated MMOS correlates with human evaluation at r = 0.826, while Gemini-3.5-Flash achieves the highest individual correlation at r = 0.837.The authors say aggregation mitigates biases from a single weaker model, such as J1 at 0.793.Source: §4.6 · p. 32
  • Chart-type and text-content judgments correlate more highly with human ratings than style and data judgments, and human raters achieve Krippendorff’s α = 0.766.The authors partly attribute the lower alignment of abstract attributes to variance and limitations in fine-grained spatial mapping by current visual encoders.Source: §4.6 · p. 32

4.7 Performance on Different Difficulty Tiers

Performance declines as difficulty rises across languages, with a sharper drop from medium to hard tasks. Qualitative comparisons also show weaknesses in spatial layout planning, fine-grained rendering, and some code generations.

4.7 Performance on Different Difficulty Tiers

  • All models decline as difficulty rises, with a much sharper drop from medium to hard than from easy to medium.Qwen3-VL-32B falls 77.2% from an average Easy score of 47.69 to 10.85 on Hard tasks; Gemini 3.1 Pro and Qwen3.6-Plus score 30.04 and 31.11 on Hard tasks, respectively.Source: §4.7 · pp. 32–33
  • Hard-tier demands—including extreme aspect ratios, intricate mathematical annotations, and dense overlapping geometry—remain difficult, especially for weaker models.Qwen3.6-Plus MMOS is 57.90, 49.82, and 31.11 at easy, medium, and hard levels, respectively.Source: §4.7 · pp. 32–33

4.8 Case Study

  • Gemini 3.1 Pro has the strongest visual reproduction among the six compared models, while others struggle with gradients, line styling, color accuracy, and dense text layouts.In the R heatmap example, GPT-5.4, Kimi-K2.5, and Qwen3-VL-235B have overlapping text that becomes illegible.Source: §4.8 · p. 33, Figure 18 · p. 34
  • Some generations fail to compile or reproduce the reference structure, including a LaTeX tree that substitutes stylistic attributes for human names.The authors identify spatial layout planning, collision avoidance, aspect-ratio control, and fine-grained aesthetic rendering such as continuous colormaps as remaining weaknesses.Source: §4.8 · pp. 33, 35

5 Conclusions

FigCodeBench evaluates unified visual perception and coding for figure reproduction across 6,194 figure-code pairs, four programming languages, and seven figure categories. Experiments reveal weaknesses in precise reproduction and inconsistency between reproduction and coding capabilities; the authors also identify limitations in generalizability and MLLM-based judging.

5 Conclusions

  • Experiments on 11 proprietary and 13 open-source MLLMs reveal inferiority in precise figure reproduction and inconsistency between figure reproduction and code-generation capabilities.The benchmark also investigates correlations across metrics, programming languages, and ability dimensions.Source: §5 · p. 35
  • The benchmark contains 6,194 figure-code pairs across four programming languages and seven function-focused figure categories, developed through a human-in-the-loop and semi-automatic process.Its design principles include comprehensiveness, fidelity, reproducibility, and determinism.Source: §5 · p. 35
  • Findings may not fully generalize beyond plotting code generation, and general-purpose MLLM judges may miss scientifically consequential but visually negligible errors in specialized figures.The authors identify subtle alignment biases as an inherent risk of MLLM-as-a-judge evaluation.Source: §5 · p. 35

Declarations

The declarations provide planned access points for FigCodeBench materials, state that the authors have no competing interests, and acknowledge funding support.

Declarations

  • The website, codebase, and dataset are planned for availability through the project website, GitHub, and Hugging Face, respectively.The authors invite community feedback and scrutiny of the benchmark.Source: Declarations · p. 35
  • The authors declare no competing interests.Source: Declarations · p. 35
  • The work received support from China’s National Key R&D Program and Shanghai’s municipal special program, in collaboration with Shanghai Artificial Intelligence Laboratory.The funding details continue across the two declaration passages.Source: Declarations · pp. 35–36

A Prompt Templates

The appendix gives templates for generating figure-reproduction code and judging rendered plots. The prompts specify generation inputs and output constraints, plus a five-criterion visual scoring procedure.

A.1 Prompt for Code Generation

  • Uniform canvas geometry, including DPI and aspect ratio, controls the resolution of rendered figures.Source: §A.1 · p. 37
  • The code-generation prompt asks an expert developer to reproduce a supplied figure in the specified programming language.Source: §A.1 · p. 37
  • The prompt supplies target figure dimensions and auxiliary package, dataset, and random-seed information to guide code generation.Source: §A.1 · p. 37
  • The generation prompt requires the model to return code only, without additional explanations.Source: §A.1 · p. 37

A.2 Prompt for Jury Models

  • The jury prompt compares a reference plot with an AI-generated plot to assess reproduction accuracy.Source: §A.2 · p. 38
  • The jury scores chart types, layout, text content, data, and style, with each criterion worth up to 20 points.Source: §A.2 · p. 38
  • The jury prompt requires a JSON response containing scores for the five criteria and their summed final score.Source: §A.2 · p. 38
Loading 2610.10066v1…