PaperlayerView exact revision on arXiv

Source-linked AI summary

VisCoder2: Building Multi-Language Visualization Coding Agents

Yuansheng Ni, Songcheng Cai, Xiangchao Chen, Jiarong Liang, Zhiheng Lyu, Jiaqi Deng, Kai Zou, Ping Nie, Fei Yuan, Xiang Yue, Wenhu Chen

arXiv:2510.23642v2cs.SEcs.AIcs.CLcs.PL

TL;DR

Visualization coding agents remain constrained by narrow language coverage, unreliable execution, and limited iterative correction resources. The paper introduces a validated twelve-language dataset, an eight-language benchmark, and VisCoder2 models, which outperform open-source baselines and reach an 82.4% execution pass rate at 32B with self-debug. The resources support systematic evaluation and realistic correction loops, although language imbalance and incomplete benchmark coverage remain limitations.

  • Problem

    Existing visualization coding resources lack broad language coverage, reliable execution validation, and multi-turn support for iterative debugging.

  • Method

    The paper introduces VisCode-Multi-679K, VisPlotBench, and VisCoder2, combining validated multi-language code, correction dialogues, standardized evaluation, and model training.

  • Results

    82.4% overall execution pass rate at the 32B scale is reached with iterative self-debug, while VisCoder2 outperforms open-source baselines and approaches GPT-4.1.

  • Takeaways & Limitations

    The resources establish a systematic foundation for multi-language visualization coding agents with realistic execution-based correction loops.

  • Takeaways & Limitations

    The training corpus is imbalanced across languages, and VisPlotBench currently covers only eight visualization languages.

Abstract

from arXiv · show

Large language models (LLMs) have recently enabled coding agents capable of generating, executing, and revising visualization code. However, existing models often fail in practical workflows due to limited language coverage, unreliable execution, and lack of iterative correction mechanisms. Progress has been constrained by narrow datasets and benchmarks that emphasize single-round generation and single-language tasks. To address these challenges, we introduce three complementary resources for advancing visualization coding agents. VisCode-Multi-679K is a large-scale, supervised dataset containing 679K validated and executable visualization samples with multi-turn correction dialogues across 12 programming languages. VisPlotBench is a benchmark for systematic evaluation, featuring executable tasks, rendered outputs, and protocols for both initial generation and multi-round self-debug. Finally, we present VisCoder2, a family of multi-language visualization models trained on VisCode-Multi-679K. Experiments show that VisCoder2 significantly outperforms strong open-source baselines and approaches the performance of proprietary models like GPT-4.1, with further gains from iterative self-debug, reaching 82.4% overall execution pass rate at the 32B scale, particularly in symbolic or compiler-dependent languages.

1 INTRODUCTION

Visualization coding agents can generate, execute, and revise code, but practical use is limited by unreliable execution, narrow language coverage, and weak iterative correction support. The paper addresses these gaps with a multi-language dataset, benchmark, and model family.

  • Existing visualization coding agents often crash, produce incorrect visuals, or lack flexibility across programming languages and libraries.
  • Visualization offers immediate execution and rendering feedback while requiring joint reasoning about data, plotting syntax, and design conventions.
  • Current datasets and benchmarks lack validated multi-language samples, runtime validation, and multi-turn interactions for iterative debugging.
  • VisCode-Multi-679K provides 679K executable visualization and correction samples across twelve languages, while VisPlotBench evaluates eight languages through initial generation and multi-round self-debug.
  • VisCoder2 substantially outperforms size-matched open-source baselines and reaches an 82.4% overall execution pass rate at 32B with iterative self-debug.

2 RELATED WORK

Prior visualization code-generation systems and benchmarks largely emphasize one or two languages, limited execution validation, and single-turn evaluation. Broader code datasets provide language coverage but lack visualization-specific knowledge and validation.

  • Most visualization code-generation systems focus on Python, Vega-Lite, or HTML and often lack systematic execution validation.
  • General multi-language code-generation datasets provide broad language coverage but lack specialized visualization knowledge, including for LaTeX and LilyPond.
  • Self-debug research enables code generation, execution, and refinement through feedback, motivating iterative evaluation for visualization agents.
  • Existing visualization benchmarks predominantly cover Python or declarative formats and usually evaluate only single-turn generation.

3 VISCODE-MULTI-679K: AN INSTRUCTION TUNING DATASET FOR VISUALIZATION ACROSS TWELVE PROGRAMMING LANGUAGES

VisCode-Multi-679K is a twelve-language instruction-tuning corpus that combines executable visualization code with feedback-driven correction dialogues. Its pipeline filters, reconstructs, validates, renders, and describes examples for both generation and iterative refinement.

  • 3 VISCODE-MULTI-679K: AN INSTRUCTION TUNING DATASET FOR VISUALIZATION ACROSS TWELVE PROGRAMMING LANGUAGES: VisCode-Multi-679K supplies multi-language generation and iterative refinement supervision across twelve programming languages.
  • 3 VISCODE-MULTI-679K: AN INSTRUCTION TUNING DATASET FOR VISUALIZATION ACROSS TWELVE PROGRAMMING LANGUAGES: The dataset combines executable repository code paired with rendered outputs and over 66K Code-Feedback dialogues for revising faulty programs from runtime signals.
  • 3 VISCODE-MULTI-679K: AN INSTRUCTION TUNING DATASET FOR VISUALIZATION ACROSS TWELVE PROGRAMMING LANGUAGES: Its construction uses library filtering, code extraction, runtime validation, and instruction generation as four pipeline stages.
  • 3.1 CODE EXTRACTION FROM PUBLIC REPOSITORIES: The source corpora combine broad code coverage, domain-specific SVG diagrams, and synthetic structured visualization snippets across diverse languages and styles.
  • 3.1 CODE EXTRACTION FROM PUBLIC REPOSITORIES: Filtering identifies approximately 5.3M visualization candidates, extracts standalone blocks, injects mock inputs when needed, and yields roughly 900K candidate blocks.
  • 3.1 CODE EXTRACTION FROM PUBLIC REPOSITORIES: SVG filtering retains about 79K candidates, while CoSyn-400K contributes 408K snippets across eight languages requiring reconstruction where code and data are separated.
  • 3.1 CODE EXTRACTION FROM PUBLIC REPOSITORIES: Runtime validation retains only blocks that execute successfully and produce valid non-monochrome images larger than 10KB, producing 245K validated plotting scripts.
  • 3.2 MULTI-TURN INSTRUCTION-FOLLOWING DIALOGUES WITH EXECUTION FEEDBACK: The corpus integrates multi-turn Code-Feedback interactions so models practice both initial generation and revision based on execution feedback.

4 VISPLOTBENCH: MULTI-LANGUAGE BENCHMARK FOR VISUALIZATION CODING AGENTS

VisPlotBench is a multi-language benchmark built from executable visualization tasks paired with instructions and rendered outputs. It standardizes execution-grounded scoring for initial generation and up to three rounds of self-debug.

  • VisPlotBench spans eight languages and standardizes evaluation of both initial code generation and multi-round self-debug across multiple visualization formats.
  • The benchmark contains 888 tasks across 13 visual categories and 116 subtypes, including underrepresented areas such as Hierarchies, Music, and Networks & Flows.
  • Each task combines a natural-language instruction, executable code, and rendered output, with verified execution, deduplication, and human quality review.
  • 4.3 TASK CONSTRUCTION: Tasks use a five-part instruction schema covering setup, plot instructions, data instructions, task description, and style description.
  • The execute–render–score pipeline emits an image, execution log, and metadata record from isolated language-specific runtimes.
  • Execution Pass Rate checks successful valid visualization execution, while Task Score and Visual Score assess instruction compliance and perceptual similarity.
  • Self-debug revisits unresolved tasks for up to three rounds using prior code and execution-log excerpts, scoring the best attempt.

5 EXPERIMENT SETUP

The experiments fine-tune Qwen2.5-Coder-Instruct across four model sizes and evaluate visualization coding on VisPlotBench with execution, semantic, perceptual, and self-debug measures.

  • Training Setup: Qwen2.5-Coder-Instruct is fine-tuned at 3B, 7B, 14B, and 32B scales to assess dataset generalizability across capacities.All models use three epochs, a 5 × 10^-6 learning rate, bfloat16 full-parameter tuning, and 8×H100 GPUs.
  • Evaluation Setup: VisPlotBench evaluation reports Execution Pass Rate, Task Score, and Visual Score, measuring executability, semantic alignment, and perceptual similarity.Models are also tested with up to three self-debug rounds using execution feedback.

6 MAIN RESULTS

VisCoder2 improves multi-language visualization execution over size-matched open-source baselines, reaches proprietary-model parity at 32B, and gains further from iterative self-debug, especially on symbolic languages.

  • Overall Comparison: Symbolic and compiler-dependent languages remain the hardest: GPT-4.1 scores below 45% on LilyPond and under 25% on Asymptote.Vega-Lite and HTML are near saturation, while Python improves steadily with scale.
  • Overall Comparison: At 32B, VisCoder2 improves overall execution pass rate by approximately 15 points over Qwen2.5-Coder and reaches parity with GPT-4.1.Across scales, it consistently outperforms size-matched open-source baselines, with SVG as the consistent shortfall.
  • Self-Debug: Self-debugging raises VisCoder2’s overall execution by nearly ten points at larger scales, with especially strong effects on LilyPond, LaTeX, and Asymptote.Iterative correction uses execution feedback to address fragile syntax and compilation errors.
  • Task and Visual Score Analysis: On LaTeX, GPT-4.1 improves from 31.3% to 66.1% execution with Self-Debug while task scores remain around 50, showing execution–semantics mismatch.VisCoder2 improves execution and task scores over baselines, but compilation errors remain frequent.
  • Task and Visual Score Analysis: On LilyPond, VisCoder2 reaches 69.1% execution at 7B versus 5.5% for Qwen2.5-Coder, with Self-Debug providing further gains.The proportion of examples scoring above 75 increases by more than tenfold.
  • Error Analysis: Self-debug reduces Python interface errors from 13 to 3 and LilyPond structural errors from 14 to 10, but semantic and runtime failures persist.Undefined-variable errors in LaTeX decrease only from 28 to 23, while Vega-Lite rendering errors remain at 2 to 2.
  • Training Data Ablation: Training-data ablations show that the full VisCode-Multi-679K mixture reaches 70.9% default and 76.4% self-debug pass rates, outperforming single-source variants.Code-Feedback primarily improves recovery, raising overall execution from 55.2% to 63.1%, while synthetic data adds symbolic-language coverage.

7 CONCLUSION

The paper addresses gaps in visualization coding agents with multi-language data, benchmarking, and models that support executable generation and iterative correction. Its experiments emphasize broad language coverage and self-debug as central to practical reliability.

  • VisCode-Multi-679K, VisPlotBench, and VisCoder2 provide a framework combining multi-language training data, standardized evaluation, and visualization models.The resources unify executable code, correction dialogues, benchmark evaluation, and model training for visualization coding agents.
  • VisCoder2 consistently outperforms open-source baselines and approaches proprietary models in execution reliability.
  • Broad multi-language coverage is essential because symbolic and compiler-dependent languages remain challenging but are decisive for generalization.The paper specifically identifies LaTeX, LilyPond, and Asymptote as difficult languages.
  • Iterative self-debug delivers large gains across models, especially where structural and semantic errors are common.
  • The proposed resources aim to support agents that operate through realistic correction loops for data analysis, reporting, and related workflows.

LIMITATIONS

The resources improve multi-language visualization coding coverage but remain constrained by imbalanced training data and incomplete benchmark language coverage.

  • High-resource languages such as Python and Vega-Lite are better represented than symbolic and domain-specific languages, potentially biasing models toward dominant languages.
  • VisPlotBench currently covers eight visualization languages, leaving extension to additional frameworks and languages as a scope boundary.

A PROMPT USED AND INSTRUCT DESIGN

The appendix documents prompts and instruction designs used to construct VisCode-Multi-679K and VisPlotBench. These prompts standardize visual description, code extraction, generation, and evaluation across languages and data settings.

  • A.1 PROMPT USED IN VISCODE-MULTI-679K: VisCode-Multi-679K construction uses prompts for multiple language families, including Python, JavaScript, TypeScript, C++, R, HTML, and SVG.
  • A.1 PROMPT USED IN VISCODE-MULTI-679K: The extraction prompt asks agents to retain minimal runnable visualization code, remove unrelated logic, and create meaningful mock data when inputs are missing.
  • A.1 PROMPT USED IN VISCODE-MULTI-679K: The-stack-v2 and svg-diagrams prompts extend descriptions to include data generation, while requiring structured plain-text responses.
  • A.1 PROMPT USED IN VISCODE-MULTI-679K: Data-driven visualization prompts require four-part descriptions covering setup, data or content, conceptual output, and style.
  • A.1 PROMPT USED IN VISCODE-MULTI-679K: Prompts for non-data-driven languages instead describe setup, visible visual elements, conceptual output, and style without language constructs.
  • A.2 PROMPT USED IN VISPLOTBENCH: VisPlotBench task-and-style prompts ask GPT-4.1 to infer a visual's task and appearance while omitting libraries and specification keywords.
  • A.2 PROMPT USED IN VISPLOTBENCH: VisPlotBench visual judging compares generated and ground-truth plots for plot correctness on a 0–100 scale, while task judging scores adherence to the stated task.
  • A.3 INSTRUCT DESIGN IN VISPLOTBENCH EVALUATION: Evaluation prompts constrain generated code by language: Python uses specified plotting libraries and intact data, whereas Vega-Lite requires one valid v6 JSON specification with exact field names.

C BREAKDOWN MAIN RESULTS

The breakdown reports VisPlotBench performance by visualization language using execution, quality, and threshold-based metrics. The supplied table passages identify the reporting structure but not the underlying model values.

  • VisPlotBench reports execution pass rate, mean visual and task scores, and the proportion of samples scoring at least 75 for each visualization language.
  • Tables 8–10 organize performance for selected languages by model using Exec Pass, Mean, and Good metrics.
  • The tables mark the best-performing model at each scale in bold and the second-best model with underlining.

D BREAKDOWN SELF-DEBUG RESULTS

The self-debug breakdown reports execution pass rates by language across up to three automatic correction rounds, covering Python, Vega-Lite, LilyPond, Mermaid, SVG, LaTeX, Asymptote, and HTML.

  • Execution pass rates are reported for each language across up to three rounds of automatic correction.The tables cover Python and Vega-Lite, LilyPond and Mermaid, SVG and LaTeX, and Asymptote and HTML.

E BREAKDOWN ERROR TYPE RESULTS

This section combines execution-error breakdowns, the self-debug protocol, task and visual scoring, and additional benchmark results. It evaluates whether iterative correction improves executable and rendered visualization outputs.

  • Error-Type Results: The error analyses report execution-error distributions for GPT-4.1 and VisCoder2-32B across self-debug rounds and eight visualization languages.The tables cover Python, Vega-Lite, LilyPond, Mermaid, SVG, LaTeX, Asymptote, and HTML.
  • Self-Debug Evaluation Protocol: The self-debug protocol retries only failed tasks for up to three feedback-driven correction rounds, then evaluates each task using its latest output.This mirrors a generate–execute–repair workflow while standardizing recovery measurement across languages and error types.
  • Task and Visual Score Metrics: Task Score compares generated plots with task instructions, while Visual Score compares them with ground-truth images.These metrics complement execution pass rate by measuring semantic task alignment and perceptual visual alignment.
  • Results on PandasPlotBench: VisCoder2 consistently outperforms base Qwen2.5-Coder models across execution, task, and visual metrics on all three PandasPlotBench libraries.Self-debug further improves results, with VisCoder2-14B approaching GPT-4.1.
  • Results on Human-Eval: VisCoder2 shows only a modest 2–3 point decrease versus base Qwen2.5-Coder models on HumanEval and HumanEval+ Pass@1.The paper reports stable general coding ability alongside gains on cross-language executable visualization generation and self-debug.

F.3 DEEP ANALYSIS OF SELF-DEBUG BEHAVIOR AND TASK/VISUAL SCORE

The deep analysis finds that self-debug chiefly improves execution reliability through early structural repairs, while semantic and visual gains vary substantially by language. It also characterizes dataset diversity, redundancy, and persistent failure modes.

  • Effect of self-debug: Self-debug consistently improves execution reliability, with especially large gains for VisCoder2-32B in symbolic languages such as LilyPond, Asymptote, and LaTeX.The mechanism is strongest for syntax- and interface-related errors, while deeper semantic issues remain harder to repair.
  • Characteristics of self-debug: The first self-debug round provides the largest improvement, while later rounds yield sharply diminishing returns and typically plateau by round three.Feedback exposes structural and interface errors more effectively than deep semantic, symbolic, or rendering-process inconsistencies.
  • Failures remain across models: Remaining failures include deep semantic inconsistencies, compiler-dependent symbolic errors, and runtime or rendering behaviors with coarse feedback.These failure types make the true source of failure difficult for current self-debug mechanisms to identify.
  • Task/Visual Score Analysis: At 32B, VisCoder2 improves from 49 to 58 in Visual Score and from 56 to 68 in Task Score, with high-quality visual samples rising from 42% to 46%.The corresponding proportion of samples with task scores at least 75 rises from 54% to 62%.
  • Cross-language relationship between Exec Pass and Task/Visual Score: Execution success and task or visual quality generally rise together in Python, Vega-Lite, Mermaid, and HTML, but decouple in LaTeX and SVG.LaTeX can reach 66.1% execution pass after self-debug while task and visual scores remain much lower; SVG execution often saturates while visual scores remain around forty to fifty.
  • Dataset analysis: VisCode-Multi-679K spans 91 visualization types across 15 categories, while measured instruction and code redundancy are 0.21% and 3.03%.These statistics indicate broad coverage and low duplication across the dataset’s visualization samples.
Loading 2510.23642v2…