PaperlayerView exact revision on arXiv

Source-linked AI summary

VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning

Xuanle Zhao, Deyang Jiang, Zhixiong Zeng, Lei Chen, Haoyue Yang, Haibo Qiu, Jing Huang, Yufeng Zhong, Liming Zheng, Yilin Cao, Lin Ma

arXiv:2511.00391v3cs.CV

TL;DR

Multimodal code-generation models remain limited by task-specific training and brittle textual rewards that struggle with syntactically diverse but semantically equivalent code. VinciCoder combines large-scale SFT, visual refinement, and coarse-to-fine ViRL, and experiments report superior performance across diverse benchmarks with validated visual alignment. Its limitations include missing domain knowledge, substantial compute requirements, and static-image rewards.

  • Problem

    Task-specific multimodal models limit generalization, while SFT and brittle textual rewards do not reliably ensure code executability and visual-semantic alignment.

    Source: §1 · p. 1, §3.3.2 · p. 5

  • Method

    VinciCoder combines 1.3M direct-generation and 300k visual-refinement SFT samples with coarse-to-fine ViRL using visual and language-alignment rewards.

    Source: §1 · p. 2, §3.3.2 · p. 5

  • Results

    VinciCoder significantly outperforms existing open-source models across extensive benchmarks, and ablations validate ViRL’s effectiveness for high-fidelity visual alignment.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Visual refinement and implementation-agnostic visual rewards provide a unified route to multimodal code generation with stronger visual alignment and code performance.

    Source: §1 · p. 2, §5 · p. 9

    Visual rewards cannot supply missing domain knowledge, require substantial computation, and currently rely on static images rather than interaction.

    Source: §3.3.1 · p. 4, Limitations · p. 9

Abstract

from arXiv · show

While recent specialized multimodal code generation models excel in tasks like chart-to-code generation, their reliance on single-task training limits generalization and hinders the development of \textbf{VI}sio\textbf{N} \textbf{C}ode \textbf{I}ntelligence. In this work, we introduce \textbf{VinciCoder}, a unified framework designed for generalized multimodal code generation. We first curate a large-scale SFT corpus comprising 1.3M direct generation pairs and 300k visual-based refinement tasks. This composition fosters self-refinement capabilities, enabling the model to directly rectify code to align with input images. Subsequently, we propose coarse-to-fine Visual Reinforcement Learning (ViRL) to overcome the brittleness of textual metrics in handling semantically equivalent but syntactically diverse code. By quantifying visual similarity across multi-scale patches, ViRL provides an implementation-agnostic reward mechanism that ensures high-fidelity alignment between rendered outputs and input visuals. Extensive experimental results across diverse benchmarks demonstrate that VinciCoder achieves superior performance, while comprehensive ablation studies validate the effectiveness of our proposed ViRL strategy. The data, code and model are available at https://github.com/DocTron-hub/VinciCoder.

1 Introduction

VinciCoder addresses limited generalization and brittle textual rewards in multimodal code generation with unified SFT and coarse-to-fine visual reinforcement learning. Across diverse benchmarks, the resulting framework achieves superior performance, while refinement training supports visual code correction.

1 Introduction

  • Specialized VLMs use narrow task scopes, while textual rewards are brittle because semantically equivalent code can differ syntactically.These limitations motivate a unified framework with implementation-agnostic visual rewards.Source: §1 · p. 1
  • VinciCoder unifies multimodal code generation through large-scale SFT and coarse-to-fine ViRL, targeting generalization, executability, and visual fidelity.Its corpus contains 1.3M direct-generation samples and 300k visual-based refinement samples.Source: §1 · pp. 1–2
  • Visual refinement training enables VinciCoder to rectify flawed code so rendered outputs align with target images, supporting iterative debugging and self-refinement.The refinement task includes logical errors or partial renderings and provides visual feedback for correction.Source: §1 · p. 2
  • ViRL compares rendered and target images at coarse and fine scales, capturing global layouts from downsampled views and local details from segmented patches.This visual-domain reward is designed to remain indifferent to specific coding implementations.Source: §1 · p. 2
  • Across diverse multimodal benchmarks, VinciCoder achieves superior performance, and ablations show improvements in code executability and visual fidelity.The reported experiments evaluate the unified model against contemporary alternatives and validate the proposed strategy.Source: §1 · p. 2

2 Related Works

Prior multimodal code-generation research is organized around specialized visual domains, while newer unified models remain primarily SFT-based. Related reinforcement-learning work broadens visual reasoning and explores visual feedback, but VinciCoder’s data pipeline supports unified code-generation training.

2.1 MLLMs for Code Generation

  • Task-specific models target charts, webpages, SVGs, or scientific plots, but their homogeneous visual patterns and single-language scopes limit generality.Unified efforts exist, yet they rely almost exclusively on SFT, which is insufficient for jointly ensuring executability and visual fidelity.Source: §2.1 · p. 2

2.2 Reinforcement Learning for MLLM

  • Recent MLLM reinforcement learning applies GRPO-based policy refinement to grounding, question answering, segmentation, and other visual-language tasks.These approaches commonly use SFT reasoning data before RL optimization.Source: §2.2 · pp. 2–3
  • VinciCoder’s data pipeline filters and diversity-samples open-source data, then separately refines existing samples and generates visual-refinement samples for SFT and RL.The two streams produce training pairs for both stages.Source: Figure 2 · p. 3
  • Visual-feedback RL methods score generated images with MLLMs or pretrained ViTs, extending reward construction beyond generated text.RRVF and RLRF exemplify this direction.Source: §2.2 · p. 3

3 Method

VinciCoder unifies multimodal code generation through direct and visual-refinement tasks, followed by SFT and coarse-to-fine ViRL. The method combines executable-data curation, instruction alignment, and multi-scale visual rewards across diverse code-generation domains.

3.1 Task Definition

  • The task definition covers direct generation from images and text plus refinement of potentially flawed code drafts into corrected versions.The refinement task targets visual alignment between the revised code output and the input image.Source: §3.1 · p. 3

3.2 Data Construction

  • The data pipeline curates and re-executes multimodal code examples across charts, webpages, SVGs, LaTeX, scientific plots, and domain-specific visualizations.Scientific plots include molecular structures, electronic schematics, diagrams, document layouts, and tabular data using general-purpose languages and DSLs.Source: §3.2 · p. 3, §3.2.1 · pp. 3–4

3.2.1 SFT Data

  • The SFT corpus combines direct-generation data with refinement examples that train multi-turn error correction from imperfect code and target images.The dataset includes dedicated chart, web-to-HTML, image-to-SVG, image-to-LaTeX, and scientific-plot collections.Source: §3.2.1 · pp. 3–4, Table 1 · p. 4

3.2.2 RL Data

  • The RL dataset is mutually exclusive from SFT data and spans chart-to-code, web-to-HTML, image-to-SVG, image-to-LaTeX, and chemical-image examples.The curated examples include hard cases and filtered or synthesized data for improving visual reconstruction.Source: §3.2.2 · p. 4, Table 2 · p. 4

3.3.1 Supervised Finetuning (SFT)

  • SFT uses an autoregressive objective to establish syntactic priors, while direct ViRL without this initialization can reward visual similarity over code integrity in niche tasks.This initialization is motivated by the need to preserve functional correctness during visual optimization.Source: §3.3.1 · pp. 4–5

3.3.2 Visual Reinforcement Learning (ViRL)

  • ViRL optimizes visual fidelity and instruction-following consistency with GRPO using weighted coarse-to-fine visual and language-alignment rewards.The visual reward averages DINOv2 similarity over a global thumbnail and non-overlapping 448 × 448 patches, assigns zero to failed renders, and the alignment reward checks the requested programming language against generated code identifiers.Source: §3.3.2 · pp. 5–6, Figure 3 · p. 5

4 Experiments

Experiments evaluate VinciCoder across multimodal code-generation benchmarks and show strong open-source performance, with SFT, ViRL, refinement data, and coarse-to-fine rewards contributing complementary gains.

4.2 Evaluation Settings

  • Evaluation covers charts, webpages, SVG, LaTeX, and molecules using benchmark-specific metrics, including execution rate and Tanimoto similarity for ChemDraw.Source: §4.2 · p. 6

4.3 Main Results

  • VinciCoder establishes a new state-of-the-art among open-source solutions across most benchmarks, leading comparable-scale models and JanusCoderV.Source: §4.3 · p. 6
  • The two-stage SFT-RL strategy performs best overall: SFT supplies a strong policy, while ViRL further improves visual similarity and execution rate.RL assigns zero reward to invalid outputs, while ViRL optimizes perceptual alignment between rendered and target images.Source: §4.3 · pp. 6–7, Table 5 · p. 7
  • In multi-turn refinement, VinciCoder fixes 18.2% of non-executable ChartMimic code and improves visual scores for already executable outputs.The refinement data support correction of both functional errors and subtle visual details.Source: §4.3 · p. 7, Table 4 · p. 7

4.4 Ablation Studies

  • SFT performance scales with dataset size before plateauing, indicating that further supervised-learning gains become marginal after the large-scale foundation is established.Source: §4.4 · p. 7, Figure 5 · p. 7
  • ViRL produces substantial gains even from a 20% SFT checkpoint, with its relative improvements strongest before the base model reaches its SFT plateau.Source: §4.4 · pp. 7–8, Table 6 · p. 8
  • The coarse-to-fine reward achieves the most balanced performance; both components matter on high-resolution benchmarks, while UniSVG remains stable because its lower resolution permits global comparison.DINOv2 captures fine-grained features, whereas image-text-aligned ViTs prioritize global semantic alignment.Source: §4.4 · p. 8, Table 7 · p. 8

5 Conclusion

VinciCoder unifies multimodal code generation through large-scale SFT followed by coarse-to-fine ViRL. Benchmarks and ablations report improved open-source performance and high-fidelity visual alignment.

  • VinciCoder combines 1.6M-sample SFT with coarse-to-fine ViRL in a unified multimodal code-generation framework.Source: §5 · p. 9

Limitations

VinciCoder improves visual quality and code performance but remains limited by missing base-model knowledge, high computational demands, and static-image rewards.

  • Visual feedback cannot supply missing domain knowledge, may produce logically incorrect shortcuts, requires substantial computation, and does not model interactive or moving web content.Source: Limitations · p. 9

Ethic Statement

The paper describes public-data usage, reproducibility practices, benchmark implementation choices, and Earth Mover’s Similarity as a patch-based visual metric.

Ethic Statement

  • Training data are collected from public sources under their original licenses, filtered for personally identifiable information, and the system is intended for research and education with human review for safety-critical software.Source: Ethic Statement · p. 9

A Benchmark Details

  • Benchmark evaluation follows official repository implementations, with repository-based metrics used when published-paper values are inconsistent and dedicated metrics for ChemDraw.Source: A Benchmark Details · p. 12

A.1 Earth Mover’s Similarity (EMS)

  • Earth Mover’s Similarity compares global patch arrangement and local pixel distributions, producing a normalized score from 0 for maximum dissimilarity to 1 for identical images.Source: §A.1 · p. 12
  • EMS emphasizes foreground content through weighted image patches and matches patches using both within-patch dissimilarity and spatial distance.Source: §A.1 · p. 12
  • Tanimoto similarity converts valid SMILES strings into molecular fingerprints and returns zero when either input is invalid.Source: §A.1 · p. 12

A.2 ChemDraw Metrics

ChemDraw evaluation uses execution rate to filter syntactically valid generations and Tanimoto similarity to measure chemical fidelity against ground truth.

  • Execution rate evaluates whether generated code and SMILES strings are chemically valid, serving as the primary syntactic-correctness filter.Source: §A.2 · p. 12
  • Tanimoto similarity measures chemical fidelity by comparing RDKit molecular fingerprints for generated and ground-truth structures.Source: §A.2 · p. 12
  • Tanimoto similarity between fingerprints A and B is |A ∩ B| / |A ∪ B| = c / (a + b − c).Here, a and b count bits set in A and B, while c counts their intersection.Source: §A.2 · pp. 12–13

B.1 Full Results of VinciCoder Variants

VinciCoder’s two-stage training improves multimodal code generation across benchmarks, with refinement data and ViRL contributing to correction, visual fidelity, and execution.

B.1 Full Results of VinciCoder Variants

  • VinciCoder’s SFT-ViRL variants improve performance after SFT across benchmarks, with results reported for both 7B and 8B model scales.The full comparison includes SFT-only and SFT-plus-ViRL checkpoints.Source: §B.1 · p. 13

B.2 Comparison with task-specific models

B.3 Effectiveness of Refinement Data

  • Adding 300k refinement tasks during SFT consistently improves performance across nearly all benchmarks while supporting iterative correction and direct-generation quality.Source: §B.3 · p. 13, Table 11 · p. 14

B.4 ViRL Results

  • Visual similarity of executable rollouts improves during ViRL training, indicating gains in both execution rate and visual fidelity.Unexecutable rollouts receive zero reward, so the plotted progression concerns executable samples.Source: §B.4 · p. 14, Figure 7 · p. 14

B.5 Fail Cases in ViRL

  • Without SFT, ViRL can produce primitive geometric shapes unrelated to chemical structures, whereas 20% SFT plus RL reconstructs chemical bonds and rings.The comparison illustrates the importance of SFT priors for syntax and domain-specific code usage.Source: §B.5 · p. 14, Figure 8 · p. 14

C Related Works about Multimodal Code Generation Benchmarks

Multimodal code-generation benchmarks span specialized visual domains and increasingly include interaction, editing, and complex dynamic artifacts; the section also presents GRPO’s optimization framework.

C Related Works about Multimodal Code Generation Benchmarks

D GRPO Alogrithm

  • GRPO samples multiple responses for each input and assigns rewards before computing group-normalized advantages for optimization.Source: D GRPO Alogrithm · pp. 14–15
  • GRPO optimizes the policy using an objective based on response probability ratios, without requiring a separate critic model.Source: D GRPO Alogrithm · pp. 14–15

E Refinement Task

The refinement task pairs incomplete generated code with concrete implementations, while evaluation prompts specify task-specific output formats and rendering instructions.

E Refinement Task

F Evaluation Prompts

  • Evaluation uses official implementations except for the constructed ChemDraw benchmark, whose prompt and molecule image are visualized in Figure 10.Source: F Evaluation Prompts · p. 15, Figure 10 · p. 16
  • The ChemDraw prompt requests a Python script that visualizes the input molecule, defines SMILES in a specified format, and uses RDKit or Indigo for plotting.Source: F Evaluation Prompts · p. 16

Related papers

Loading 2511.00391v3…