PaperlayerView exact revision on arXiv

Source-linked AI summary

VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models

Lingjie Jiang, Shaohan Huang, Xun Wu, Yixia Li, Dongdong Zhang, Furu Wei

arXiv:2508.09945v1cs.CLcs.AIcs.CV

TL;DR

Existing multimodal models remain limited in generating functional code from visual inputs. VisCodex merges vision and coding models with task vectors and introduces MCD and InfiBench-V; it achieves state-of-the-art open-source performance and approaches GPT-4o.

  • Problem

    Multimodal models excel at visual understanding but remain limited in generating syntactically and functionally correct code from visual inputs.

    Source: §1 · p. 1

  • Method

    VisCodex merges a vision-language model and coding LLM through task vectors while preserving visual components, alongside MCD and InfiBench-V.

    Source: §1 · pp. 1–2

  • Results

    VisCodex achieves state-of-the-art performance across evaluated multimodal coding benchmarks; VisCodex-33B scores 72.3 on average versus GPT-4o's 73.3.

    Source: §1 · p. 2, §6.1 · p. 7

  • Takeaways & Limitations

    Task vector-based merging provides an efficient path to combining visual comprehension and code generation without full-scale retraining.

    Source: §1 · p. 2, §8 · p. 10

    Rewriting existing Web2Code HTML while retaining its original structure produced rendering artifacts and visually incongruous layouts.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have significantly advanced the integration of visual and textual understanding. However, their ability to generate code from multimodal inputs remains limited. In this work, we introduce VisCodex, a unified framework that seamlessly merges vision and coding language models to empower MLLMs with strong multimodal code generation abilities. Leveraging a task vector-based model merging technique, we integrate a state-of-the-art coding LLM into a strong vision-language backbone, while preserving both visual comprehension and advanced coding skills. To support training and evaluation, we introduce the Multimodal Coding Dataset (MCD), a large-scale and diverse collection of 598k samples, including high-quality HTML code, chart image-code pairs, image-augmented StackOverflow QA, and algorithmic problems. Furthermore, we propose InfiBench-V, a novel and challenging benchmark specifically designed to assess models on visually-rich, real-world programming questions that demand a nuanced understanding of both textual and visual contexts. Extensive experiments show that VisCodex achieves state-of-the-art performance among open-source MLLMs and approaches proprietary models like GPT-4o, highlighting the effectiveness of our model merging strategy and new datasets.

1 Introduction

VisCodex addresses the underexplored challenge of generating functional code from visual inputs by merging vision and coding models, supported by new training data and evaluation resources. Experiments report state-of-the-art open-source multimodal code generation performance and competitiveness with proprietary models.

Motivation

  • Multimodal code generation requires interpreting UI layouts, chart structures, and programming screenshots while producing syntactically and functionally correct code.The paper identifies insufficient programming knowledge in current multimodal models as a central challenge.Source: §1 · p. 1

VisCodex

  • VisCodex merges a vision-language model with a coding LLM using task vectors, while preserving the vision encoder and cross-modal projection modules.The approach integrates code understanding and generation with visual perception without relying on costly pre-training.Source: §1 · pp. 1–2

Resources

  • MCD provides 598k instruction-tuning samples spanning enhanced HTML, chart-to-code pairs, image-augmented StackOverflow QA, and algorithmic coding problems.The dataset is designed to address the lack of large-scale, high-quality multimodal coding data.Source: §1 · p. 2
  • InfiBench-V evaluates visually rich programming questions from real user scenarios where images are indispensable to solving the problem.It is intended as a more realistic and demanding testbed than text-only code QA or simpler visual tasks.Source: §1 · p. 2

Results

  • VisCodex achieves state-of-the-art performance among open-source multimodal code generators and competes with proprietary models such as GPT-4o.The contribution is presented as an efficient path to multimodal coding capability enhancement.Source: §1 · p. 2

2 VisCodex

VisCodex combines a flexible vision-language backbone with coding expertise through task-vector merging, then fine-tunes the language backbone on multimodal coding data while preserving visual components.

2.1 Model Architecture

  • Qwen2.5-VL provides the foundation because its 2D RoPE-based vision architecture supports images with arbitrary resolutions.The architecture dynamically generates visual tokens while preserving absolute scale and spatial relationships.Source: §2.1 · p. 2

2.2 Model Merging

  • Task vectors represent parameter shifts from task-specific fine-tuning and act as transferable units for combining multimodal and coding capabilities.The method computes task vectors from a base model and specialized fine-tuned variants.Source: §2.2 · p. 3
  • VisCodex merges the vision-language and coding task vectors only in the language backbone, leaving the vision encoder and projection modules unchanged.This selective design preserves visual understanding while integrating code comprehension and generation capabilities.Source: §2.2 · pp. 3–4
  • The merging formula balances multimodal and coding expertise with λ, where θVisCodex is initialized from the base model plus weighted task vectors.λ ∈[0, 1] controls the trade-off between retaining multimodal representations and integrating code expertise.Source: §2.2 · pp. 3–4

2.3 Model Training

  • After merging, supervised fine-tuning on MCD updates only the language backbone to align the model with multimodal coding tasks.The vision encoder and projection modules remain frozen during this training stage.Source: §2.3 · p. 4

3 MCD Dataset

MCD is a 598k-sample instruction-tuning dataset combining enhanced HTML, chart image-code pairs, image-augmented StackOverflow QA, and algorithmic code.

3 MCD Dataset

  • MCD contains four complementary data sources designed to support multimodal code understanding, generation, and algorithmic reasoning.Its components cover enhanced HTML, chart image-code pairs, image-augmented StackOverflow QA, and algorithmic code.Source: §3 · p. 4

3.1 Enhanced HTML Code

  • 200,000 newly generated HTML code-image pairs were produced from curated webpage images and filtered for rendering failures and visual artifacts.The pairs were converted into instruction-following format using the Web2Code framework.Source: §3.1 · pp. 4–5

3.2 Chart Image to Code Data

  • 210,000 chart image-code pairs combine 164,000 synthetic Chart2Code samples with 46,000 quality-filtered real-world GitHub examples.Filtering removed non-executable scripts, blank or improperly sized images, and visually corrupted charts.Source: §3.2 · p. 5

3.3 Image-Augmented Code QA

  • The image-augmented StackOverflow component contains 59,000 cleaned and refined QA pairs with Python or HTML code.The pipeline removed invalid or poor-quality entries and used GPT-4o to improve clarity and concision.Source: §3.3 · p. 5

3.4 Algorithmic Code

  • The algorithmic component contains 129,000 instruction-following examples drawn from five established coding categories and datasets.These examples preserve the model’s core reasoning and problem-solving abilities.Source: §3.4 · p. 5

4 InfiBench-V

InfiBench-V evaluates multimodal coding QA using visually indispensable StackOverflow-derived questions, expert curation, paraphrasing, and three complementary scoring methods.

4 InfiBench-V

  • InfiBench-V targets complex programming questions whose correct answers require integrating textual and visual information, unlike text-focused InfiBench [LGL+24].The benchmark is intended to assess multimodal reasoning where visual context is critical.Source: §4 · p. 5

4.1 Data Curation

  • The curation pipeline narrowed approximately 1 million accepted-answer image questions to 10,000 high-relevance questions where images were indispensable.GPT-4o helped filter out questions solvable from text alone.Source: §4.1 · p. 6
  • Experts selected 322 final questions spanning 13 programming languages and five domains: front-end, back-end, DS&ML, mobile and desktop, and ITOps.Each question was mapped to a high-level programming category.Source: §4.1 · p. 6
  • Prompt paraphrasing reduces memorization risk while preserving semantic content, and annotations provide categories, key phrases, metrics, and reference answers.These annotations support robust scoring.Source: §4.1 · p. 6

4.2 Evaluation Criteria

  • Evaluation combines keyword matching, executable unit tests, and GPT-4o judging for natural-language answers.The methods respectively target required terms, code correctness, and correctness plus completeness.Source: §4.2 · p. 6

5 Experimental Setup

The experiments evaluate multimodal coding across UI-to-code, chart-to-code, visually rich algorithmic QA, and InfiBench-V, using task-specific metrics and tuned merge coefficients.

5 Experimental Setup

  • VisCodex is evaluated on four benchmarks covering visual UI-to-code, chart specification generation, visually rich algorithmic problem-solving, and multimodal coding QA.The benchmarks are Design2Code, ChartMimic, MMCode, and InfiBench-V.Source: §5 · p. 6
  • Design2Code reports Low-L feature scores for block, text, position, and color alongside High-L semantic fidelity.This evaluates translation from visual UI designs into executable code.Source: §5 · p. 6
  • ChartMimic uses the Direct Mimic task on test-mini and reports Low-L and GPT-4o-assessed High-L scores.The benchmark evaluates chart specification generation from images.Source: §5 · p. 6
  • MMCode measures visually rich algorithmic problem-solving with pass@1 accuracy, while InfiBench-V reports the average across its evaluation metrics.These benchmarks cover complementary coding abilities in visual contexts.Source: §5 · p. 6
  • For the 8B model, λ = 0.7 was selected from four candidates using MMCode, while the 33B variant used λ = 0.85 without additional tuning.The 8B setting weights the vision-language task vector at 0.7 and the code task vector at 0.3.Source: §5 · p. 7

6 Experimental Results

VisCodex achieves state-of-the-art multimodal code-generation performance, while analyses show that task-vector merging, code-specialized backbones, and MCD contribute to visual grounding and executable coding quality.

6.1 Main Results

  • VisCodex-8B averages 68.8 and VisCodex-33B averages 72.3, outperforming open-source models while approaching GPT-4o at 73.3.VisCodex-8B also surpasses GPT-4o-mini in its size class.Source: §6.1 · p. 7, Table 1 · p. 7
  • Design2Code scores reach 90.1/90.9 for VisCodex-8B and 90.5/91.1 for VisCodex-33B, comparable to or exceeding GPT-4o.The two values represent Low-Level and High-Level performance.Source: §6.1 · p. 7

6.2 Analysis

  • Merging improves the 8B model on Design2Code from 89.6 to 90.1, ChartMimic from 73.4 to 74.8, and MMCode from 6.8 to 11.0.The gains are reported across benchmarks and model scales.Source: §6.2 · p. 7, Table 2 · p. 8
  • Code-specialized LLMs improve benchmark performance over a general-purpose LLM, with Nemotron raising MMCode pass@1 from 6.8 to 11.0.OpenThinker2-7B and Qwen2.5-Coder-7B improve Design2Code and ChartMimic, while Nemotron further boosts MMCode.Source: §6.2 · pp. 7–8, Table 3 · p. 8
  • Model merging outperforms one-stage and two-stage backbone replacement, especially on visually intensive Design2Code and ChartMimic tasks.The paper attributes this to preserving visual alignment while adding code-generation capability.Source: §6.2 · p. 8, Table 4 · p. 8
  • MCD achieves the highest low-level and high-level Design2Code scores versus WebCode2M and Web2Code, particularly on layout-sensitive metrics.The results indicate improved structural alignment, visual grounding, and semantic consistency.Source: §6.2 · pp. 8–9, Table 5 · p. 9

6.3 Case Study

  • VisCodex-8B produces chart reconstructions and HTML outputs closer to ground truth than GPT-4o-mini, InternVL3-78B, and Qwen2.5-VL-7B.The qualitative comparison covers ChartMimic and Design2Code.Source: §6.3 · p. 9, Figure 2 · p. 9

7 Related Work

Prior MLLM research established strong visual-language reasoning and several multimodal coding benchmarks, while model merging emerged as a capability-integration strategy. The paper positions VisCodex as a unified solution for multimodal code generation.

7.1 Multimodal Large Language Models

  • MLLMs integrate vision components with LLMs to support text-image interaction, VQA, chart understanding, and multimodal reasoning.Representative architectures use Q-Formers or MLP projectors to map visual features into language-model space.Source: §7.1 · pp. 9–10

7.2 Multimodal Code Generation

  • Existing multimodal coding work spans webpage-to-code, visually grounded algorithmic coding, and chart-generation benchmarks, but lacks a complete unified solution.The surveyed resources include Design2Code, WebSight, Pix2Code, Web2Code, Webcode2M, MMCode, Human-V, ChartMimic, Plot2Code, ChartCoder, and CodeV.Source: §7.2 · p. 10

7.3 Model Merging for MLLMs

  • Model merging combines capabilities in parameter space through weighted averaging and more advanced strategies, including prior applications to multimodal reasoning and preference transfer.The paper extends this line by using merging for multimodal code understanding and generation.Source: §7.3 · p. 10

8 Conclusion

The paper concludes that task-vector model merging provides an efficient route to strong multimodal code generation, supported by MCD and InfiBench-V. Additional experiments show transfer across model architectures and generality beyond the Qwen2.5 backbone.

8 Conclusion

  • VisCodex combines visual comprehension and code generation through task-vector merging without full-scale retraining, achieving state-of-the-art open-source performance competitive with GPT-4o.The paper also introduces MCD with 598k examples and InfiBench-V for realistic multimodal coding evaluation.Source: §8 · p. 10

A.1 Generalizability of MCD

  • Fine-tuning InternVL3-8B and llava-llama3.1-8b on MCD consistently improves Design2Code, ChartMimic, and MMCode performance.The results support MCD’s transferability across different model architectures.Source: §A.1 · p. 16, Table 6 · p. 16

A.2 Generality of Code Model Merging

  • Merging llava-llama3.1-8b with a DeepSeek-R1-Distill-Llama-8B code task vector raises MMCode pass@1 from 4.2 to 6.5 and outperforms the non-merged baseline across tasks.These results extend the reported merging behavior beyond the Qwen2.5 backbone.Source: §A.2 · p. 16, Table 7 · p. 16

B Training Parameters and Training Cost

Training uses MCD with AdamW, warm-up and cosine decay, and requires approximately 16 hours for the 8B model and two days for the 33B model on multi-node A100 hardware.

B Training Parameters and Training Cost

  • Training uses 598K MCD examples, AdamW, 10% linear warm-up, cosine decay, a 1 × 10^-5 maximum learning rate, batch size 128, and an 8K-token sequence limit.The 8B model trains for two epochs in approximately 16 hours on 8 nodes with 8×A100 40 GB GPUs; the 33B model takes approximately two days.Source: B Training Parameters and Training Cost · p. 16

C Human Evaluation

The human-evaluation study compares anonymized outputs from five MLLMs on 100 InfiBench-V questions against expert judgments, while automatic scores are binarized for agreement analysis. InfiBench-V agrees with human annotators more often than the annotators agree with one another.

C Human Evaluation

  • Five MLLMs produced 500 outputs from 100 randomly sampled InfiBench-V questions for the human-alignment study.Source: C Human Evaluation · p. 17
  • Three computer-science graduate students independently judged whether each anonymized model answer correctly solved the question, using the accepted Stack Overflow answer as reference.Source: C Human Evaluation · p. 17
  • Annotators were trained and calibrated before evaluation through practice, criteria review, feedback, and a readiness test.Source: C Human Evaluation · p. 17
  • Automatic response scores range from 0–100 and are thresholded into pass/fail labels for direct comparison with binary human judgments.Source: C Human Evaluation · p. 17
  • 86.07% agreement was achieved between InfiBench-V and human annotators, versus 79.33% agreement among annotators.Source: C Human Evaluation · p. 17, Table 8 · p. 17

D.1 Data Statistics of MCD

The supplementary materials document MCD and InfiBench-V dataset statistics, the automatic-evaluation prompt, and representative MMCode and InfiBench-V examples. Together, they specify benchmark coverage, scoring criteria, and the kinds of multimodal programming problems evaluated.

D.1 Data Statistics of MCD

  • 598k deduplicated MCD examples span HTML, Chart, QA, and Algorithm domains after SimHash filtering against evaluation benchmarks.Source: §D.1 · p. 17, Table 9 · p. 18

D.2 Distribution of selected benchmark questions in InfiBench-V

  • InfiBench-V contains 322 questions across Front-End, Back-End, DS & ML, IT Ops, and Mobile & Desktop categories in 13 programming languages.Source: §D.2 · p. 18, Table 10 · p. 18

Prompt Template for GPT-4o Automatic Evaluation on InfiBench-V

F.1 MMCode Case 1

  • An MMCode example asks models to minimize bit changes needed to eliminate the substring "010" from a binary string.Source: §F.1 · p. 20, Figure 3 · p. 20

F.2 MMCode Case 2

  • A second MMCode example requires ordering translations of nonintersecting sticks off a square table under shortest-path and non-contact constraints.Source: §F.2 · p. 21, Figure 4 · p. 21

F.3–F.7 InfiBench-V Cases

Related papers

Loading 2508.09945v1…