PaperlayerView exact revision on arXiv

Source-linked AI summary

JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence

Qiushi Sun, Jingyang Gong, Yang Liu, Qiaosheng Chen, Lei Li, Kai Chen, Qipeng Guo, Ben Kao, Fei Yuan

arXiv:2510.23538v3cs.AIcs.CLcs.CVcs.SE

TL;DR

Neural code intelligence needs to extend beyond text to visual program outputs, but progress is constrained by scarce, difficult-to-validate multimodal data. The paper introduces a synthesis toolkit and JanusCode-800K, then trains unified JanusCoder models that perform strongly across text- and vision-centric coding tasks, approaching or exceeding leading commercial models.

  • Problem

    Multimodal code intelligence lacks high-quality, diverse data and remains limited by underdeveloped visual-code modeling across varied tasks.

    Source: §1 · pp. 1–2

  • Method

    The paper combines a multimodal data-synthesis toolkit, execution and reward-model quality control, JanusCode-800K, and unified JanusCoder models.

    Source: §1 · p. 2, §3.2 · p. 5, §3.4 · p. 5

  • Results

    JanusCoder models show superior performance across text-centric and vision-centric benchmarks, with 7B–14B models approaching or exceeding leading commercial models.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    JanusCoder provides an open-source foundational visual-programmatic interface for multimodal code intelligence across diverse visual-code tasks.

    Source: §1 · p. 2, §7 · p. 10

Abstract

from arXiv · show

The scope of neural code intelligence is rapidly expanding beyond text-based source code to encompass the rich visual outputs that programs generate. This visual dimension is critical for advanced applications like flexible content generation and precise, program-driven editing of visualizations. However, progress has been impeded by the scarcity of high-quality multimodal code data, a bottleneck stemming from challenges in synthesis and quality assessment. To address these challenges, we make contributions from both a data and modeling perspective. We first introduce a complete synthesis toolkit that leverages reciprocal synergies between data modalities to efficiently produce a large-scale, high-quality corpus spanning from standard charts to complex interactive web UIs and code-driven animations. Leveraging this toolkit, we construct JanusCode-800K, the largest multimodal code corpus to date. This powers the training of our models, JanusCoder and JanusCoderV, which establish a visual-programmatic interface for generating code from textual instructions, visual inputs, or a combination of both. Our unified model is a departure from existing approaches that build specialized models for isolated tasks. Extensive experiments on both text-centric and vision-centric coding tasks demonstrate the superior performance of the JanusCoder series, with our 7B to 14B scale models approaching or even exceeding the performance of commercial models. Furthermore, extensive analysis provides key insights into harmonizing programmatic logic with its visual expression. Our code and checkpoints are available at https://github.com/InternLM/JanusCoder.

1 INTRODUCTION

Code intelligence is expanding from textual source code toward visual program outputs, but multimodal modeling and high-quality data remain underdeveloped. JanusCoder addresses these gaps with a synthesis toolkit, JanusCode-800K, and unified models that perform strongly across text- and vision-centric tasks.

Motivation

  • Visual program outputs are becoming a central frontier for code intelligence, motivating interfaces that harmonize code logic with visual expression.This expansion targets visualizations, interactive front-ends, and other program-generated content.Source: §1 · p. 1

Challenges

  • Existing multimodal code systems remain fragmented: fine-grained perception and generation are underdeveloped, while specialized models target isolated scenarios.Such specialization limits generalization across chart-to-code, WebUI-to-code, and related tasks.Source: §1 · pp. 1–2
  • High-quality multimodal code data is scarce because corpora must cover diverse languages, instructions, visual outputs, validation environments, and quality-control requirements.The target range includes static charts, interactive WebUIs, and code-driven animations.Source: §1 · p. 2

Contributions

  • The authors release a versatile synthesis toolkit that automatically creates multimodal code data across heterogeneous domains and programming languages, reducing curation effort.Supported domains include charts, Web UIs, visual artifacts, and code-driven animations.Source: §1 · p. 2
  • JanusCode-800K is introduced as the largest multimodal code intelligence corpus to date, including large-scale animation and artifact data absent from previous works.The corpus is the data foundation for the JanusCoder models.Source: §1 · p. 2
  • JanusCoder and JanusCoderV provide a unified interface for a broad spectrum of visual–programmatic tasks, with results approaching or exceeding leading commercial models.The evaluation covers seven established and newly proposed benchmarks in text-centric and vision-centric settings.Source: §1 · p. 2

2 RELATED WORKS

Related work spans text-centric generation of visual interfaces and vision-centric production or interpretation of symbolic code. Representative tasks include visualization and web-interface generation, chart understanding, and chart-to-code reconstruction.

Code Generation for Visual Interfaces

  • Text-centric research generates visual interfaces from language, covering data visualizations, web pages, interactive UIs, charts, and web-based artifacts.Earlier efforts emphasized Python visualization libraries before expanding toward chart editing and structured interfaces.Source: §2 · p. 3

Visually-Grounded Code Generation and Understanding

  • Vision-centric research interprets visual inputs to produce or reason about symbolic code, including chart understanding and chart-to-code generation.Chart-to-code benchmarks require reproducing scientific plots from images with captions or instructions.Source: §2 · p. 3

3 METHOD

The method combines heterogeneous data sourcing, multi-strategy synthesis, execution-based validation, and reward-model filtering to build JanusCode-800K. Cross-domain and cross-modal synergies expand coverage while retaining functional and visual-quality checks.

Toolkit Overview

  • The toolkit sources heterogeneous data, synthesizes instruction–code and vision-centric examples through multiple strategies, and applies reward modeling after execution checks.The workflow is organized around data sourcing, synthesis and curation, validation, and quality control.Source: §3 · p. 3, Figure 2 · p. 3

3.1 DATA COLLECTION

  • Long-form code-only files are decomposed with Abstract Syntax Trees into semantically coherent, self-contained units for learning.This addresses complex scripts such as multi-minute Manim animations.Source: §3.1 · p. 4
  • The synthesis process builds text-centric instruction–code pairs and vision-centric instruction–code–visual-output triplets using guided evolution, reverse instruction, and bidirectional translation.Bidirectional translation transfers conceptual intent between semantically analogous domains and uses source code as a structural template.Source: §3.2 · pp. 4–5

3.2 DATA CURATION

  • Generated code is executed in a sandbox with interpreters and renderers; failed samples are sent back for retry and refinement, allowing only functionally valid samples to proceed.Execution produces visual outputs or evaluates collected or generated test cases.Source: §3.2 · p. 5

3.3 DATA SYNERGY

  • The pipeline transfers knowledge across related domains and modalities, using outputs from one task to construct data for another and thereby broadening dataset coverage.Examples include transferring scientific-computing logic across languages and using Python visualizations to create chart-to-code data.Source: §3.3 · p. 5

3.4 DATA QUALITY CONTROL

  • Because executability alone cannot ensure visual alignment, a VLM or LLM scores instruction–code–visual-output or instruction–code samples across quality dimensions and filters low-scoring data.Visual samples receive averaged 1–5 metric scores and are retained only above a predefined threshold.Source: §3.4 · p. 5

3.5 JANUSCODE-800K

  • The toolkit produces JanusCode-800K, a diverse multimodal corpus whose text-centric and vision-centric data are balanced; JanusCoderV uses the full corpus while JanusCoder uses text-centric data.Detailed corpus statistics are reported in Table 2.Source: §3.5 · pp. 5–6, Table 2 · p. 6

4 DTVBENCH

DTVBENCH evaluates code generation for dynamic theorem visualizations across explanatory animations and interactive symbolic visualizations. Its multidimensional scoring rewards executable outputs that are structurally faithful, instruction-aligned, and perceptually faithful.

  • DTVBENCH combines MANIM and Wolfram Mathematica to test dynamic, logically coherent, visually meaningful code generation from natural-language instructions.Source: §4 · p. 6
  • The benchmark uses raw samples from 3Blue1Brown video segments and official Wolfram demonstrations, with 102 visualization tasks manually curated.Source: §4.1 · p. 6
  • Outputs are evaluated for executability, code similarity, instruction alignment, and faithfulness to the intended visual content.Source: §4.2 · p. 6
  • The overall score combines execution success with code similarity, instruction alignment, and faithfulness, so only executable code receives further quality evaluation.Source: §4.2 · p. 6

5 EXPERIMENTS

The experiments evaluate JanusCoder across broad unimodal, multimodal, and general coding benchmarks against open and commercial baselines. Across visual coding tasks, the models show strong performance, often matching or surpassing GPT-4o and specialized systems.

Experimental setup

  • Training uses Qwen3 backbones for JanusCoder and Qwen2.5-VL plus InternVL backbones for JanusCoderV, with comparisons against open-weight models and GPT-4o.Source: §5.1 · p. 7

Main results

  • JanusCoder models achieve strong results across seven unimodal and multimodal benchmarks, with 8B and 14B variants exceeding baselines and often matching or surpassing GPT-4o.On Python visualization tasks, error rates are below 10%, with comparable or superior task completion and visual similarity to GPT-4o.Source: §5.2 · p. 7, Table 3 · p. 7, §5.3 · p. 8

Evaluation scope

  • The evaluation includes text-to-code visualization, artifact, and animation benchmarks; chart-to-code, webpage, and scientific demonstration tasks; and BigCodeBench and LiveCodeBench for general coding.Source: §5.2 · p. 7

Unimodal tasks

  • JanusCoder performs strongly on ArtifactsBench and DTVBENCH, exceeding GPT-4o on interactive visual artifacts and approaching it on dynamic-content generation.Source: §5.2 · p. 7, Table 3 · p. 7, §5.3 · p. 8

Multimodal tasks

  • On ChartMimic, JanusCoderV consistently outperforms baselines, GPT-4o, and specialized chart-to-code multimodal models across high- and low-level metrics.Source: §5.2 · p. 7, §5.4 · p. 8, Table 4 · p. 8
  • On WebCode2M and DesignBench, the models significantly improve both webpage visual quality and structural similarity between generated and reference code.Source: §5.2 · p. 7, §5.4 · p. 8, Table 4 · p. 8

6 ANALYSIS

The analysis tests whether cross-domain data, reward modeling, and the unified design improve robustness and generality. Results support transferable benefits from heterogeneous data and show balanced performance across general and visualization coding.

6.1 Ablation studies

  • Data from non-target domains and modalities improves specialized visual tasks, especially data-scarce areas such as animations and visual artifacts.The ablations indicate that text-centric data can contribute to multimodal coding scenarios through transferable coding capabilities.Source: §6.1 · p. 9, Table 5 · p. 9
  • Removing reward-model filtering causes a clear performance drop at fixed training-set size, showing that execution success alone does not ensure high-quality multimodal data.Source: §6.1 · p. 9

6.2 Data effectiveness

  • JanusCode-800K yields significant improvements across backbones with different scales and post-training strategies, supporting the robustness of the data design.Source: §6.2 · p. 9, Figure 4 · p. 9

6.3 Generalist capability

  • JanusCoder combines strong general coding performance with performance exceeding specialist VisCoder on its target visualization domain and GPT-4o in both scenarios.Source: §6.3 · pp. 9–10, Figure 5 · p. 10

7 CONCLUSION

JanusCoder establishes a unified visual-programmatic interface supported by scalable multimodal data synthesis. Across representative benchmarks, its 7B–14B models approach or exceed leading commercial models while offering a basis for future multimodal code intelligence.

  • JanusCoder combines a scalable synthesis toolkit with unified handling of diverse visual code tasks, including a newly proposed benchmark.Source: §7 · p. 10
  • The 7B–14B JanusCoder models approach or exceed leading commercial capabilities, while analysis identifies principles for harmonizing programmatic logic with visual expression.Source: §7 · p. 10

A DATA TOOLKIT DETAILS

The toolkit processes complex Manim animations through static AST analysis, extracting structured semantic features without execution. It also represents web-page edits as canonicalized meta tasks that can be instantiated into concrete instructions and code changes.

Manim animation processing

  • AST-based processing parses Manim scripts without execution, avoiding environment-specific dependencies and improving reproducibility.Each script is converted into an Abstract Syntax Tree before downstream extraction.Source: §A.1 · p. 16
  • Scene identification locates classes inheriting from canonical Manim bases and analyzes their construct() methods as the primary animation logic.Detected bases include Scene and ThreeDScene.Source: §A.1 · p. 16
  • Feature extraction records animation objects, invoked animations, textual content, imports, and concise code excerpts while filtering project-specific dependencies.Examples include Circle, Text, Create, and Write.Source: §A.1 · p. 16
  • The extracted animation elements are consolidated into JSONL entries containing file identifiers, scene classes, features, and prompt templates.This representation preserves semantic intent for the data toolkit.Source: §A.1 · p. 16

Web-page edit representation

  • A meta task is an abstract, canonicalized web-page edit operation that captures user intent independently of context, location, or wording.Concrete instructions can then ground the operation to DOM elements and code edits.Source: §A.2 · p. 16

B DATA COLLECTION DETAILS

The toolkit draws on documented data sources and applies automated reward filtering to construct the JANUSCODE-800K corpus. The supplied passages identify the source inventory through Table 6 but do not enumerate its entries.

Data sources

Scope of supplied details

Quality filtering

  • An automated reward mechanism filters generated code samples and their multimodal outputs to improve synthetic-data quality.The mechanism is described as systematic filtering before corpus construction.Source: C REWARD MODELING DETAILS · p. 17

C.1 REWARDING TEXT-CENTRIC DATA

The reward pipeline evaluates text-centric samples with an LLM and vision-centric samples with a vision-language model, using stricter multimodal checks for visual alignment and editing validity. Filtered samples form JANUSCODE-800K, while the supplied benchmark passages mainly identify evaluation tables and DTVBENCH composition.

Rewarding text-centric data

  • Text-centric samples are scored by Qwen-2.5-72B-Instruct for instruction alignment, logical coherence, and code robustness, retaining only samples rated 5.The scoring follows criteria in the Reward Prompt - Artifacts.Source: §C.1 · p. 17

Rewarding vision-centric data

  • Vision-centric samples are assessed by Qwen2.5-VL-72B-Instruct for alignment among instructions, code, and rendered visual outputs.The data includes visualizations, WebUI layouts, and animations.Source: §C.2 · p. 17
  • The vision-centric reward uses a 5-point standard with multimodal alignment checks and binary filtering for visual editing tasks.Editing samples must meet minimum thresholds across key dimensions, including a threshold of ≥3 in the supplied description.Source: §C.2 · p. 17

Corpus construction

  • Reward filtering converts the raw synthetic pool into the final JANUSCODE-800K corpus.The pipeline is designed to remove misaligned or low-quality generated data at scale.Source: §C.2 · p. 17

Benchmark data

  • DTVBENCH contains 52 Manim animation tasks and 50 Wolfram tasks collected from open-source demonstrations and scripts.The supplied passage also notes an optional subjective evaluation with college-level student annotators.Source: D DTVBENCH DETAILS · pp. 17–18

E TRAINING DETAILS

Training uses standardized optimization and distributed-computing settings across model scales, while evaluation spans plotting, chart mimicry, webpage generation, and editing. On DesignBench, JanusCode-7B leads open-weight generation and remains strong at editing, whereas proprietary models lead their respective tasks.

Training configuration

  • Training uses bfloat16, a 1 × 10−5 learning rate, three epochs, and distributed acceleration through FlashAttention-2, Liger-Kernel, and DeepSpeed.These settings are shared across the reported training configurations.Source: E TRAINING DETAILS · p. 18
  • The 4B, 7B, and 8B models use 8 H800 GPUs with ZeRO-2, while 14B models use 16 H800 GPUs with ZeRO-3; total batch size remains 128.Per-device batch sizes are 2 for smaller models and 1 for 14B models, with gradient accumulation of 8.Source: E TRAINING DETAILS · p. 18

Benchmark coverage

  • The evaluation reports complete results for PandasPlotBench and ChartMimic, including direct and customized mimic settings.The corresponding results are provided in Tables 7–9.Source: §F.1 · p. 18, §F.2 · p. 18, Table 8 · p. 19

F.3 DETAILED RESULTS ON DESIGNBENCH

  • DesignBench separates screenshot-to-code generation from screenshot-and-source-guided editing, directly testing the visual–programmatic linkage.Table 10 compares proprietary and open-weight models on both tasks.Source: §F.3 · p. 20, Table 10 · p. 20

Evaluation metrics

  • Evaluation uses CLIP similarity, GPT-4o’s 0–10 MLLM Score, and Code Match Score to measure visual alignment, judged edit quality, and modified-line overlap.The MLLM Score was validated against human evaluation in the original work.Source: §F.3 · p. 20

F.3 DETAILED RESULTS ON DESIGNBENCH

  • Claude-3.7-sonnet has the strongest proprietary generation capability, while GPT-4o achieves the highest proprietary editing MLLM Score.Both maintain competitive CMS, indicating robust editing quality.Source: §F.3 · p. 20, Table 10 · p. 20
  • JanusCode-7B ranks first among open-weight models in DesignBench code generation and also delivers strong editing results.InternVL3.5-8B instead records the highest CMS among open-weight models for fine-grained code modifications.Source: §F.3 · p. 20, Table 10 · p. 20

F.4 DETAILED RESULTS ON WEBCODE2M

On WebCode2M, JanusCoder combines strong visual alignment with superior structural preservation, while broader evaluations show competitive performance across interactive science and artifact-generation tasks. Cross-model adjudication further reports robust agreement across frontier evaluators.

WebCode2M

  • The WebCode2M evaluation measures webpage appearance with Visual and DOM-level structural correctness with TreeBLEU, based on matched 1-height subtrees.TreeBLEU is defined as the fraction of candidate-tree 1-height subtrees matched in the reference tree.Source: §F.4 · pp. 20–21
  • JanusCoder-7B and JanusCoder-8B achieve higher TreeBLEU than all proprietary counterparts, establishing state-of-the-art structural preservation on WebCode2M.Proprietary models generally lead in appearance-level alignment, but JanusCoder prioritizes DOM-level structure without severely sacrificing visual similarity.Source: §F.4 · p. 21, Table 11 · p. 21

INTERACTSCIENCE

  • JANUSCODERV-7B and JANUSCODERV-8B outperform InternVL3.5 and Llama-3.2-11B across nearly all INTERACTSCIENCE metrics, including a +5–10 point VLM-Judge gain.The gains cover both programmatic correctness in PFT and visual alignment in VQT.Source: §F.5 · pp. 21–22, Table 12 · p. 22

ArtifactsBench

  • JanusCoder-14B achieves the highest ArtifactsBench average score, 41.10, and leads WEB at 44.47, SI at 41.49, and MS at 45.04.Its SVG-generation performance is relatively lower, despite strong results across practical application domains.Source: §F.6 · p. 22, Table 13 · p. 22

Cross-Model Adjudication

  • Cross-model adjudication of 30K Python and WebUI samples using Qwen2.5-VL-72B-Instruct, Gemini-2.5-Pro, and GPT-5 yields Krippendorff’s α above 0.70 across both domains.The study reports strong interannotator agreement and supports the robustness of the reward scores as data-quality indicators.Source: §G.3 · p. 23, Table 14 · p. 23

G.4 HUMAN EVALUATION ON SUBJECTIVE TASKS

Human evaluation assesses generated Manim animations and interactive WebUIs for visual fidelity and instruction alignment, complementing automated evaluation. The JANUSCODER series consistently outperforms baseline models on these subjective tasks.

Human Evaluation

  • Three annotators rated 100 generated Manim animations and InteractScience WebUIs on a 1–5 scale for visual fidelity and instruction alignment.The human study targets subjective visual quality that automated metrics may not fully capture.Source: §G.4 · p. 23
  • JANUSCODER consistently outperforms baseline models in human evaluation, with results correlating strongly with automated VLM-based assessments.The evaluation supports its ability to generate perceptually high-quality visual content.Source: §G.4 · p. 23, Table 15 · p. 23

Illustrative Example

  • The illustrative edit example receives a final binary outcome of 0 because the edit spills onto an unrelated white fence.Instruction Adherence scores 4, while Edit Quality & Realism and Preservation of Unrelated Areas each score 2.Source: OUTPUT SPECIFICATION · p. 28
  • The illustrative visualization example scores 4.0 overall despite missing the explicitly requested chart title.Task Completion scores 3, while Solution Coherence & Code Quality, Visual Clarity, and Task Relevance score 4, 4, and 5.Source: OUTPUT SPECIFICATION · p. 26, ILLUSTRATIVE EXAMPLE · p. 31

Related papers

Loading 2510.23538v3…