PaperlayerView exact revision on arXiv

Source-linked AI summary

PrismaCoder: Pushing the Boundaries of Open Multimodal Code Intelligence through Decomposed Rubric Synthesis

Qiushi Sun, Jingyang Gong, Yang Liu, Qiaosheng Chen, Zhangyue Yin, Zixian Huang, Hanxu Hu, Jing Xiong, Fangzhi Xu, Haochen Tan, Haoli Bai, Lifeng Shang, Xiaoguang Li, Ben Kao, Qipeng Guo, Lingpeng Kong, Jiahui Gao

arXiv:2610.20538v1cs.CLcs.AIcs.CVcs.LGcs.SE

TL;DR

Open multimodal code intelligence lacks data and evaluation signals that can assess both a program and the visual artifact it renders across task types. PrismaForge synthesizes instruction-specific rubrics for data curation and reinforcement learning, supporting PrismaCoder-2M and the PrismaCoder models. Those models lead open systems and rival or surpass proprietary systems on several reported tasks.

  • Problem

    Public multimodal code data is limited and uneven in quality, while execution checks, code similarity, and task-specific verifiers do not jointly assess rendered correctness across tasks.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    PrismaForge synthesizes instruction-specific rubrics to filter training data and provide structured rewards for reinforcement learning, supporting construction of PrismaCoder-2M.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    PrismaCoder-9B and PrismaCoder-35B lead the open state of the art by a wide margin and match or surpass leading proprietary systems on charts, front-end pages, and semi-structured documents.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Analyses report that the data and pipeline generalize across base models, and that rubric-guided curation outperforms direct-judge selection while rubric scores align closely with human judgments.

    Source: §1 · p. 2, §7 · p. 12

Abstract

from arXiv · show

Multimodal code intelligence connects the symbolic world of programs with the visual world their execution creates. Interest in this capability is burgeoning, as models are increasingly asked to produce charts, webpages, animations, and semi-structured documents. Recently, proprietary models have pushed far ahead, producing interactive artifacts even from a text description or a screenshot alone. Yet their data and training remain closed, and open progress is held back by the quantity and quality of available data and by the pipelines and methods for producing it. Quality is the harder of the two, since a sample must be correct as a program, well built, and above all visually right once rendered. No existing signal covers all three. Execution often returns just a pass or fail, code similarity does not map to the visual output, and task-specific verifiers do not generalize. To push open progress further, we start from the data side with PrismaForge, a hierarchical data pipeline centered on a synthesized rubric that decomposes these factors into checkable criteria. The same rubric serves as both filter and reward: it curates the supervised corpus, then drives reinforcement learning on visual-coding tasks. Leveraging PrismaForge for data synthesis and filtering, we build PrismaCoder-2M, the most diverse open dataset for multimodal code intelligence and the largest to date. On the model side, we train PrismaCoder-9B and PrismaCoder-35B on this data and release both. They lead the open state of the art by a wide margin and rival leading proprietary systems. Our analyses further show that the proposed rubric-based pipeline improves multimodal coding performance across model backbones. We release our code and data at https://qiushisun.github.io/PrismaCoder-Home/.

1. Introduction

Open multimodal code intelligence spans visual inputs and rendered outputs, but public data is limited in scale and quality, and existing curation signals are narrow. PrismaForge uses instruction-specific rubrics to curate broad training data and guide reinforcement learning; the resulting models lead open benchmarks and rival proprietary systems.

1. Introduction

  • Public multimodal code data remains limited in scale and uneven in quality, while single scores offer little diagnostic detail and task-specific verifiers do not generalize.Open efforts have often focused on individual task types, complicating generalization across domains and output formats.Source: §1 · pp. 1–2
  • PrismaForge synthesizes instruction-specific rubrics that combine reusable and task-specific checks, separating code-level from rendered-artifact criteria for curation and reinforcement-learning rewards.The number of criteria is adjusted to estimated task difficulty.Source: §1 · p. 2
  • PrismaCoder-2M covers more than 12 categories and 10 programming languages, combining text-centric and vision-centric data from complementary sources.Candidate programs are executed and rendered, then rubric scores are used to filter or refine examples.Source: §1 · p. 2
  • PrismaCoder-9B and PrismaCoder-35B lead the open state of the art by a wide margin and match or surpass proprietary systems on several visual-coding tasks.Analyses also report generalization across base models and close alignment between rubric scores and human judgments.Source: §1 · pp. 2–3

2. Related Work

Multimodal code tasks require judging both programs and the visual artifacts they produce, but current learning signals do not consistently measure correctness across task types. PrismaForge addresses this with one synthesized rubric per instruction for both data filtering and reinforcement-learning rewards.

2. Related Work

  • Visual-programmatic tasks require correctness beyond syntax, including layout, semantics, interaction, and editability, while reading values and layouts from renders remains a perception bottleneck.The task range includes screenshot-to-code, front-end engineering, chart generation and editing, graphics, and interactive artifacts.Source: §2 · p. 3
  • Execution can establish that code runs, but not that its rendered output is faithful, well composed, or semantically correct.Task-specific visual checks can work individually but do not compose across artifact types.Source: §2 · p. 3
  • A single synthesized rubric per instruction serves both training-data filtering and reinforcement-learning rewards across artifact formats.The paper positions this as finer-grained and more general than existing rubric and reward signals.Source: §2 · p. 3, Figure 2 · p. 4

3. PrismaForge

PrismaForge is a rubric-based scoring framework for data curation and training. Its elastic rubric scorer combines reusable domain checks with criteria tailored to each instruction.

3.1. Elastic Rubric Scorer

  • A holistic score is hard to verify and reveals little about why a visualization candidate succeeds or fails.Fixed rubrics can miss instruction-specific priorities, whereas manually creating a rubric for every instruction is costly and difficult to scale.Source: §3.1 · p. 4
  • The elastic rubric combines reusable domain-level criteria with criteria synthesized specifically for each instruction.The number and focus of criteria can adapt to task difficulty and the instruction's priorities.Source: §3.1 · p. 4
  • Each candidate sample is represented by an instruction, its response code, and the visual artifact obtained by executing that code.The instruction may include text and optional visual references.Source: §3.1 · p. 4
  • Code rubrics assess properties verifiable from the program, while visual rubrics assess properties observable in the rendered artifact.Together, the two rubric sets distinguish program-level from visual-level evaluation.Source: §3.1 · p. 4

3.2. Rubric-Based Policy Optimization

Rubric scores provide a structured reward for optimizing policies on visual-coding tasks. Failed execution or rendering receives zero reward, and successful rollouts are optimized using standard policy objectives.

3.2. Rubric-Based Policy Optimization

  • Rubric scores are normalized within each modality and aggregated into a common reward scale across samples with different numbers and types of criteria.The evaluator is used as a structured reward function during policy optimization.Source: §3.2 · p. 6
  • Rollouts that fail to execute or render receive reward 0.Successful rollout trajectories are scored by the judge model, and those scores contribute to within-group normalized advantages.Source: §3.2 · p. 6
  • The policy is optimized with the standard GRPO clipped objective; MoE models also use GSPO with Rollout Routing Replay to stabilize expert training.The paper reports using these techniques in practice for MoE models.Source: §3.2 · p. 6

4. PrismaCoder-2M

PrismaCoder-2M is built from complementary visual-coding seeds, then gated and curated with PrismaForge rubrics for supervised training and reinforcement learning. RL focuses on instructions where the SFT policy has mixed success across rollouts.

4. PrismaCoder-2M

  • PrismaForge collects seeds, gates them by rubric score into a supervised corpus, and reuses the rubrics as rewards for RL.Figure 4 depicts the dataset composition by input modality and task family.Source: §4 · p. 6, Figure 4 · p. 7

4.1. Data Sourcing

  • PrismaCoder-2M combines plotting datasets, scientific code and figures, front-end pages and demos, and symbolic-notation tasks.The data spans four seed types, including plotting languages, web interaction, chemical structures, LATEX, Asymptote, LilyPond, and Manim.Source: §4.1 · pp. 6–7

4.2. Iterative Data Gating

  • Candidates are executed or rendered, scored against code and visual rubrics, and discarded, resynthesized, or retained according to score thresholds.Mid-band samples can be regenerated using failed rubric items as feedback; retained candidates can seed new samples.Source: §4.2 · pp. 7–8

4.3. RL Data Curation

  • RL uses held-out medium- or hard-difficulty instructions where SFT sometimes succeeds and sometimes fails, providing rollout-quality contrast for group-relative optimization.For Python, HTML, and SVG, five rollouts are scored per candidate and the 5K instances with the largest rubric-score dispersion are retained.Source: §4.3 · p. 8

5. Experiments

Across vision-centric and text-centric benchmarks, PrismaCoder models lead open baselines on many tasks and approach or exceed proprietary systems in several reported comparisons. The experiments evaluate both modalities using benchmark-specific protocols and deterministic metrics alongside judge scores.

5.1. Experimental Setup

  • PrismaCoder-9B and PrismaCoder-35B are post-trained from Qwen3.5 models, and the study also trains Gemma-4-31B-it and Qwen3-VL-30B-A3B-Instruct.Baselines include open multimodal code models Janus-CoderV, VinciCoder, and VisCoder2-7B.Source: §5.1 · p. 8

5.2. Benchmarking

  • The benchmark suite covers vision-centric and text-centric tasks, and evaluation follows each benchmark’s official protocol with deterministic metrics and judge scores.The authors also check whether gains hold across judges because some designated judges share a model family with their pipeline.Source: §5.2 · p. 9

5.3. Vision-Centric Results

  • On front-end generation, PrismaCoder-9B and PrismaCoder-35B score more than 20 points above task-specific open baselines; both trail all Claude 4.6 and Gemini 3 variants only on DesignBench editing.PrismaCoder-9B also exceeds a proprietary model on DesignBench generation and WebCode2M CLIP score.Source: §5.3 · p. 9, Table 2 · p. 9
  • On ChartMimic, both models lead open baselines, and PrismaCoder-35B outperforms most proprietary models under both settings.PrismaCoder-9B makes the largest improvement over its base among evaluated models and surpasses most proprietary models.Source: §5.3 · p. 9, Table 2 · p. 9
  • PrismaCoder-35B parses every chemical-structure output and achieves the table’s highest Tanimoto similarity, 97.56; both models lead open baselines on InteractScience’s visual lens.VinciCoder-8B leads the image-conditioned SVG column among open baselines.Source: Table 2 · p. 9, §5.3 · p. 10

5.4. Text-Centric Results

  • On ArtifactsBench, both models exceed all open baselines and GPT-4o on overall score; PrismaCoder-35B scores 28.00 on InteractScience PFT, 3.20 points above PrismaCoder-9B.The PFT score is reported as comparable to Gemini 3 Flash.Source: §5.4 · p. 10, Table 3 · p. 10
  • PrismaCoder-35B leads open baselines on all three VisPlotBench metrics, with a Mean score 11.02 points above their best and comparable to Claude Sonnet 4.6.Source: §5.4 · p. 10, Table 3 · p. 10

6. Analysis

The analysis finds that rubric-filtered data outperforms direct-judge filtering, training gains extend to additional backbones, and rubric-based judging agrees more closely with human ratings. These findings are examined through matched-data comparisons, cross-backbone training, and a human study.

6.1. Rubric Efficacy

  • Rubric-filtered training data scores higher than equal-sized direct-scored data on every benchmark, by as much as 24 points on Vega-Lite.The gains are largest on text-centric formats and small on vision-centric benchmarks; the authors attribute the gap to which samples each judge retains.Source: §6.1 · pp. 10–11, Figure 5 · p. 11

6.2. Backbone Generalization

  • On Gemma-4-31B-it and Qwen3-VL-30B-A3B-Instruct, all twelve metrics finish above the base model, and RL adds to SFT on every metric.SFT contributes most of the ChartMimic gain, while RL drives most HTML gains and recovers every metric that dips after SFT alone.Source: §6.2 · p. 11, Figure 6 · p. 11

6.3. Rubric Agreement with Humans

  • Across three proprietary judges, rubric-based judging achieves substantially higher agreement with human judgments and greater consistency in induced partial ordering than direct scoring.The study sampled 100 training examples and assessed both absolute score agreement and pairwise ordering.Source: §6.3 · p. 12

7. Conclusion

The paper contributes a released data pipeline, dataset, and models, with rubric criteria spanning code and visual quality. Appendix procedures describe script decomposition, execution-based gatekeeping, and benchmark-test decontamination.

7. Conclusion

  • PrismaForge uses coarse-to-fine rubrics split into code-level and visual-level criteria, and PrismaCoder models lead open results while rivaling proprietary systems.The conclusion also reports generalization beyond the released models and task recipes.Source: §7 · p. 12, Figure 7 · p. 12

A.1. AST-Based Decomposition

  • Plotting and Manim files are parsed into self-contained units, each retaining needed imports and statements while dropping project-private helpers.A unit produces one figure from a plotting script or one scene from a Manim file.Source: §A.1 · p. 18

A.2. Per-Format Gatekeepers

  • Per-format gatekeepers execute or render candidates in matched environments, rejecting errors, timeouts, or empty artifacts; failed RL rollouts receive reward 0.Source: §A.2 · p. 19

A.3. Decontamination

  • Synthesized training samples are decontaminated against benchmark test instructions by removing samples whose instructions share a long n-gram with a test instruction.Source: §A.3 · p. 19

B. Data Sources

The appendix reports data-source routing and compares open-work coverage, then illustrates how synthesized code and vision rubrics expose different aspects of webpage quality. A reconstruction example shows why rendered-page checks can reveal failures missed by implementation checks and holistic scoring.

B. Data Sources

  • The source-routing statistics show that higher-quality, task-aligned datasets retain more than half of original responses, while broad GitHub and arXiv sources require more regeneration.Table 4 summarizes source samples and counts at each gating stage.Source: B. Data Sources · p. 19, Table 4 · p. 19
  • Table 5 compares data types and methods across open multimodal code-intelligence works, marking corpus presence, absence, or unspecified coverage.The comparison is based on what the works' public papers report.Source: B. Data Sources · p. 19, Table 5 · p. 20

C. Rubric Synthesis Prompts and Examples

D. RL Reward Specification

The reward specification combines code and rendered-artifact scores, with a holistic component for tasks where fixed rubrics may miss visual mismatches. The section also reports a focused comparison, evaluation scope, and training and benchmark details.

D. RL Reward Specification

  • A dispatcher routes each rollout to one task-family scorer for a reward from 0 to 1, while invalid or unscoreable completions receive zero.The procedure first extracts code of the required type; unscoreable cases include execution, rendering, or persistent judge failures.Source: D. RL Reward Specification · p. 23
  • Code and rendered artifacts are scored separately and combined by task-specific weighted averages, with a holistic score added for interface reproduction when flexible layouts can evade predefined rubrics.The relative importance of implementation versus visual readiness and interactivity varies across task families.Source: D. RL Reward Specification · p. 23
  • Rubric-based scoring improves ChartMimic overall by 6.91 points over direct scoring in a focused Python-plotting experiment.A full GRPO comparison was not possible under resource constraints; rubric scoring requires about twice the API cost of scoring visual outputs alone, and one training round used approximately 808K judgment requests.Source: D. RL Reward Specification · p. 24

E. Benchmark Details and Evaluation Settings

  • Code-driven video and animation, and theorem and symbolic visualization, are included as training data but excluded from evaluation.Evaluation otherwise follows benchmark protocols, including designated judges where specified.Source: E. Benchmark Details and Evaluation… · p. 24

F. Training Details

  • SFT runs for five epochs, followed by one epoch of GRPO/GSPO from the SFT checkpoint, with eight candidate responses generated per prompt during rollouts.The models initialize from Qwen3.5 variants; the online rollout judge is Gemini-3-Flash.Source: F. Training Details · p. 25

G. Full Per-Benchmark Results

  • The appendix provides complete per-benchmark metric sets in separate tables, mirroring the evaluation workbook.The main text instead presents a compact headline metric set for each benchmark.Source: G. Full Per-Benchmark Results · p. 25

G.1. Visualization

Across visualization, web, and interactive-artifact evaluations, PrismaCoder models show strong results against open baselines, with performance varying by benchmark and metric. The appendix also reports agreement in training-effect direction across two judges, with one HTML task-score ranking disagreement.

G.1. Visualization

  • ChartMimic Direct-1800 overall scores are 77.70 for PrismaCoder-9B and 82.05 for PrismaCoder-35B, versus 74.20 for GPT-4o; 35B exceeds Claude Sonnet 4.6 and Claude Opus 4.6.The 35B model also records 96.16 execution rate, 90.41 text fidelity, and 94.00 layout.Source: §G.1 · p. 25, Table 8 · p. 26
  • On VisPlotBench, PrismaCoder-35B has a 28.86% incorrect-code rate and a 62.47 mean task score, versus 23.95 for the best VinciCoder model.It leads the table on mean and good-plot scores under both lenses; failures remain concentrated in LilyPond, LATEX, and Asymptote.Source: §G.1 · p. 26, Table 10 · p. 27, Table 12 · p. 27
  • PrismaCoder-35B leads UniSVG's final score at 72.96 and every ChemDraw column, parsing all 128 outputs and matching the reference molecule exactly in 92.19%.On UniSVG, both PrismaCoder models lead open baselines on text-conditioned TSVGen; on ChemDraw, 95.31% of 35B outputs reach Tanimoto similarity 1.Source: §G.1 · p. 27, Table 11 · p. 27, Table 13 · p. 28

G.2. WebDev

  • On WebCode2M, PrismaCoder-9B's Visual scores remain 90.06, 90.02, and 91.15 across short, mid, and long pages, while CLIP declines from 90.29 to 87.32.Both models score below JanusCoder and VinciCoder on TreeBLEU, which measures reference HTML-tree similarity rather than rendered-page fidelity.Source: §G.2 · p. 28, Table 14 · p. 29
  • On DesignBench generation, PrismaCoder-9B scores 69.95 on Judge and 84.77 on CLIP, while 35B has the lowest MAE and highest SSIM among evaluated models.Both models' Editing-Judge scores are about 8.9, above prior open baselines and matching GPT-4o, but slightly behind Gemini 3.1 Pro.Source: §G.2 · pp. 28–29, Table 15 · p. 29
  • On Design2Code, PrismaCoder-9B reaches a 91.54 average Visual score, with its largest gains over VinciCoder-8B in Position, Color, and Block.Those dimensions reflect spatial alignment, color fidelity, and coverage of visual blocks.Source: §G.2 · p. 29, Table 16 · p. 30

G.3. Artifacts and Demos

  • On ArtifactsBench, both models outperform open and proprietary baselines overall, while InteractScience results show higher VQT scores than other open models and competitive PFT performance.The two models have different ArtifactsBench strengths: 9B on SVG and web applications, and 35B on game development and simulation.Source: §G.3 · pp. 30–31, Table 17 · p. 30, Table 18 · p. 31

H.1. Backbone Generalization

  • Across two judges, the direction of every Gemma-arm training effect agrees, except the HTML task-score ranking: GPT-4.1 ranks RL first while GPT-5.5 ranks SFT first.Figure 10 plots base and final scores for each metric, with DesignBench Edit scaled by ten for the percentage-scale display.Source: §H.1 · p. 31, Table 19 · p. 32, Figure 10 · p. 32

H.2. Cross-Family Adjudication and Human Evaluation

In the human study, rubric-mode judge scores align more closely with human ratings than direct holistic scores. The case studies then show visual failures in outputs that execute, alongside improved outputs after RL or model training.

H.2. Cross-Family Adjudication and Human Evaluation

  • Across three judges, agreement with human ratings rises from 0.30–0.51 to 0.86–0.92 for Krippendorff's α, and from 0.28–0.44 to 0.64–0.67 for Kendall's τ_b under rubric scoring.The study samples 100 SFT examples and compares direct holistic scoring with PrismaForge rubric scoring.Source: §H.2 · p. 32, Table 20 · p. 33

I. Case Studies

  • In the six VisPlotBench cases, the middle-panel outputs execute but fail visually; examples contrast PrismaCoder-9B with its base and PrismaCoder-35B with its SFT-only variant.The cases span LATEX, SVG, Mermaid, Vega-Lite, HTML, and Python.Source: §I · p. 33, Figure 13 · p. 34, Figure 14 · p. 34
  • On ChartMimic, the training-stage case shows SFT recovering one of a line chart's two cycles and RL restoring both, unlike the open reference.The paired-reference comparison reports that only the proprietary models reproduce the chart.Source: Figure 16 · p. 35, Figure 17 · p. 35
  • In Design2Code generations, PrismaCoder-9B repairs a broken title layout and only PrismaCoder-35B reproduces the second shown layout.The case-study set also includes two instruction-driven DesignBench edits.Source: §I · p. 33, Figure 18 · p. 36, Figure 19 · p. 37
  • In DesignBench editing cases, only PrismaCoder-9B adds the requested story points, column sums, and header menu while preserving the board; only 35B applies the other requested restyling.The second case requests renaming, rounded corners, and countdown restyling.Source: Figure 20 · p. 37, Figure 21 · p. 38

Related papers

Loading 2610.20538v1…