PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Leads Now? Token-Level Modality Arbitration for Chart-to-Code Generation

Qinghao Fu, Yarong Wang, Shunlei Ning, Yilin Wang, Shunwen Bai, Xinda Wang, Jiaotuan Wang, Yinan Nie, Wei Zhou

arXiv:2608.15510v1cs.AI

TL;DR

Chart-to-code generation requires coordinating visual grounding and code generation, but existing methods do not explicitly model their varying interaction across inputs and generation contexts. MoCA separates visual and code branches and dynamically arbitrates their contributions, achieving strong executability and competitive visual fidelity across three benchmarks.

  • Problem

    Existing chart-to-code methods do not explicitly adjust visual grounding and code generation contributions across inputs and intermediate representations.

  • Method

    MoCA preserves separate visual and code branches while CAB dynamically arbitrates their relative contributions per input, layer, and generation step, with self-distilled supervision and multi-dimensional reinforcement learning.

  • Results

    Across three benchmarks, MoCA achieves strong code executability and competitive visual fidelity; on Plot2Code, its Code Pass rate is 87.88, outperforming the next-best baseline by 2.27 points.

  • Takeaways & Limitations

    MoCA’s gains highlight the importance of complementary branch initialization and input-conditioned arbitration over static parameter merging and fixed branch weighting.

Abstract

from arXiv · show

Chart-to-code generation requires a model to read the fine-grained visual details of a chart and write executable code that reproduces it. Existing chart-to-code methods either train visual and coding abilities separately, or fine-tune on chart-to-code data with the two abilities entangled. Neither strategy accounts for the distinct nature of the two abilities or the interference that arises when they are optimized together. We propose MoCA (Mixture of Cross-modal Arbitration), which separates the two abilities rather than blending them. MoCA is built on Cross-modal Arbitration Block (CAB), which maintains a visual branch and a code branch as two distinct pathways, and a lightweight arbiter that arbitrates their relative contributions at every layer and generated token. We train MoCA in two stages: a supervised warm-up on self-distilled reasoning trajectories that decomposes visual understanding into explicit steps, followed by reinforcement learning with rewards on both the reasoning process and the final code. Analysis shows that the arbiter learns structured rather than arbitrary allocations, with expert contributions varying systematically across tokens, layers, and instances. Across three benchmarks, MoCA delivers competitive performance against general-domain and chart-specialized models. Ablation results show that the gains cannot be attributed to a larger model size alone, but instead arise from the joint contributions of complementary visual and code branch initialization and input-conditioned arbitration through CAB.

Introduction

Chart-to-code generation requires both visual grounding and executable, visually faithful code, but their relative importance varies across charts and intermediate representations. MoCA addresses this adaptive coordination problem with separate visual and code branches, token- and layer-level arbitration, and two-stage supervised and reinforcement learning.

  • Motivation: Chart-to-code generation must recover chart data and structure as syntactically correct, visually faithful executable plotting code.This makes it more challenging than chart captioning or question answering.
  • Motivation: Visual grounding and code generation have input-dependent importance: some charts demand dense-label and color recognition, while others demand complex plotting logic.Examples include stacked structures, reference lines, and customized axes.
  • Method: MoCA preserves visual and code expertise in separate branches and uses CAB’s lightweight Arbiter to allocate branch contributions for each token at every layer.The Arbiter predicts weights from intermediate hidden representations, producing instance-specific capability allocation.
  • Training: MoCA trains with supervised self-distilled Snippet-of-Thought trajectories, then reinforcement learning using rewards for format, reasoning, structure, execution, and visual similarity.The two stages establish initial visual-code alignment and optimize both reasoning and final code quality.
  • Evaluation: Across ChartMimic, Plot2Code, and ChartX, MoCA achieves competitive performance, while ablations indicate gains arise from coordination rather than model size alone.Analysis reports structured allocation patterns across tokens, layers, and instances.

Related Work

Related work spans multimodal language models, model merging, and chart-to-code generation. While prior methods improve multimodal alignment, capability integration, or task-specific supervision, internal coordination between visual grounding and code synthesis remains underexplored.

  • MLLMs: MLLMs extend language models with visual understanding for joint reasoning over images and text across multimodal tasks.Early approaches align visual features with language representations through cross-modal projection or query-based interaction mechanisms.
  • Model Merging: Model merging integrates capabilities from multiple pretrained models through parameter-space combination, often without retraining source models or accessing original training data.Merged parameters can be deployed directly or used to initialize subsequent task-specific adaptation.
  • Model Merging: Recent merging methods improve integration through parameter- or layer-level weighting, subspace or low-rank decomposition, and input-dependent merging coefficients.These directions respectively target merging quality, reduced task interference, and input-conditioned capability combination.
  • Chart-to-Code Generation: Chart-to-code research has progressed from datasets and benchmarks toward generation improvements using preference learning and reinforcement learning.Recent methods employ textual, structural, execution-based, or visual rewards to improve code executability and chart reconstruction quality.
  • Chart-to-Code Generation: Existing chart-to-code methods emphasize task-specific training signals and feedback objectives, while internal coordination between visual grounding and code synthesis remains underexplored.This gap motivates maintaining distinct visual and code expertise rather than focusing only on improved supervision or reward design.

Methodology

MoCA separates visual grounding from code generation through distinct branches and an input-conditioned arbiter that adjusts their contributions across tokens, layers, and instances. It trains this architecture with self-distilled reasoning supervision followed by reinforcement learning using process- and outcome-level feedback.

  • Cross-modal Arbitration Block: MoCA addresses heterogeneous chart-to-code requirements by preserving separate visual and code transformations instead of using static parameter merging.The mechanism dynamically adjusts branch contributions from intermediate hidden representations.
  • Branch Initialization: MoCA initializes the visual branch from Qwen2.5-VL-7B feed-forward parameters and the code branch from corresponding Qwen2.5-Coder-7B parameters.Other backbone components retain Qwen2.5-VL-7B initialization.
  • Cross-modal Arbitration Block: CAB maintains visual and code branches while a lightweight arbiter predicts their relative contributions from the current hidden representation.Branch weights can vary across input instances, network layers, and generation tokens.
  • Two-Stage Training: Training proceeds in two stages: SFT on self-distilled structured reasoning trajectories and executable code, followed by GRPO-based RL with execution- and structure-aware feedback.The first stage connects visual observations, chart structures, and code decisions; the second uses relative advantages without a separately learned critic.
  • Reward Design: RL combines five rewards covering format, reasoning-code consistency, chart structure, code execution, and rendered-chart visual similarity.The aggregation includes process-level rewards Rfmt, Rcon, and Rstr, plus outcome-level rewards Rexec and Rvis.

Experiment

The experiment section evaluates MoCA comprehensively across three benchmarks, combining comparisons with existing models and ablation analyses of its effectiveness and key components.

  • MoCA is evaluated comprehensively on three benchmarks.
  • The experiments compare MoCA with existing models.
  • Detailed ablations assess MoCA’s effectiveness and key components.

Experimental Settings

The experimental setup builds MoCA on Qwen2.5-VL-7B with a CAB replacing each Transformer block’s FFN, using separately initialized visual and code branches. Evaluation spans three chart-to-code benchmarks under unified prompting, with GPT-4o-based visual similarity assessment and specified SFT/RL training regimes.

  • Model and Initialization: The visual and code branches initialize from the Qwen2.5-VL-7B and Qwen2.5-Coder-7B FFN parameters, respectively.Comparisons include both general-domain and chart-specialized models.
  • Evaluation: MoCA is evaluated on three widely used chart-to-code benchmarks with unified prompting across models.Plot2Code results use the full evaluation set, including non-executable outputs.
  • Evaluation: GPT-4o serves as the automatic evaluator for GPT-based visual similarity metrics, following prior work.The evaluation compares MoCA with general-domain and chart-specialized baselines, including GPT-4o-mini, Qwen2.5-VL variants, ChartLlama-13B, ChartCoder-7B, and ChartVLM-L-14B.
  • Training: SFT runs for one epoch on 8 NVIDIA H200 GPUs at a learning rate of 1 × 10^-5 and global batch size 64, while RL runs for three epochs on 32 GPUs at 1×10^-6.RL generates eight responses per prompt; custom QKV sharding and SwiGLU kernels are integrated with vLLM for efficient dual-branch execution.

Overall Performance

MoCA performs strongly across three chart-to-code benchmarks, achieving the best result on four of seven reported metrics and remaining competitive on the others. It leads all three ChartMimic metrics and obtains the highest Plot2Code Code Pass rate.

  • MoCA achieves the best result on four of the seven reported metrics across three chart-to-code benchmarks.It remains competitive on the remaining metrics.
  • On ChartMimic, MoCA leads all three metrics with an execution rate of 88.83, low-level score of 75.13, and high-level score of 7…
  • 87.88, MoCA achieves the highest Plot2Code Code Pass rate, outperforming the next-best baseline by 2.27 points.Together with its leading ChartMimic execution rate, this indicates a consistent executability advantage across datasets.

Visualization of Arbitration Behavior

CAB’s arbiter learns structured, input-conditioned coordination rather than uniform branch weighting. Visual and code contributions vary across tokens, layers, and inputs, with layer-specific shifts between visual emphasis and code generation.

  • Token-Level Arbitration: Branch weights vary across token positions within a layer, especially in Layer 17, showing fine-grained coordination rather than sequence-level weighting.The arbiter adjusts visual and code contributions according to evolving hidden representations during generation.
  • Layer-Level Arbitration: Layer 7 emphasizes the visual branch, Layer 17 increases and varies code contribution, and Layer 25 shifts back toward visual processing.This nonmonotonic pattern indicates stage-dependent coordination across network depth.
  • Layer-Level Arbitration: The arbiter produces structured coordination across depth rather than uniform or arbitrary branch weights.Shallow layers emphasize visual processing, intermediate layers support code generation, and deeper layers may return to visual refinement.
  • Instance-Level Arbitration: Branch-weight differences between two Plot2Code inputs become more pronounced in several middle and deeper layers, with localized token-level variation.The visualization provides qualitative evidence that arbitration patterns are instance-specific rather than identical across inputs.

Ablation Study

Ablations show that MoCA’s gains arise from complementary training and architecture choices rather than task-specific training or model capacity alone. SFT enables valid code generation, while RL, self-distillation, complementary initialization, and input-conditioned arbitration each provide additional benefits.

  • SFT and RL: SFT achieves a Code Pass rate of 83.33, a Text Match score of 34.11, and a GPT-4o Score of 3.66, while RL alone fails.MoCA without task-specific training and MoCA trained with RL alone fail to produce valid structured outputs.
  • SFT and RL: Building on SFT, RL raises the Code Pass rate to 87.88, the Text Match score to 41.07, and the GPT-4o Score to 4.45.The improvement uses a multi-dimensional reward function.
  • Model capacity: The SFT-and-RL Qwen2.5-VL-7B backbone reaches 84.10 Code Pass, 36.11 Text Match, and 3.87 GPT-4o, remaining below MoCA across all metrics.Task-specific training substantially improves the original backbone but does not match MoCA.
  • Branch initialization: Visual + Visual scores 85.26, 37.48, and 3.94, while MoCA improves by 2.62, 3.59, and 0.51 points, respectively.The comparison supports complementary visual and coding expertise rather than parameter count as the source of gains.
  • Input-conditioned arbitration: MoCA outperforms fixed weighting, which scores 86.74 Code Pass, 39.82 Text Match, and 4.26 GPT-4o, demonstrating input-conditioned arbitration.The fixed-weight variant uses αvis = αcode = 0.5 for every token and layer.
  • Self-distillation: Self-distilled SoT trajectories increase Code Pass from 80.30 to 83.33 under the same SFT configuration.The result suggests improved alignment with the model’s output distribution, narrowing the training–inference gap and improving generalization.

Case Study

The case study shows that MoCA more faithfully reconstructs chart semantics than the compared models. It correctly captures data distributions, reference lines, legends, and color mappings where baselines make scale, layout, or code errors.

  • Bar chart: MoCA correctly reconstructs the bar chart’s data distribution, dashed reference lines, legend, and consistent color mapping.GPT-4o-mini instead produces incorrect y-axis scales and misplaces the legend, resulting in incomplete semantics.
  • Stacked chart: The stacked chart is more challenging because it has more categories, complex color assignments, and requires accurate value accumulation.These factors increase the chart’s reconstruction difficulty relative to the bar-chart example.
  • Stacked chart: Qwen2.5-VL-32B fails to generate a valid stacked chart because of code errors.GPT-4o-mini produces a vertical layout instead of the original, although the supplied passage truncates before completing the comparison.

Conclusion

MoCA coordinates distinct visual and code branches through token- and layer-level arbitration, combining self-distilled supervised training with multi-dimensional reinforcement learning to produce executable code with competitive visual fidelity.

  • Framework: MoCA coordinates visual and code branches to generate executable code with faithful visual reconstruction.The framework keeps the two branches distinct rather than blending their contributions.
  • Cross-modal arbitration: CAB dynamically adjusts each branch’s relative contribution for every generated token and layer using intermediate hidden representations.A lightweight arbiter performs the input-conditioned allocation across the generation process.
  • Training and results: Combining supervised fine-tuning on self-distilled SoT trajectories with reinforcement learning and multi-dimensional rewards, MoCA achieves strong code executability and competitive visual fidelity.The training strategy jointly targets the reasoning process and final code quality.
Loading 2608.15510v1…