PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual-ERM: Reward Modeling for Visual Equivalence

Ziyu Liu, Shengyuan Ding, Xinyu Fang, Xuanlang Dai, Penghui Yang, Jianze Liang, Jiaqi Wang, Kai Chen, Dahua Lin, Yuhang Zang

arXiv:2603.13224v2cs.CVcs.AI

TL;DR

Vision-to-code reinforcement learning is hindered by rewards that miss fine-grained visual errors and can be exploited through reward hacking. Visual-ERM evaluates rendered outputs with interpretable, task-agnostic discrepancy feedback, improving performance across chart, table, and SVG tasks while also supporting test-time revision.

  • Problem

    Existing textual and coarse visual rewards fail to capture fine-grained visual discrepancies, making reinforcement-learning supervision challenging for vision-to-code.

  • Method

    Visual-ERM is a multimodal generative reward model that evaluates rendered vision-to-code outputs and provides discrepancy feedback for reinforcement learning and test-time scaling.

  • Results

    +8.4 on chart-to-code, +2.7 on table-to-markdown, and +4.1 on SVG-to-code are reported for Qwen3-VL-8B-Instruct, with further gains from reflection and revision.

  • Takeaways & Limitations

    Visual-ERM consistently improves vision-to-code performance across multiple tasks as a supervisor for reinforcement learning and test-time scaling.

Abstract

from arXiv · show

Vision-to-code tasks require models to reconstruct structured visual inputs, such as charts, tables, and SVGs, into executable or structured representations with high visual fidelity. While recent Large Vision Language Models (LVLMs) achieve strong results via supervised fine-tuning, reinforcement learning remains challenging due to misaligned reward signals. Existing rewards either rely on textual rules or coarse visual embedding similarity, both of which fail to capture fine-grained visual discrepancies and are vulnerable to reward hacking. We propose Visual Equivalence Reward Model (Visual-ERM), a multimodal generative reward model that provides fine-grained, interpretable, and task-agnostic feedback to evaluate vision-to-code quality directly in the rendered visual space. Integrated into RL, Visual-ERM improves Qwen3-VL-8B-Instruct by +8.4 on chart-to-code and yields consistent gains on table and SVG parsing (+2.7, +4.1 on average), and further strengthens test-time scaling via reflection and revision. We also introduce VisualCritic-RewardBench (VC-RewardBench), a benchmark for judging fine-grained image-to-image discrepancies on structured visual data, where Visual-ERM at 8B decisively outperforms Qwen3-VL-235B-Instruct and approaches leading closed-source models. Our results suggest that fine-grained visual reward supervision is both necessary and sufficient for vision-to-code RL, regardless of task specificity.

1. Introduction

Vision-to-code converts structured visuals into executable or structured representations, but existing rewards miss important visual discrepancies. Visual-ERM addresses this gap with fine-grained, interpretable, task-agnostic feedback in rendered visual space and improves multiple downstream settings.

  • Vision-to-code converts charts, tables, and SVGs into executable code or markup for downstream uses including UI generation, scientific parsing, and knowledge management.
  • Existing rewards are misaligned: textual metrics ignore rendering, while vision-encoder similarities remain coarse and opaque.
  • Visual-ERM scores rendered outputs with a multimodal generative model trained on discrepancy-annotated reference–prediction image pairs.
  • Visual-ERM also supplies localized feedback for reflection and revision during test-time scaling.
  • +8.4 points on chart-to-code, +2.7 on table-to-markdown, and +4.1 on SVG-to-code are reported for Qwen3-VL-8B-Instruct with Visual-ERM-guided reinforcement learning.

2. Analysis of Reward Signals for Vision-to-Code

The analysis shows that text-based and vision-based rewards can score visually wrong outputs highly and encourage reward hacking. Reliable vision-to-code rewards therefore need rendered-image evaluation that is fine-grained, interpretable, and task-agnostic.

  • Setup: Visual equivalence requires rendered predictions to match references in layout, axes, labels, and numeric content.
  • Text-based rewards: TEDS and edit distance can reward textually similar code that renders incorrectly and penalize visually correct equivalent code.
  • Vision-based rewards: DINO and CLIP tolerate spatial or structural shifts that alter chart or table meaning, while their scalar scores hide the erroneous element.
  • Reward hacking under RL: Text-based and vision-based rewards plateau under RL because policies maximize proxy scores without improving rendered output.
  • Downstream impact: Policies trained with those rewards fail to consistently improve over SFT and can underperform it on Table-to-Markdown, whereas Visual-ERM yields the largest gains in comparison.
  • Summary: The analysis identifies four required reward properties: visual, fine-grained and locally sensitive, interpretable, and task-agnostic.

3. Methods

Visual-ERM is trained on fine-grained discrepancy annotations for rendered reference–prediction pairs and deployed as a unified evaluator across chart, SVG, and table tasks. Its outputs support both RL rewards and interpretable test-time revision, alongside the VC-RewardBench diagnostic benchmark.

  • 3.1. Visual-ERM: Data, Annotation, and Training: Visual-ERM uses a discrepancy-annotated dataset and a single LVLM trained across chart, table, and SVG tasks.The annotations encode discrepancy category, severity, location, and description.
  • 3.1. Visual-ERM: Data, Annotation, and Training: Training pairs combine targeted edits that inject predefined errors with natural inferences that reflect rollout errors.Targeted edits cover structural, data, textual, and stylistic failures.
  • 3.1. Visual-ERM: Data, Annotation, and Training: A stronger bootstrap teacher proposes discrepancy annotations, which are filtered using rendering-consistency checks before supervised fine-tuning.The paper reports that Qwen3-VL-235B-Instruct misses subtle structural and textual deviations.
  • 3.2. Visual-ERM as an RL Reward: During RL, Visual-ERM evaluates rendered predictions, combines discrepancy severities with render success, and supplies the resulting bounded reward to GRPO.Unrenderable code is excluded before visual fidelity is scored, and policy optimization uses a KL anchor to a reference policy.
  • 3.3. Visual-ERM for Test-Time Scaling: For test-time scaling, Visual-ERM returns structured discrepancy feedback that triggers iterative revision when the initial reward falls below a threshold.The loop can repeat for up to T steps, but gains are bounded by the policy’s editing ability because the policy weights are not updated.
  • 3.4. VisualCritic-RewardBench: VC-RewardBench evaluates fine-grained image-to-image discrepancy judgment using 1,335 annotated instances consolidated through multi-model annotation and expert review.Its evaluation matches predicted and reference discrepancies to compute Precision, Recall, F1, and severity-score correlation.

4. Experiments

The experiments evaluate Visual-ERM as an RL reward, discrepancy judge, and test-time critic across chart, table, and SVG tasks. It improves downstream parsing and supports iterative reflection and revision.

  • Evaluation Setup: Visual-ERM is evaluated for RL utility, reward quality, and test-time scaling across three vision-to-code tasks.The setup uses Qwen3-VL-8B-Instruct for GRPO and evaluates chart-to-code, table-to-markdown, SVG-to-code, and VC-RewardBench.
  • Reinforcement Learning: +11.8 and +4.9 average points are gained on ChartMimic-v2 direct and customized settings using Qwen3-VL-8B-Instruct.Using VinciCoder-8B-SFT also yields +10.3 and +9.8 under the two settings.
  • Reinforcement Learning: +2.7 overall gain is achieved on Table-to-Markdown, spanning textual-recognition and structural-reconstruction metrics.The comparison includes TEDS and DINO-based rewards, which do not provide the same consistent improvements.
  • Reinforcement Learning: Visual-ERM-guided RL delivers consistent SVG-to-Code gains, while DINO-based RL degrades the strong VinciCoder-8B-SFT baseline.The result supports more precise guidance from Visual-ERM than standard feature-similarity rewards.
  • Reward Quality: +36.8/+38.2/+40.9 improvements in F1_h/F1_s/Sc over the base are reported on VC-RewardBench.Even Qwen3-VL-235B-Instruct struggles with fine-grained visual and textual discrepancies.
  • Test-Time Scaling: +8.0 Avg over the base and a further +3.1 over the Visual-ERM-RL-tuned policy are obtained with three reflection rounds on Chart-to-Code.Increasing reflection rounds yields consistent improvements, with diminishing returns beyond three rounds.

5. Related Work

Prior reward-modeling work is largely text-centric, whereas vision-to-code quality depends on visual fidelity. Visual-ERM is introduced to provide visual-space supervision for these tasks.

  • Reward Models: Reward models commonly use Bradley–Terry scalar rewards or generative feedback, but most prior systems target text-centric generation.These forms do not directly support visual-to-code evaluation, where rendered visual fidelity is central.
  • Visual-ERM: Visual-ERM evaluates vision-to-code outputs in visual space with fine-grained, interpretable, and task-agnostic supervision.It is proposed as a visual equivalence reward model for visual-to-code tasks.
  • Visual-to-Code Tasks: Vision-to-code converts charts, tables, and SVGs into executable or structured representations such as Python, Markdown, HTML, or vector code.The task family supports downstream applications including structured perception and document-related uses.
  • RL for Visual-to-Code Tasks: Supervised fine-tuning is data-intensive, while existing RL rewards often use textual or structural proxies rather than direct visual evidence.These limitations motivate a reward model aligned with rendered outputs.

6. Conclusion

The paper concludes that Visual-ERM provides effective supervision for vision-to-code RL and test-time scaling, supported by a dedicated discrepancy benchmark and analyses across structured visual domains.

  • Conclusion: Visual-ERM evaluates vision-to-code outputs in visual space and provides fine-grained, interpretable, and task-agnostic supervision.The model is used for both reinforcement learning and test-time scaling.
  • Conclusion: VC-RewardBench directly evaluates image-to-image discrepancy judgment across vision-to-code tasks.The benchmark targets fine-grained discrepancy detection and interpretable feedback.
  • Appendix: The appendix documents models, datasets, benchmark construction, experimental setups, computational efficiency, ablations, prompts, and qualitative case studies.Additional analyses include VQA, multi-task data mixing, judge robustness, and reward-design studies.
  • Scope and Model: Qwen3-VL-8B-Instruct is not a reliable image-to-image discrepancy judge out of the box, especially for text- and layout-centric structured visuals.Visual-ERM is trained on this backbone to improve fine-grained judgment.
  • Data and Models: Visual-ERM is trained on 104K Chart-to-Code, 125K Table-to-Markdown, and 111K SVG-to-Code reward-modeling instances.The corpus is built from injected errors and naturally occurring parsing errors with structured annotations.
  • Benchmark: VC-RewardBench contains 1,335 curated examples spanning 595 charts, 442 SVGs, and 298 tables.Its annotations cover diverse realistic failure modes, including style, data, text, numeric, layout, shape, and structural discrepancies.

A.5. Computational Efficiency and Training Latency

The efficiency study compares text-based rules, DINOv2-large similarity, and Visual-ERM under identical training configurations. Visual-ERM achieves lower per-step latency than the smaller DINO baseline while providing stronger policy performance than text-based rewards.

  • Training Latency: Visual-ERM requires 0.17 hours per training step, compared with 0.24 hours for the 0.3B DINOv2-large baseline.The study benchmarks all reward paradigms under identical experimental configurations.
  • Training Latency: Text-based rewards are most efficient at 0.15 hours per step but yield the least effective policy performance.The result highlights a latency–policy-effectiveness trade-off among the compared reward signals.

B.1. Evaluation on General VQA Benchmarks

Visual-ERM-guided RL preserves general VQA performance while improving structured visual parsing, and mixed-task reward-model training produces more balanced cross-task judgments.

  • General VQA performance: General VQA performance remains essentially stable after Visual-ERM-guided RL, with slight average improvements for two of three policies.Compared with base Qwen3-VL-8B-Instruct at AVG 78.3, Chart-to-Code RL reaches 78.4 and SVG-to-Code RL 78.5, while Table-to-Markdown RL reaches 78.1.
  • General VQA performance: CharXiv-RQ improves for all RL variants, while DocVQA and InfoVQA remain nearly unchanged.CharXiv-RQ changes from 46.0 to 47.2, 46.7, and 46.4; DocVQA remains near 95.6 and InfoVQA is stable with a small SVG-policy gain.
  • Interpretation: The results indicate that vision-to-code gains do not require sacrificing broad chart- and document-centric multimodal competence.The evaluation specifically targets fine-grained visual perception, text recognition, and cross-modal reasoning beyond code generation.
  • Multi-task reward modeling: Mixed chart, table, and SVG training yields the best overall reward-model performance through transfer across overlapping error patterns.The authors attribute the benefit to shared recognition and layout failure modes across tasks.
  • Multi-task reward modeling: Single-task reward models show strong task bias, whereas mixed-data training produces more balanced cross-task performance.For example, Chart-Data-Only performs well on Chart but poorly on Table and SVG, while Table-Data-Only generalizes poorly to Chart.

B.2.2. Effect of Multi-Task Data Mixing on RL

On Table-to-Markdown RL, both table-only and mixed-data reward models improve performance, while the mixed model provides larger overall gains and stronger generalization.

  • RL impact: +2.7 average improvement is achieved in Table-to-Markdown RL with the richer multi-task reward signal.The mixed reward model improves overall performance despite a slight TA decrease relative to the table-only model.
  • RL impact: The table-only reward model reaches TEDS = 81.4 and TEDS-S = 86.3 on OmniDocBench while reducing Edit-Dist to 20.7.It also improves TEDS and TEDS-S and lowers Edit-Dist on olmOCRBench-v1.5.
  • Comparison: Mixed-data training yields larger overall gains than table-only training, indicating a more robust optimization direction for RL.The mixed model’s TA is 78.1 versus 79.4 for the table-only model, but its aggregate gains are larger.

B.3. Ablation on Different Judge Models

VC-RewardBench evaluation is robust to the choice of LLM judge: deterministic scores are identical, while judge-dependent matching metrics vary only modestly.

  • Evaluation protocol: The protocol uses constrained error-to-error matching rather than open-ended preference ranking, reducing sensitivity to judge reasoning style.The judge classifies matched, hallucinated, incorrectly described, and missed errors before computing strict and relaxed F1.
  • Results: F1_h ranges from 40.8 to 43.0 and F1_s from 42.4 to 45.1 across judges, limiting judge-dependent variability.At the task level, Table F1_s ranges from 55.7 to 57.8.
  • Results: S_c is identical across all judges, including Chart 61.2, Table 74.8, SVG 59.6, and AVG 58.4.This score is computed deterministically and does not depend on the LLM matcher.
  • Conclusion: Replacing the judge model yields consistent conclusions, supporting the robustness of VC-RewardBench evaluation.The protocol evaluates structured discrepancy matching under specified criteria.

B.5. More Test-Time Scaling Results

Visual-ERM-guided reflection improves SVG-to-Code test-time scaling both without and with RL, with additional gains after RL initialization.

  • Reflection without RL: +1.0 overall-score improvement results from Visual-ERM-guided reflection without RL, rising from 64.2 to 65.2.CLIPScore increases from 73.3 to 74.6, while SSIM remains comparable and LPIPS decreases from 60.0 to 58.1.
  • Mechanism: Visual-ERM uses discrepancy feedback to support iterative self-refinement at inference time.The policy produces an initial prediction and applies evaluator feedback across reflection and revision rounds.
  • Reflection with RL: Reflection further improves the Visual-ERM-guided RL policy, raising overall score from 69.4 to 69.8 (+0.4).CLIPScore increases from 77.5 to 78.1, while LPIPS remains essentially stable at 49.7 versus 49.2.
  • Reflection with RL: The RL policy with reflection achieves the best absolute performance in both CLIPScore and overall score.These results indicate that inference-time reflection adds gains on top of RL.

D.1. Data Cases

The cases use original/re-rendered image pairs with fine-grained error descriptions to expose domain-specific visual discrepancies in charts, tables, and SVGs. Together, they show how these annotations support interpretable supervision for training and evaluation while highlighting limitations of existing reward models.

  • Data Format: Training and benchmark examples pair original images with re-rendered images and fine-grained error descriptions.VC-RewardBench examples receive stronger proprietary-model labeling and additional human filtering than the training data.
  • Chart Cases: Chart cases identify geometry, color-mapping, text-placement, and numeric-scaling errors that textual or structural cues may miss.Examples include incorrect axis scaling, collapsed category palettes, misplaced labels, and distorted histogram distributions.
  • Table Cases: Table cases show that OCR and layout discrepancies can create major structural errors through missing header characters, units, punctuation, or symbol changes.Examples include comma-versus-dot decimal changes and confusion between the letter O and digit 0.
  • SVG Cases: SVG cases emphasize geometry and structure, including altered borders, glyph shapes, path details, composition, and layering.These discrepancies motivate judging rendered images directly because precise spatial relationships and vector geometry determine SVG fidelity.
  • Overall Analysis: The examples span diverse realistic discrepancy patterns, with annotations providing interpretable and actionable supervision for training and evaluation.Additional cases are presented to demonstrate limitations of existing reward models.
Loading 2603.13224v2…