PaperlayerView exact revision on arXiv

Source-linked AI summary

CodePercept: Code-Grounded Visual STEM Perception for MLLMs

Tongkun Guan, Zhibo Yang, Jianqiang Wan, Mingkun Yang, Zhengtao Guo, Zijian Hu, Ruilin Luo, Ruize Chen, Songtao Jiang, Peng Wang, Wei Shen, Junyang Lin, Xiaokang Yang

arXiv:2603.10757v2cs.CV

TL;DR

The paper asks whether STEM visual-reasoning failures arise from perceptual deficiencies or reasoning limitations. It performs independent scaling analysis and develops executable-code-based training and evaluation, finding that perception is the primary bottleneck and that code enables deterministic assessment of comprehensive visual understanding.

  • Problem

    The central gap is determining whether STEM visual-reasoning failures reflect perception deficiencies or reasoning limitations, while existing evaluation mainly measures problem-relevant understanding.

  • Method

    The paper decouples and independently scales perception and reasoning, then introduces ICC-1M, two code-grounded training tasks, and STEM2Code-Eval for executable image reconstruction.

  • Results

    Perception scaling consistently yields greater gains than reasoning scaling, identifying perception as the primary bottleneck in STEM visual reasoning.

  • Takeaways & Limitations

    Executable code provides a structured perceptual medium for training and a deterministic, verifiable way to assess comprehensive STEM visual understanding.

Abstract

from arXiv · show

When MLLMs fail at Science, Technology, Engineering, and Mathematics (STEM) visual reasoning, a fundamental question arises: is it due to perceptual deficiencies or reasoning limitations? Through systematic scaling analysis that independently scales perception and reasoning components, we uncover a critical insight: scaling perception consistently outperforms scaling reasoning. This reveals perception as the true lever limiting current STEM visual reasoning. Motivated by this insight, our work focuses on systematically enhancing the perception capabilities of MLLMs by establishing code as a powerful perceptual medium--executable code provides precise semantics that naturally align with the structured nature of STEM visuals. Specifically, we construct ICC-1M, a large-scale dataset comprising 1M Image-Caption-Code triplets that materializes this code-as-perception paradigm through two complementary approaches: (1) Code-Grounded Caption Generation treats executable code as ground truth for image captions, eliminating the hallucinations inherent in existing knowledge distillation methods; (2) STEM Image-to-Code Translation prompts models to generate reconstruction code, mitigating the ambiguity of natural language for perception enhancement. To validate this paradigm, we further introduce STEM2Code-Eval, a novel benchmark that directly evaluates visual perception in STEM domains. Unlike existing work relying on problem-solving accuracy as a proxy that only measures problem-relevant understanding, our benchmark requires comprehensive visual comprehension through executable code generation for image reconstruction, providing deterministic and verifiable assessment. Code is available at https://github.com/TongkunGuan/Qwen-CodePercept.

1. Introduction

The paper identifies perception, rather than reasoning, as the primary bottleneck in STEM visual reasoning and proposes executable code as a perceptual medium. It introduces code-grounded training data and a reconstruction-based benchmark to enhance and directly evaluate comprehensive visual understanding.

  • Perception scaling consistently produces greater performance gains than reasoning scaling, identifying perception as the true lever in current STEM visual reasoning.The analysis decouples visual perception from reasoning and scales each independently while holding the other fixed.
  • Knowledge-distilled STEM captions are limited by teacher-model hallucinations involving spatial positions, quantitative relationships, and element interactions.These errors motivate grounding perception in executable code rather than relying solely on generated natural-language descriptions.
  • Existing problem-solving accuracy measures only problem-relevant understanding, leaving comprehensive STEM visual perception without a direct evaluation paradigm.The paper frames this limitation as a gap in evaluating visual comprehension itself.
  • Executable Python reconstruction code provides deterministic and verifiable assessment because accurate image reproduction requires complete visual comprehension.The benchmark evaluates whether generated code can reconstruct the target image.
  • STEM2Code-Eval contains 1,000 manually annotated images spanning mathematics, physics, chemistry, and electrical engineering.Its construction uses code-agent generation, candidate selection, and human annotation to ensure code quality.
  • ICC-1M contains over 1M STEM Image-Caption-Code pairs supporting Code-Grounded Caption Generation and STEM Image-to-Code Translation.Its synthesis pipelines reproduce images from executable Python, diversify STEM concepts across contexts, and target solid-geometry code generation.

2. Related Work

Prior STEM MLLM research has concentrated on improving reasoning, while perceptual deficiencies and direct evaluation of comprehensive STEM visual understanding remain comparatively underaddressed.

  • Existing STEM MLLM research primarily enhances reasoning through cold-start thinking data, reinforcement learning, and related strategies.The paper contrasts this emphasis with the limited attention given to perception.
  • Few works explicitly address perceptual deficiencies, motivating executable code as verifiable ground truth for comprehensive visual understanding.The proposed direction targets systematic training methods for perceptual capabilities.

3. CodePercept Methodology

CodePercept builds STEM image-code data through reproduction, diversification, and template-based solid-geometry synthesis, then uses code-grounded tasks and post-training to enhance visual perception. Its pipeline combines executable code, natural-language captions, quality filtering, supervised fine-tuning, and reinforcement learning.

  • Image-Code Pair Construction: CodePercept expands public STEM images into image-code pairs through image reproduction, image diversity, and solid-geometry synthesis pipelines.The solid-geometry pipeline uses code templates to address deficiencies in generating precise spatial configurations.
  • Image-Code Pair Construction: Image reproduction generates executable Python code from an image and an intermediate caption, while image diversity produces multiple code variations from extracted scientific principles.The diversity pipeline preserves STEM rigor while introducing structurally novel instantiations.
  • Image-Code Pair Construction: Solid-geometry templates are parameterized by attributes such as arrangements, viewing angles, and spatial configurations, enabling structured geometric correctness and visual diversity.Parameters are sampled systematically from each template’s parameter space.
  • Training: CodePercept uses native image-code pairs for code-grounded caption generation and STEM image-to-code translation, followed by supervised fine-tuning and reinforcement learning.The resulting ICC-1M dataset contains over 1M curated image-caption-code triplets.
  • Image-Code Pair Construction: The pipeline applies composite filtering for image quality, code quality, and image-code consistency before retaining generated pairs.Quality metrics are evaluated using a specialized MLLM.
  • Code-Grounded Caption Generation: Code-grounded caption generation combines a natural-language draft with code-derived and execution-verified visual facts to produce precise captions.The execution tracer records coordinates, dimensions, spatial relationships, quantitative attributes, rendering semantics, and STEM mappings.

4. STEM2Code-Eval Benchmark

STEM2Code-Eval directly measures visual perception by requiring executable Python code that reconstructs STEM images, rather than relying on end-task problem-solving accuracy. The benchmark uses curated images and evaluates reconstruction quality, code quality, and execution success.

  • Benchmark Design: STEM2Code-Eval requires models to generate executable Python code that faithfully reproduces STEM images for deterministic and verifiable perception assessment.Its images come from six VQA-based STEM benchmarks spanning multiple domains.
  • Evaluation Metrics: STEM2Code-Eval reports Image Scoring, Code Scoring, and Exec Rate to assess visual similarity, generated-code quality, and execution success.The evaluation uses 1k samples.
  • Benchmark Construction: The benchmark selects and refines 1,000 examples through similarity ranking, difficulty-based iteration counts, expert scoring, and human code refinement.Ten expert annotators score style, content, and functionality on a 5-point scale before the highest-scoring examples are selected.

5. Experiments

Experiments show that CodePercept improves both problem-solving and direct image-reproduction perception, with reinforcement learning further strengthening executable reconstruction. Ablations also support complementary roles for code-grounded captions and image-to-code training.

  • 5.1. Main Results: CodePercept-4B/8B-S1 improve problem-solving performance by 2.8%/3.0% with Qwen3-30A3-Thinking and 2.9%/3.4% with Qwen3-235A22-Thinking.CodePercept-8B-S1 also surpasses Qwen2.5-VL-72B by 6.2%.
  • 5.1. Main Results: 54.09 and 59.64 are achieved by CodePercept-4B-S1 and CodePercept-8B-S1 on STEM2Code-Eval, improving over Qwen3-VL baselines by 10.6 and 12.3.The benchmark directly evaluates image-reproduction perception rather than only problem-relevant understanding.
  • 5.1. Main Results: 61.44 and 63.56 are achieved by CodePercept-4B-R1 and CodePercept-8B-R1 after reinforcement learning optimization for executability and reconstruction accuracy.Both models surpass the cited super-large models, including Seed1.6-Vision and Qwen3-VL-Plus.
  • 5.2. Ablation Study: Average scores rise from 60.91 to 62.15 to 62.75 across the three data-generation pipelines.The passage identifies ID-CodeCap, which includes diverse procedurally generated STEM images with verified code, as producing the greatest performance.
  • 5.2. Ablation Study: Image-to-caption and image-to-code tasks reinforce each other: captions provide semantic context, while code supplies structured representations for complex mathematical visuals.The results characterize code as complementary to image captioning and useful both as intermediate supervision and as a direct target.

6. Conclusion

The conclusion identifies visual perception as the primary STEM bottleneck and presents CodePercept as a code-based solution. Its contributions include ICC-1M, two code-grounded training approaches, and STEM2Code-Eval for executable image reconstruction.

  • 6. Conclusion: CodePercept treats executable Python code as a medium for enhancing MLLM visual perception in STEM.The paradigm uses code both for caption supervision and for structured understanding.
  • 6. Conclusion: ICC-1M provides large-scale image-caption-code triplets for Code-Grounded Caption Generation and STEM Image-to-Code Translation.These are the two training approaches named in the conclusion.
  • 6. Conclusion: STEM2Code-Eval evaluates perception through executable code generation for image reconstruction rather than traditional problem-solving proxies.The conclusion characterizes executable code as essential for enhancing perception.

Supplementary Material

The supplementary material specifies quality-control procedures for code, rendered images, and image-code consistency, with explicit qualification criteria and verdict standards.

  • Quality Control: Quality control checks generated code, rendered images, and image-code consistency at three critical checkpoints.The framework uses explicit prompts to maintain technical and semantic reliability.
  • Code Quality: High-value samples encode recognizable mathematical structures, algorithms, or domain-standard visual-generation tools.Examples include parabolas, Fibonacci spirals, procedural noise, and pseudo-3D rendering techniques.
  • Code Quality: Low-value samples contain arbitrary operations, magic numbers, inaccessible resources, or structureless random generation.External files, databases, network APIs, and unstructured np.random.rand() outputs are disqualified.
  • Image Quality: Class A rendering errors cause failure when they severely compromise geometry, annotations, data visibility, density, or layout.Minor technical glitches and Class B aesthetic or content flaws must not determine a Fail verdict.
  • Image-Code Consistency: Consistency review compares a visual evidence inventory with a code-derived design blueprint to detect fundamental structural or categorical mismatches.The blueprint includes quantities, coordinates, and connection relationships inferred from variables, loops, and control flow.

2. Python Implementation Code

The implementation constructs a seven-hexagon honeycomb with a dashed orange path using Python and matplotlib, then illustrates perception errors on a complex 8×8 grid image.

  • Implementation: The code uses matplotlib and numpy to generate a hexagonal grid with a path.The implementation is organized into setup, geometry generation, drawing, path construction, labeling, and plot finalization.
  • Spiraling Path: The path is assembled from explicitly defined vertices and intermediate points, then drawn through successive line segments.Vertex markers and numeric labels are added before the plot limits and aspect ratio are finalized.
  • Perception Example: 8×8 grid images with varying dot patterns and 63-arrow paths challenge current MLLMs’ perceptual accuracy.Gemini 2.5 Pro reportedly miscounted nodes, spatial arrangements, and arrows, whereas code-grounded captions provide precise, verifiable information.

1 The Training Details of CodePercept-S1

CodePercept-S1 is trained from Qwen3-VL-Instruct on ICC-1M, while the reported training curves compare S1 and R1 model sizes and show better performance for larger models.

  • Training setup: CodePercept-S1 is built on Qwen3-VL-Instruct and fine-tuned on a curated mixture of ICC-1M data.The dataset derives from public STEM seed data including Mulberry-SFT and GRAMMAR.
  • Training setup: S1 training uses AdamW, DeepSpeed Zero-2, cosine learning-rate decay from 3 × 10−6 to 2 × 10−7, and 5% warm-up steps.Training runs for 1 epoch with an effective batch size of 4 per device.
  • Training curves: Figure 5 reports training curves for both CodePercept-S1 and CodePercept-R1 model series.Panel (a) covers S1 models, while panel (b) covers R1 models.
  • Training curves: Across both training stages, 4B, 8B, and 32B CodePercept models show better performance at larger model sizes.The reported scaling pattern appears in both supervised fine-tuning and reinforcement-learning stages.

E. The Construction Pipeline of STEM2Code-Eval Benchmark

STEM2Code-Eval constructs image-code evaluation examples and uses prompts for image reconstruction, code repair, visual scoring, and iterative re-coding.

  • Image-to-code generation: The image-plus-caption-to-code task asks a Python and Matplotlib programmer to generate standalone code reproducing the input image.The prompt requires precise reproduction from the image and its detailed description.
  • Benchmark construction: STEM2Code-Eval includes 1000 image-code pairs checked and revised by human annotators.The benchmark pipeline is presented in Figure 6.
  • Image-to-code generation: Visual matching requires reproducing coordinate ranges, scales, colors, line styles, element positions, sizes, axes, ticks, and grids when described.The prompt also requires clear, structured code whose logic is reflected in the rendered image.
  • Code repair: A repair prompt receives generated code and an error message, then requires only the complete corrected Python code.The corrected code must be enclosed in a single Python code block.
  • Image scoring: The scoring prompt compares reference and AI-generated images on structural completeness, positional accuracy, text fidelity, visual consistency, and clarity.The evaluation produces comments and a final score out of 100.
  • Re-coding: The re-coding stage receives the original image, current render, and current code to improve scientific-chart reproduction through visual comparison.The supplied prompt frames the task around accurately recreating STEM images.

F. The Evaluation Implementation of STEM2Code-Eval Benchmark

STEM2Code-Eval measures generated outputs through image fidelity, code quality, and execution success, combining multimodal judging, code review, and sandboxed execution checks.

  • Evaluation metrics: The benchmark uses Image Score, Code Score, and Execution Rate to evaluate generated image-code outputs.These metrics cover visual fidelity, code quality, and direct execution success.
  • Evaluation metrics: Image Score uses Gemini2.5-Pro to compare the ground-truth image with the image generated by the model-under-test’s code.The process provides both images to the multimodal evaluator with a crafted evaluation prompt.
  • Evaluation metrics: Code Score uses GPT-4o to assess logical correctness, readability, efficiency, robustness, and overall quality on a 0-to-100 scale.The evaluator scores multiple dimensions before producing an overall quality score.
  • Evaluation criteria: The code evaluation prioritizes final visual consistency, including pixel-level geometric details, element relationships, and visual attributes over implementation elegance.Equivalent rendered images receive equal treatment regardless of coding style.
  • Evaluation criteria: The visual criteria cover layout, canvas attributes, element completeness, coordinates, spatial relationships, connectivity, stacking order, and style.The code criteria additionally assess clarity, correctness, efficiency, and reproducibility.
  • Evaluation metrics: Execution Rate counts samples whose code exits with code 0 and generates an image file in the working directory.Each snippet runs in an isolated sandbox with standard Python data-science libraries.

G. Experiments with 32B

The experiments report consistent CodePercept improvements across six STEM reasoning benchmarks and evaluate STEM2Code-Eval with image, code, and execution metrics.

  • Results: CodePercept obtains consistent improvements in the reported evaluations on public STEM benchmarks and STEM2Code-Eval.The text refers to results in Tables 4 and 5.
  • Results: Table 4 compares various MLLMs across six STEM reasoning benchmarks.The table is used for the public benchmark evaluation.
  • Results: Table 5 evaluates 1k STEM2Code-Eval samples using Image Scoring, Code Scoring, and Exec Rate.The metrics respectively assess visual similarity, generated-code quality, and execution success.
Loading 2603.10757v2…