Source-linked AI summary
Multimodal Thinking with Renderable Programs
Sunli Chen, Ding Zhong, Ziqiao Ma, Jiaxin Liu, Zeyuan Yang, Hao Zhang, Lie Lu, Joyce Chai, Chuang Gan
TL;DR
VLMs have difficulty with structured visual documents and with grounding intermediate reasoning in visual evidence, while raster or latent representations can limit tractable visual editing. SVGLM incorporates renderable SVG programs into VLM reasoning and reports stronger performance than standard fine-tuned VLMs and thinking-with-images baselines. The authors note that resource constraints prevented validation with larger models and larger SVG datasets.
Problem
VLMs struggle with infographics and structured visual documents, and their intermediate rationales can be weakly grounded in visual evidence.
Source: §1 · p. 1Method
SVGLM integrates editable, renderable SVG programs into VLM reasoning and provides an 8K-sample SVG dataset for fine-tuning open-source models.
Source: §1 · pp. 2–3Results
SVGLM outperforms thinking-with-images baselines and standard fine-tuned VLMs, with an 8% boost over zero-shot and 6.5% over direct SFT.
Source: §4.4 · p. 9, §5 · p. 10Takeaways & Limitations
The paper concludes that SVG can support multimodal reasoning and may suit document QA and grounded reasoning tasks.
Source: §5 · p. 10Limited compute and paid API access prevented verification of results with larger models and larger SVG datasets.
Source: §5 · p. 11
Abstract
from arXiv · showhide
Current vision-language models (VLMs) excel at visual content understanding and text-based reasoning, yet their structure limits the advancement of incorporating images into the reasoning chain. Though Omnimodal models have made efforts in unifying text and image generation, they focus on visual tasks in the open-domain, lacking tractability due to rasterized or latent representations of images. We introduce SVGLM, a framework that uses scalable vector graphics (SVG) primitives to connect text and image in reasoning tasks. We exploit the duality of SVG as both image description and text instructions, yielding a more compact, interpretable solution to equip general VLMs with the capability of generating images within the reasoning process. We provide a large curated dataset of SVG-based image editing dataset, as well as the paradigm to tune open-source VLMs. Experiments on a mathematical reasoning benchmark demonstrate that SVGLM achieves strong SVG generation power as well as think-with-image intelligence. Our results highlight SVG as a suitable medium for building more robust digital domain agents, bridging the gap between text-based thinking and pixel-based images.
1 Introduction
VLMs struggle with structured visual documents and can produce rationales weakly grounded in visual evidence. SVGLM uses editable SVG programs as an explicit visual reasoning medium and introduces an 8K-sample dataset and tuning scheme.
1 Introduction
- VLMs struggle with infographics and structured visual documents, while fluent rationales may remain weakly grounded in visual evidence.This is especially acute when a task requires an explicit visual operation, such as adding an auxiliary geometry line.Source: §1 · pp. 1–2
- Visual workspaces can externalize intermediate steps such as auxiliary lines, marks, or plots, supporting interpretability and step-by-step correction.The paper frames explicit visual rationales as useful in geometric and mathematical settings.Source: §1 · p. 2
- Raster or latent images can be token-inefficient, while imperative Python editing can depend on brittle software stacks and runtime environments.The paper presents SVG as a declarative, hierarchical, resolution-independent alternative whose structured primitives make modifications editable and verifiable by rendering.Source: §1 · p. 2
- SVGLM introduces SVG-enhanced multimodal reasoning, an 8K-sample SVG image-editing dataset, and a scheme for integrating SVG into VLM thinking.The paper also reports improved performance on downstream tasks with proper SVG integration.Source: §1 · pp. 2–3
2 Related Work
Prior work explores structured graphics programs, visual prompts as reasoning steps, and code-generated visual reasoning. This section contrasts SVG's editable, explicit representation with raster or latent images and imperative code.
2.1 Visual Representation by Graphics Programs
- SVG offers a declarative, hierarchical, resolution-independent representation in which models manipulate high-level primitives directly through structured text.The text contrasts this with rasterized pixels and imperative code, emphasizing edits that are grounded, verifiable, and editable.Source: §2.1 · p. 3
2.2 Visual Prompts and Thinking with Images
- Visual-prompt methods make auxiliary lines, marks, or plots explicit reasoning steps, enabling step-by-step verification and correction.The section contrasts these explicit visual rationales with approaches that internalize visual manipulation in latent space.Source: §2.2 · p. 3
2.3 Visual Reasoning with Code
- Python-based visual reasoning can hide geometric details in procedural logic, whereas SVG states geometric attributes explicitly in the text.This contrast links the symbolic program more directly to the resulting spatial layout.Source: §2.3 · p. 3
3 Method
SVGLM treats SVG as structured text that can be rendered into intermediate visual states within multimodal reasoning. The method combines tool-calling interactions with an 8K-example geometry dataset curated and checked for visual accuracy.
3.1 Scalable Vector Graphics as Programs
- SVG encodes grounded, editable, and semantically meaningful visual entities, and its deterministic rendering returns an image to the agent during multimodal interaction.SVG entities can be referenced in the rendered image and manipulated at the XML-tree level.Source: §3.1 · p. 4
3.2 Thinking with Renderable Programs
- SVGLM lets a model make normal text generation before calling a rendering tool with XML-encoded SVG and a canvas specification, then either render again or answer.The canvas can be the question image or a blank image; the pipeline is paired with supervised fine-tuning on SVG tool-calling conversations.Source: §3.2 · p. 5
3.3 Data Curation
- The dataset contains 8,000 SVG examples for auxiliary geometry, filtered for constructible edits and annotated with keypoint coordinates on a 1000 × 1000 canvas.Gemini generates SVG annotations, GPT assesses rendered synthesis, and the conversation data includes no, concise, or complete reasoning.Source: §3.3 · pp. 5–6
- Only 1% of evaluated samples had structural errors in human assessment of the collected SVGs.The evaluation checked whether the SVG reflected the image shift and whether its keypoints were accurate.Source: §3.3 · pp. 5–6
4 Experiments
SVGLM improves mathematical visual reasoning over zero-shot, direct-SFT, and thinking-with-images baselines, while its experiments examine SVG generation, grounded reasoning, and framework components. Evaluation uses image-containing geometry questions and tests three open-source models under multiple training settings.
4.1 Dataset and Metric
- Evaluation uses 1,244 image-containing question-answer pairs, 78.8% of the selected geometry samples, scored by category and weighted overall.Later sub-questions receive weights 30% larger than the preceding sub-question.Source: §4.1 · p. 6
4.2 Evaluation setting
- Three roughly similarly sized open-source models are evaluated with zero-shot, direct-SFT, and SVGLM settings in an interactive tool-calling setup.Direct SFT uses MathCanvas-Instruct questions; SVGLM fine-tunes on the SVG-enhanced conversation dataset.Source: §4.2 · pp. 6–7
4.3 Generated SVGs
- Fine-tuned models generate varied useful SVG constructions, while Qwen-Image-Edit and Nano Banana show hallucination and domain misalignment on the same prompt.Observed constructions include connecting points, drawing parallel or perpendicular lines, marking intersections, and indexing angles.Source: §4.3 · p. 7, Figure 3 · p. 8
4.4 Results
- SVGLM improves weighted benchmark performance by 8% over zero-shot and 6.5% over direct SFT, and beats GPT-4o and two thinking-with-images baselines.Across all three selected base models, SVGLM achieves better benchmark results.Source: §4.4 · pp. 7, 9
4.5 Generalization to grounded reasoning
- On POPE grounded reasoning, all three models improve over vanilla SFT after training with SVGs that render bounding boxes over image objects.The study uses 30,000 Visual Genome samples to reproduce the SVGLM training process.Source: §4.5 · p. 9, Table 4 · p. 9
4.6 Ablation studies
- Ablations find that solution images simplify zero-shot questions, rendered SVG feedback further improves results, and full SVGLM scores highest across models.The comparison varies SVG generation and rendering components; the reasoning-density ablation also reports that each tested density yields a significant increase in weighted accuracy.Source: §4.6 · pp. 9–10, Table 5 · p. 10, Table 6 · p. 10
5 Conclusion
SVGLM combines renderable-program reasoning with an 8K question–SVG fine-tuning dataset, and the authors report stronger performance than other thinking-with-images baselines and standard fine-tuned VLMs. They also note that results were not verified at larger model or dataset scales.
5 Conclusion
- SVGLM pairs renderable-program reasoning with an 8K question–SVG dataset and outperforms other thinking-with-images baselines and standard fine-tuned VLMs.The authors also report that proper tool-calling integration lets agents with weak reasoning abilities learn to think with vector graphics.Source: §5 · p. 10
- Success in math and spatial reasoning supports SVGLM’s potential to boost smaller models’ reasoning abilities.Source: §5 · p. 11
- Results remain unverified for larger models and larger SVG datasets because of limited compute resources and paid API requests.Source: §5 · p. 11
A.1 Prompt design for SVG data generation
Figure 4 shows the template used for the SVG generation task and identifies variables to replace in cyan.
A.1 Prompt design for SVG data generation
- Figure 4 provides the SVG generation template, with cyan highlighting variables to replace.Source: Figure 4 · p. 15
B Training setup
Training used three open-source VLMs with full fine-tuning of language-model parameters while freezing vision towers and multimodal projectors; Figure 5 is the SVG quality-assessment template.
B Training setup
- Training used LLaVa-Next-Mistral-7B, Qwen2.5-VL-7B-Instruct, and InternVL-3-8B, with vision towers and multimodal projectors frozen.Source: B Training setup · p. 15
- Models were trained for three epochs with a 10^-5 learning rate and effective batch size 8 on eight Nvidia H100 GPUs.The setup used full fine-tuning of language-model parameters and a cosine learning-rate scheduler.Source: B Training setup · p. 15
- Figure 5 shows the template used for SVG quality assessment.Source: Figure 5 · p. 16