PaperlayerView exact revision on arXiv

Source-linked AI summary

GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning

GLM-V Team, :, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Haochen Li, Jiale Zhu, Jiali Chen, Jiaxing Xu, Jiazheng Xu, Jing Chen, Jinghao Lin, Jinhao Chen, Jinjiang Wang, Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingdao Liu, Mingde Xu, Mingzhi Zhang, Qinkai Zheng, Ruiliang Lyu, Shangqin Tu, Sheng Yang, Shengbiao Meng, Shi Zhong, Shiyu Huang, Shuyuan Zhao, Siyan Xue, Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianyu Tong, Wei Jia, Wenkai Li, Xiao Liu, Xiaohan Zhang, Xin Lyu, Xinyu Zhang, Xinyue Fan, Xuancheng Huang, Yadong Xue, Yanfeng Wang, Yanling Wang, Yanzi Wang, Yifan An, Yifan Du, Yiheng Huang, Yilin Niu, Yiming Shi, Yu Wang, Yuan Wang, Yuanchang Yue, Yuchen Li, Yusen Liu, Yutao Zhang, Yuting Wang, Yuxuan Zhang, Zhao Xue, Zhengxiao Du, Zhenyu Hou, Zihan Wang, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, Jie Tang

arXiv:2507.01006v6cs.CVcs.AIcs.LG

TL;DR

The paper addresses how to build general-purpose VLMs with stronger multimodal reasoning across increasingly complex tasks. It develops a reasoning-centric training framework combining broad pre-training, supervised fine-tuning, and RLCS-based reinforcement learning. Across 42 benchmarks, GLM-4.5V reaches state-of-the-art performance on nearly all tasks among similar-size open-source models, while the released models support further research.

  • Problem

    VLMs must handle complex reasoning tasks beyond visual perception, but the report focuses on developing general-purpose multimodal reasoning capabilities across diverse domains.

  • Method

    The framework combines knowledge-intensive multimodal pre-training, domain-specific supervised fine-tuning, and Reinforcement Learning with Curriculum Sampling across diverse domains.

  • Results

    Across 42 public benchmarks, GLM-4.5V achieves state-of-the-art performance on nearly all tasks among similar-size open-source models and is comparable or superior to Gemini-2.5-Flash on 22 benchmarks.

  • Takeaways & Limitations

    The open-sourced GLM-V models provide a competitive family for general-purpose multimodal reasoning across visual, textual, mathematical, scientific, and agentic domains.

  • Takeaways & Limitations

    RL can reinforce incorrect or hallucinated reasoning chains because current reward models typically evaluate final outcomes without verifying intermediate steps.

Abstract

from arXiv · show

We present GLM-4.1V-Thinking, GLM-4.5V, and GLM-4.6V, a family of vision-language models (VLMs) designed to advance general-purpose multimodal understanding and reasoning. In this report, we share our key findings in the development of the reasoning-centric training framework. We first develop a capable vision foundation model with significant potential through large-scale pre-training, which arguably sets the upper bound for the final performance. We then propose Reinforcement Learning with Curriculum Sampling (RLCS) to unlock the full potential of the model, leading to comprehensive capability enhancement across a diverse range of tasks, including STEM problem solving, video understanding, content recognition, coding, grounding, GUI-based agents, and long document interpretation. In a comprehensive evaluation across 42 public benchmarks, GLM-4.5V achieves state-of-the-art performance on nearly all tasks among open-source models of similar size, and demonstrates competitive or even superior results compared to closed-source models such as Gemini-2.5-Flash on challenging tasks including Coding and GUI Agents. Meanwhile, the smaller GLM-4.1V-9B-Thinking remains highly competitive-achieving superior results to the much larger Qwen2.5-VL-72B on 29 benchmarks. We open-source both GLM-4.1V-9B-Thinking and GLM-4.5V. We further introduce the GLM-4.6V series, open-source multimodal models with native tool use and a 128K context window. A brief overview is available at https://z.ai/blog/glm-4.6v. Code, models and more information are released at https://github.com/zai-org/GLM-V.

1 Introduction

The report develops a family of vision-language models for increasingly complex multimodal reasoning tasks. Its reasoning-centric framework combines broad multimodal pre-training, supervised fine-tuning, and scalable reinforcement learning, with strong results across diverse benchmarks.

  • VLMs increasingly need advanced reasoning for tasks spanning scientific problem solving and autonomous agents, beyond visual content perception.
  • The training framework combines broad knowledge-intensive multimodal pre-training, domain-specific supervised fine-tuning, and scalable reinforcement learning around a unified reasoning objective.
  • 42 public benchmarks show GLM-4.5V achieves state-of-the-art performance on nearly all tasks among similar-size open-source models.
  • Multi-domain reinforcement learning demonstrates cross-domain generalization and mutual facilitation, with joint training yielding greater improvements across domains.
  • RLCS dynamically selects informative rollout problems, while ratio-based EMA expands sampling to improve reinforcement-learning effectiveness and stability.
  • The models are open-sourced with domain-specific reward systems and support flexible thinking and non-thinking modes in GLM-4.5V and GLM-4.6V.

2 Overview and Architecture

The GLM-V models share a three-part vision-language architecture that converts visual inputs into multimodal token sequences for language decoding. Spatial adaptations support varied image formats, while timestamp tokens provide explicit temporal information for videos.

  • The shared architecture contains a vision encoder, MLP adapter, and large language model decoder for multimodal token processing.
  • 2D-RoPE and interpolated absolute position embeddings enable the vision transformer to process extreme aspect ratios and resolutions beyond 4K.
  • Normalized patch coordinates are sampled from the original position-embedding table with bicubic interpolation to produce resolution-adapted embeddings.
  • 3D-RoPE extends spatial awareness in the language model while preserving its text-related capabilities.
  • Video inputs receive timestamp-based time-index tokens after each frame token, encoding temporal distances for video understanding and grounding.

3 Pre-training

Pre-training builds a broad multimodal foundation from curated image-text, interleaved, OCR, grounding, video, and instruction data. Filtering, recaptioning, and staged training aim to improve knowledge coverage, data quality, and general multimodal capability.

  • GLM-4.1V-9B-Base substantially outperforms a similar-scale state-of-the-art pre-trained base model on non-multiple-choice MathVista pass@k.
  • The corpus includes image-text captions, interleaved image-text documents, OCR images, natural-image and GUI grounding, video-text data, and instruction-tuning examples.
  • Data processing combines heuristic and relevance filtering, concept-balanced resampling, factual-centered recaptioning, and merging original with recaptioned data.
  • Video data are filtered for integrity and multimodal embedding similarity to remove corrupted files and semantic redundancy.
  • 50 million instruction-tuning samples cover visual perception, multimodal reasoning, document contexts, GUI agents, and UI coding for the reinforcement-learning pipeline.
  • Training proceeds through multimodal pre-training followed by long-context continual training, using model-specific parallelism strategies.

4 Supervised Fine-Tuning

Supervised fine-tuning bridges pre-training and reinforcement learning by teaching standardized long-chain-of-thought reasoning across verifiable and open-ended multimodal tasks. The recipe also supports tool use and selectable thinking modes.

  • SFT transforms a base VLM into a model capable of long chain-of-thought inference across verifiable and non-verifiable tasks.
  • The long-CoT dataset trains coherent multi-step solutions in a standardized format to support stable and scalable reinforcement learning.
  • Responses use <think> and <answer> fields, with boxed final answers for verifiable tasks to enable accurate answer extraction during RL.
  • GLM-4.6V uses an explicit XML-style tool-call protocol that decomposes function parameters into argument keys and values.
  • Clean and consistent cold-start data produce more stable RL convergence and higher overall performance, although noisy reasoning data can remain useful.
  • GLM-4.5V and GLM-4.6V mix thinking and non-thinking data, using /nothink to request empty thinking content during inference.

5 Reinforcement Learning: What Is Challenging and What Works

The framework combines multi-domain RL with curated data, unified rewards, and training refinements such as RLCS and dynamic sampling. The authors emphasize that reward quality and rollout selection determine whether training remains effective and stable across multimodal capabilities.

  • Framework components: Large-scale RL combines RLVR and RLHF across STEM, grounding, OCR, video understanding, GUI agents, chart understanding, and other multimodal capabilities.The framework includes data preparation, reward design, training, and infrastructure components.
  • Reward system: A unified reward system shares common evaluation logic while using robust, targeted verifiers for each multimodal subdomain.The reward system supervises perception, comprehension, reasoning, and agent behavior.
  • Reward system: Weak reward signals in one capability can derail unified training, with flawed multi-image QA rewards causing collapse across domains despite high-quality STEM rewards.The authors attribute this failure to unstable or hackable verifiers.
  • Sampling and efficiency: RLCS adaptively matches sample difficulty to evolving model capabilities using offline and online pass@k-based grading combined with human annotations.Samples are partitioned into difficulty tiers and updated with rollout outcomes during training.
  • Sampling and efficiency: Dynamic sampling expansion uses an exponential moving average of invalid-rollout rates to oversample and select batches with balanced correct and incorrect responses.All-correct or all-incorrect batches provide no useful gradient when entropy and KL losses are removed.
  • Empirical findings: Multi-domain RL frequently transfers gains across domains, while proper training can equalize checkpoints whose cold-start scores differ and per-sample loss improves stability.The authors also report that entropy loss caused garbled outputs and eventual collapse, so it was removed.

6 Evaluation

GLM-4.5V and GLM-4.1V-9B-Thinking are evaluated across 42 public benchmarks spanning eight multimodal capability categories. GLM-4.5V leads comparable open-source models broadly, while cross-domain reinforcement learning shows transfer and synergy alongside domain-specific trade-offs.

  • 6.1 Evaluation Setting: 42 public benchmarks cover General VQA, STEM, OCR & Document, Visual Grounding, Spatial Reasoning, GUI Agents, Coding, and Video Understanding.The evaluation uses a common toolchain, policies, and prompt templates across models, with a minimum 95% successful request rate per benchmark.
  • 6.2 Comparison to Other Advanced MLLMs: GLM-4.5V establishes state-of-the-art performance among open-source models across the evaluated multimodal tasks.The comparison includes models such as Step-3, Qwen-VL, Kimi-VL, and Gemma-3.
  • 6.2 Comparison to Other Advanced MLLMs: GLM-4.5V-Thinking leads across General VQA, STEM, chart and document understanding, GUI Agents, multimodal Coding, Video Understanding, and Spatial Reasoning.It achieves the highest results on several named STEM, chart, video, and spatial benchmarks, while remaining solid on Visual Grounding.
  • 6.2 Comparison to Other Advanced MLLMs: 29 out of 42 benchmarks favor the compact GLM-4.1V-9B-Thinking over the much larger Qwen2.5-VL-72B.The reported examples include MMStar, MUIRBENCH, MMMU Pro, and ChartMuseum.
  • 6.3 Investigating Cross-Domain Generalization in Reinforcement Learning: Single-domain reinforcement learning improves performance in other domains, and GUI-agent training improves all evaluated domains.The reported transfer includes STEM training improving grounding, GUI-agent interaction, and general VQA, while OCR & Chart training improves STEM, GUI-agent, and general VQA.
  • 6.3 Investigating Cross-Domain Generalization in Reinforcement Learning: Mix-all training yields greater gains than single-domain reinforcement learning in STEM, OCR & Chart, and general VQA, but not in grounding or GUI-agent performance.The results suggest that grounding and GUI-agent capabilities may require more targeted or specialized multi-domain strategies.

7 Discussion: Limitations and Future Work

The models show strong multimodal reasoning performance, but the report identifies limitations in reasoning verification, RL stability, perception, and evaluation. Future work targets more process-aware rewards, robust training, multimodal transfer, and diagnostic benchmarks.

  • Limitations: Reward models may reinforce incorrect or hallucinated reasoning chains when they evaluate only final answers.The report calls for rewards that assess intermediate reasoning steps as well as outcomes.
  • Limitations: RL training remains sensitive to setup changes, producing substantial variation in reasoning depth or output style.Improved reward design and cold-start data have increased stability, but consistency and robustness remain challenges.
  • Limitations: Complex scenes with clutter, occlusion, or ambiguous visual details can cause perceptual errors that undermine grounded reasoning.Under these conditions, the models may rely on guesswork or generic assumptions.
  • Future Work: Future reward models should detect hallucinations and logical inconsistencies in intermediate reasoning steps.The report also highlights the need to prevent reward hacking on subjectively evaluated tasks.
  • Future Work: Multimodal training may benefit text-only reasoning tasks, including coding tasks involving code presented in images.The report proposes studying how visual and textual reasoning mutually reinforce one another.
  • Future Work: Many current benchmarks are saturating or fail to identify errors such as hallucinated reasoning chains.The report advocates more challenging and diagnostic evaluations targeting shortcut reasoning and hallucination.

8 Contribution

This section identifies the contributors and advisors involved in the work and presents qualitative examples illustrating model capabilities in front-end code generation and video description.

  • 8 Contribution: The contributor list distinguishes core contributors, contributors, tech leads, and academic advisors.Core contributor names are sorted alphabetically by first name.
  • 8 Contribution: The section demonstrates model capabilities through various qualitative examples.The examples are presented as demonstrations of the model’s capabilities and advantages.
  • 8 Contribution: Figure 7 shows a case of generating front-end code from a UI snapshot.The figure is a qualitative example of UI-to-code generation.

A.2 Video Description

The video description depicts two people interacting in a softly lit domestic room while one performs sewing or mending. Their gestures and expressions convey attentive, warm collaboration.

  • Setting & Atmosphere: The scene takes place indoors near white curtains, with a wooden table holding a plaid bag and other small objects.The setting is described as softly lit, calm, and domestic.
  • Characters & Actions: A seated person in a dark blue, green, and yellow striped shirt holds fabric and a needle for a sewing or mending task.The description treats the fabric’s identity as uncertain.
  • Characters & Actions: A standing person in a teal top leans toward the seated person and adjusts their hand position from the shoulders toward the head.The gestures are described as supportive and affectionate.
  • Interaction & Mood: The seated person occasionally glances at the standing person, while later smiling and appearing engaged in conversation.The standing person listens attentively and smiles during the interaction.
  • Interaction & Mood: The overall interaction is characterized as close, collaborative, and mutually supportive in an everyday domestic setting.The description connects the shared craft activity with a warm interpersonal atmosphere.
  • Qualitative Example: Figure 8 presents a case demonstrating detailed and precise video description.The figure caption states the capability without specifying an evaluation result.

A.3 Video Description (in Chinese)

The Chinese video description presents a warm indoor scene in which one person sews at a table while another offers gentle physical attention. The interaction develops through eye contact, touch, and smiling.

  • 场景与环境: The room has translucent white curtains, soft lighting, and a wooden table with clothing, fabric items, and a colorful bag.These elements are described as creating a calm, warm, home-like atmosphere.
  • 人物与互动: A seated person wearing green-and-yellow stripes focuses on sewing with needle and thread while handling clothing.The clothing is tentatively described as possibly baby clothing.
  • 人物与互动: A standing person in a teal sweater leans forward and gently touches the seated person’s shoulder, arm, and later head.The gestures are characterized as caring and intimate.
  • 人物与互动: As the video progresses, the seated person shifts from concentrating on sewing to engaging with the standing person and smiling.The description highlights changing eye contact and facial expressions.
  • 氛围与情感: The video is characterized as portraying companionship and shared daily life through gentle interaction in a stable home setting.The account frames the moment as emotionally warm and intimate.
  • Qualitative Example: Figure 9 presents a case demonstrating detailed and precise video description.The figure caption states the capability without specifying an evaluation result.

A.4 Video QA

The video example shows the model analyzing a visual experiment and explaining why water appears in the middle cup. Its explanation invokes capillary action through the connecting paper towel.

  • Video reasoning: The model identifies the setup as a demonstration of capillary action involving three cups and a paper towel.It describes the paper towel as a porous pathway between the colored liquids.
  • Video reasoning: The response explains that liquid moves through gaps between the paper towel’s fibers, which act like narrow capillary tubes.The explanation attributes the movement to capillary effects and surface tension.
  • Video reasoning: Figure 10 presents video question answering as using perception, knowledge, and reasoning.

A.5 GUI Agent

The examples cover GUI interaction, visual recognition, grounding, spatial reasoning, coding, and multimodal problem solving. They also describe evaluation and prompting procedures for GUI agents, coding, and visual grounding.

  • GUI interaction: The GUI example requires creating a timed calendar event with a specified title, description, date, and duration.The model uses the calendar view and floating action button to interpret the task interface.
  • Multimodal capabilities: The visual examples demonstrate chart understanding, geographic inference, chemistry and mathematics solving, long-document interpretation, and tool use.
  • Coding and GUI capabilities: The examples include integrated code recognition, debugging, and correction alongside GUI recognition and operation.
  • Spatial reasoning: The model reasons about object positions by updating spatial relations after a cup is moved relative to the keyboard and mouse.
  • Coding evaluation: The coding evaluation compares rendered HTML outputs with reference screenshots using GPT-o4-mini as a visual judge.A score of at least 80 counts as a faithful UI2Code reproduction because rendering differences make perfect scores impractical.
  • Visual grounding: Visual grounding prompts ask the model to locate a described object and return its bounding box coordinates.Each box is represented as [x1,y1,x2,y2], using normalized coordinates scaled by 1000.

C Experimental Results of GLM-4.1V-9B-Thinking

GLM-4.1V-9B-Thinking is evaluated against other sub-10B-parameter models across diverse visual-language benchmarks. It achieves leading results on most of the reported benchmarks.

  • Benchmark results: 23 out of 28 benchmarks show GLM-4.1V-9B-Thinking setting a new state-of-the-art among models under 10B parameters.The comparison includes MiMo-VL, Kimi-VL, InternVL3, and Qwen-VL series baselines.
  • Benchmark results: Table 3 compares GLM-4.1V-9B-Thinking with other models on diverse visual-language benchmarks.Bold results identify the best open-source models under 10B parameters; asterisks and daggers indicate reproduced and third-party results.
Loading 2507.01006v6…