Source-linked AI summary
GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents
GLM-V Team, :, Wenyi Hong, Xiaotao Gu, Ziyang Pan, Zhen Yang, Yuting Wang, Yue Wang, Yuanchang Yue, Yu Wang, Yanling Wang, Yan Wang, Xijun Liu, Wenmeng Yu, Weihan Wang, Wei Li, Shuaiqi Duan, Sheng Yang, Ruiliang Lv, Mingdao Liu, Lihang Pan, Ke Ning, Junhui Ji, Jinjiang Wang, Jing Chen, Jiazheng Xu, Jiale Zhu, Jiale Cheng, Ji Qi, Guobing Gan, Guo Wang, Cong Yao, Zijun Dou, Zihao Zhou, Zihan Wang, Zhiqi Ge, Zhijie Li, Zhenyu Hou, Zhao Xue, Zehui Wang, Zehan Qi, Zehai He, Yutao Zhang, Yusen Liu, Yukuo Cen, Yuchen Li, Yuan Wang, Yu Yang, Yongbin Liu, Yijian Lu, Yifan Xu, Yanzi Wang, Yanxiao Zhao, Yanfeng Wang, Yadong Xue, Yabo Xu, Xinyu Zhang, Xinyu Liu, Xiao Liu, Wenyi Zhao, Wenkai Li, Tianyu Tong, Tianshu Zhang, Shudan Zhang, Shengdong Yan, Qinkai Zheng, Mingde Xu, Licheng Bao, lat Long long, Jiaxing Xu, Jiaxin Fan, Jiawen Qian, Jiali Chen, Jiahui Lin, Jiadai Sun, Haozhi Zheng, Haoran Wang, Haochen Li, Hanyu Lai, Han Xu, Fan Yang, Dan Zhang, Da Yin, Chuangxin Zhao, Chengcheng Wu, Boyan Shi, Bowen Lv, Bowei Jia, Bo Li, Bin Chen, Baoxu Wang, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, Jie Tang
TL;DR
Multimodal agents need to perceive and integrate heterogeneous contexts within reasoning and action, but current systems still face fine-grained perception challenges. GLM-5V-Turbo integrates multimodal perception throughout model design, training, reinforcement learning, and agent infrastructure, achieving strong multimodal agentic performance, including 94.8 on Design2Code.
Problem
General-purpose agents need native processing of heterogeneous multimodal contexts integrated with perception, reasoning, and decision-making.
Method
GLM-5V-Turbo combines a specialized vision encoder, integrated multimodal training, and joint reinforcement learning across more than 30 perception, reasoning, and agentic task categories.
Results
GLM-5V-Turbo achieves strong multimodal agentic and coding performance, including 94.8 on Design2Code, outperforming Claude Opus 4.6.
Takeaways & Limitations
The development process highlights multimodal perception, hierarchical optimization, and reliable end-to-end verification as practical considerations for building multimodal agents.
Takeaways & Limitations
Multimodal context management remains a bottleneck because retaining images and videos over long trajectories consumes context budget and complicates faithful compression.
Abstract
from arXiv · showhide
We present GLM-5V-Turbo, a step toward native foundation models for multimodal agents. As foundation models are increasingly deployed in real environments, agentic capability depends not only on language reasoning, but also on the ability to perceive, interpret, and act over heterogeneous contexts such as images, videos, webpages, documents, GUIs. GLM-5V-Turbo is built around this objective: multimodal perception is integrated as a core component of reasoning, planning, tool use, and execution, rather than as an auxiliary interface to a language model. This report summarizes the main improvements behind GLM-5V-Turbo across model design, multimodal training, reinforcement learning, toolchain expansion, and integration with agent frameworks. These developments lead to strong performance in multimodal coding, visual tool use, and framework-based agentic tasks, while preserving competitive text-only coding capability. More importantly, our development process offers practical insights for building multimodal agents, highlighting the central role of multimodal perception, hierarchical optimization, and reliable end-to-end verification.
1 Overview
GLM-5V-Turbo targets native multimodal agentic capability by integrating multimodal processing into model design, training, and infrastructure. It achieves strong multimodal agentic results while retaining strong text-based agentic and coding performance relative to GLM-5-Turbo, and highlights perception, hierarchical optimization, verification, and controlled evaluation as key development principles.
- Motivation: The overview motivates general-purpose agentic models that combine advanced intelligence with native processing of complex multimodal context for real-world interaction and productivity-oriented domains [49].Such applications include knowledge work, software engineering [20], and graphical-user-interface tasks [16] [43].
- Model design and training: GLM-5V-Turbo develops native multimodal modeling through CogViT, a vision encoder for fine-grained understanding, and Multimodal Multi-Token Prediction for text-only and multimodal inputs.The prediction approach is designed to remain friendly to large-scale infrastructure, while vision and language are integrated throughout pre-training and super...
- Results: The coordinated advances give GLM-5V-Turbo native multimodal agentic capability while retaining strong text-based agentic and coding performance relative to the language-only GLM-5-Turbo.These capabilities are reflected in both benchmark results and practical agentic settings, including Z.ai, Claude Code, and OpenClaw.
- Results: GLM-5V-Turbo achieves strong multimodal agentic benchmark results, including 30.7 on ImageMining, 51.9 on BrowseComp-VL, 72.9 on MMSearch [18], 78.2 on SimpleVQA [7], 75.7 on AndroidWorld [30], and 62.3 on OSWorld [44].It also scores 87.0/80.7 on PinchBench, 57.7/75.0 on ClawEval [46], and 57.6 on a further Claw-based evaluation, though the supplied passage is truncated after that value.
- Lessons: The development process identifies perception as foundational, hierarchical optimization as preferable to monolithic end-to-end training, and clear specifications, reliable verification, and controlled evaluation as essential for end-to-end agent tasks.These principles support effective construction, assessment, and optimization of agentic systems.
2 Model, Training, and Infrastructure
GLM-5V-Turbo integrates multimodal perception into model design, training, reinforcement learning, and infrastructure to support efficient, scalable agentic capability. Its joint multimodal optimization improves several perceptual and agentic skills, while revealing risks of degradation outside the trained task distribution.
- Model Design: CogViT is a parameter-efficient vision encoder with strong general, fine-grained, geometric, and spatial perception, trained through two-stage representation learning and cross-modal alignment.Figure 1 compares CogViT with other state-of-the-art vision encoders across general and fine-grained multimodal tasks.
- Model Design: MMTP supports text-only and multimodal inputs, and the adopted <|image|>-based design lowers training loss and improves convergence stability while reducing pipeline-parallel communication.The design avoids propagating visual embeddings across pipeline-parallel stages and improves system scalability and engineering maintainability.
- Multimodal Training and RL: RL over more than 30 task categories improves grounding and pointing by 4.8% on RefCOCO-avg and 3.2% on PointBench versus SFT.The training also uses improvements such as relative visual policy optimization for UI-to-code tasks [45].
- Multimodal Training and RL: Joint multi-task RL generally reduces cross-domain interference, allowing multiple domains to improve together with stable gains compared with trade-offs often observed in SFT.Collaborative training can also stabilize optimization in narrower-distribution domains where single-task RL often oscillates.
- Multimodal Training and RL: Capabilities outside the RL task distribution can decline after post-training as model capacity and learned thinking patterns concentrate on sampled tasks.This limitation especially affects abilities that are more orthogonal to the trained task distribution.
- Training Infrastructure: The training stack unifies heterogeneous tasks and rewards while decoupling pipeline stages, managing multimodal memory, and dynamically balancing visual-input workloads.The infrastructure includes a unified VLM RL Gym, centralized verifier orchestration, asynchronous stage overlap, targeted recomputation with CPU offloading, and topology-aware partitioning.
3 Multimodal Agent Capabilities and Ecosystem
GLM-5V-Turbo extends multimodal agents from isolated tool use toward complete perception–planning–execution loops across visual search, coding, research, and digital interfaces. It combines expanded multimodal tooling, agent-framework integration, and document-grounded generation to support autonomous multimodal task resolution.
- Multimodal Agent Capabilities: GLM-5V-Turbo supports long-horizon perception–planning–execution by switching among multimodal search, annotation, screenshotting, and webpage-reading tools.The expanded toolchain enables agents to interpret visual environments, choose actions, and adapt based on outcomes.
- Multimodal Agent Capabilities: MMSearch-Plus reaches 30.0, nearly eightfold above GLM-4.6V, while BrowseComp-VL reaches 51.9 and ImageMining reaches 30.7.These results demonstrate gains on complex multimodal tasks involving web navigation and visual mining.
- Agent Framework Integration: GLM-5V-Turbo integrates with Claude Code and AutoClaw, delegating execution logic while serving as a multimodal reasoning core for autonomous tasks across digital interfaces.This integration shifts the model from passive code generation toward active system-level collaboration.
- ImageMining: ImageMining evaluates “think with image, deep search with image” through agentic visual mining rather than traditional VQA [19] [35].Its 217 cases span seven domains and require multi-step operations such as localized cropping or magnification, with performance tied to precise on-image tool use.
- Multimodal Deep Research: The model performs multimodal deep research by iteratively gathering information, consolidating evidence, updating state, and generating interleaved reports, slide decks, and document-style write-ups.It extracts textual and visual evidence together from webpages, charts, structured documents, and screenshots, preserving their connection in downstream outputs.
4 Design Lenses from Development
The development process yielded three practical, non-universal design lenses: perception underpins higher-level multimodal capability, agent training benefits from hierarchical optimization, and long-horizon tasks require clear specification, reliable verification, and controlled evaluation. Together, these perspectives emphasize improving perceptual foundations, distributing optimization across capability levels, and making end-to-end tasks stable enough to provide reusable feedback.
- Lens 1: Perception remains foundational: Perception remains foundational because fine-grained and spatial errors can propagate into multimodal reasoning, decision-making, and execution.The authors frame this as a recurring development observation rather than a universal rule.
- Lens 1: Perception remains foundational: Multimodal coding, grounding, and self-critique training were used to strengthen perceptual learning and reduce recurring GUI-perception failures.Frontend or SVG coding exercises layout, structure, relative position, and local detail; critic data targets misreading interfaces, misidentifying elements, and incorrect next actions.
- Lens 2: Hierarchical optimization: Agent capability can be built more efficiently through hierarchical optimization across element perception, GUI grounding, single-step actions, and trajectory-level actions.This strategy addresses costly environments and data, difficult long-horizon compositions, non-unique solution paths, and dependence on evolving environment state.
- Lens 3: Reliable end-to-end task design: End-to-end long-horizon tasks require clear specifications, reliable outcome verification, and controlled evaluation to produce stable, reusable optimization feedback.Realistic tasks are often open-ended, underspecified, ambiguously bounded, and dependent on intermediate decisions; Vision2Web [14] operationalizes richer task specifications and workflow-based verification.
5 Evaluation
GLM-5V-Turbo performs strongly across multimodal coding, tool-use, and GUI-agent benchmarks while maintaining solid text-only coding capability. Its vision-enabled abilities also transfer effectively to general agent frameworks and execution-oriented evaluations.
- Evaluation Scope: The evaluation spans multimodal coding, multimodal tool use, GUI agents, and text-only coding and Claw tasks.Benchmarks include Design2Code, Flame-VLM-Code, Vision2Web, BrowseComp-VL, MMSearch, AndroidWorld, WebVoyager, CC-Bench-V2, PinchBench, ClawEval, and ZClawBench.
- Multimodal Coding, ToolUse, and GUI Agent: GLM-5V-Turbo achieves strong multimodal results across coding, tool-use, and GUI-agent benchmarks, including UI-to-code, visual website development, multimodal search, visually grounded QA, and grounded interaction.The evaluation covers multimodal coding, multimodal tool use, and GUI-agent tasks.
- Text-only Coding and Claw: The model remains solid on text-only coding benchmarks, suggesting that adding visual capability does not materially erode its underlying coding performance.CC-Bench-V2 evaluates Claude Code performance across CC-Backend, CC-Frontend, and CC-Repo-Exploration.
- Text-only Coding and Claw: GLM-5V-Turbo transfers effectively to vision-enabled agent frameworks, natively perceiving on-screen content and acting on it more effectively in execution-oriented evaluations.Strong results are reported on PinchBench, ClawEval, and ZClawBench within Claw agent frameworks.
6 Remaining Challenges
The main remaining challenges are enabling genuine agentic strategy emergence, managing long-horizon multimodal context, and understanding how model capability is jointly shaped by the surrounding harness. These problems concern exploration, information retention, and system-level design rather than isolated capability improvement.
- Overall challenges: Future agentic model development therefore requires solving interconnected problems in strategy emergence, long-horizon multimodal context management, and model–harness design.These challenges are presented as more central than isolated capability improvement.
- Strategy emergence: Agentic development must move beyond local improvement on familiar reasoning and action paths toward the emergence of genuinely better strategies.Hand-crafted or strongly filtered cold-start trajectories narrow exploration; greater trajectory diversity can help reinforcement learning discover nearby improved variants and may be a condition for strategy emergence.
- Multimodal context management: Long-horizon agents struggle to retain multimodal context because images, especially videos, consume context budget more aggressively than text.Dropping earlier visual observations as context grows saves budget but can discard information needed for later reasoning, planning, or verification.
- Model–harness interaction: Agent capability is increasingly co-shaped by the model and its harness, including task decomposition, tool use, memory mechanisms, and verification loops.This expands the design space but makes development substantially more complex because system behavior depends on the surrounding harness as well as the model.
7 Contribution
The section acknowledges the contributors, tech leads, and academic advisors involved in the work. Contributors’ names are listed in reverse alphabetical order by first name.
- Contributors are listed in reverse alphabetical order, from Z to A, by first name.
- The core contributors include Ziyang Pan, Zhen Yang, Yuting Wang, Yue Wang, and additional listed contributors.
- Wenyi Hong and Xiaotao Gu are identified as tech leads, while Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, and Jie Tang are academic advisors.
A Demo Cases
This section demonstrates GLM-5V-Turbo’s capabilities and advantages through typical qualitative examples across various scenarios.
- A Demo Cases: GLM-5V-Turbo’s capabilities are illustrated through typical qualitative examples.
- A Demo Cases: The demonstrations span various scenarios, showing the model’s advantages qualitatively.
A.1 In Combination with Agent Systems and Skills
GLM-5V-Turbo is demonstrated in agent systems and skills across stock analysis, GUI-based webpage replication, and PRD-driven website generation. These cases show the model gathering information, collecting assets, producing reports, and implementing websites through integrated agent workflows.
- Stock analysis: GLM-5V-Turbo combines OpenClaw with the glmv-stock-analyst skill to gather multi-source information and produce a professional NVIDIA stock analysis report.The report includes technical analysis, fundamental analysis, analyst sentiment, and an action plan.
- Webpage replication: With Claude Code and the glmv-web-replication skill, GLM-5V-Turbo explores a target website through a GUI, collects assets, and recreates the page in HTML.The task specifies high visual fidelity and functional completeness.
- PRD-driven website generation: With Claude Code and the glmv-prd-to-app skill, GLM-5V-Turbo uses a product requirements document and project contents to design and implement a website in a working directory.The demonstrated task builds the project under ./act_workspace from the PRD and act-folder contents.
A.2 Multimodal Coding
GLM-5V-Turbo demonstrates multimodal coding across full-stack website creation, visual interface reconstruction, agentic asset retrieval, and research-content website or slide generation. These examples show the model translating high-level requests or visual references into executable, visually organized interfaces and presentations.
- Full-stack website generation: The model generates complete full-stack e-commerce websites with multiple functional pages, dark mode, parallax scrolling, and dynamic checkout features.The demonstrated site includes welcome, shopping, brand-story, and one-page checkout interfaces with shipping calculation and address suggestion.
- UI recreation: Given a reference image, GLM-5V-Turbo reconstructs a mobile mood-tracking interface in executable web code and extends it with stylistically consistent pages and interactions.The example covers both interface recreation and plausible follow-up page or interaction mockups.
- Agentic UI recreation: In an agentic UI-recreation task, the model jointly understands a screenshot, retrieves its image assets, and reconstructs the webpage in HTML.This demonstrates coordinated visual understanding, asset collection, and faithful interface recreation.
- Research-content generation: The model also turns research papers into visually organized English websites and presentation-ready PowerPoint slide decks with interleaved text and figures.The examples summarize motivation, methods or architecture, system design, and key findings for GLM-5 and Attention Is All You Need.
A.3 Multimodal Deep Research
GLM-5V-Turbo is demonstrated collecting authoritative, high-quality image assets for an Apple Wearables feature report and presenting them in an English illustrated format with interleaved images and text.
- A.3 Multimodal Deep Research: GLM-5V-Turbo collects image assets for an Apple Wearables feature report through the official z.ai website.The example cites the original source for each selected image.
- A.3 Multimodal Deep Research: The workflow requires authoritative sources and high image quality for the collected assets.Each chosen image includes a citation to its original source.
- A.3 Multimodal Deep Research: The output is an English illustrated report that interleaves images with explanatory text.These formatting and language requirements are part of the demonstrated query.
A.4 Document-Based Writing … A.8 Spatial Reasoning
The examples demonstrate document-based writing, multilingual OCR and transcription, multimodal visual search, video tracking, visual grounding, and spatial reasoning across varied inputs and output formats.
- A.4 Document-Based Writing: The model reads a 103-page Chinese Beijing travel guide and writes commentary summarizing ten must-visit attractions for foreigners.
- A.5 OCR and Document Parsing: The model recognizes words and their languages in images and transcribes textbook pages containing text, tables, and figures into Markdown.
- A.6 Visual Search and Reasoning: The model combines image information with multimodal search tools to answer complex questions and rank hotels by total price, reviews, and experience suggestions.
- A.7 Visual Recognition and Grounding: The model tracks basketball players and described objects through videos, producing per-second results with bounding boxes and globally consistent labels.
- A.7 Visual Recognition and Grounding: The model grounds people, names, handwritten answers, writing errors, and circuit-board components, including search-based component identification and comparison with the H100.
- A.7 Visual Recognition and Grounding: The model outputs 3D furniture and plant bounding boxes in JSON using nine values for centers, sizes, and rotation angles.
- A.8 Spatial Reasoning: The model performs spatial counting and localization by counting fingers and marking every finger position in [[x,y]] format.