GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents

1 Overview

• The coordinated advances include CogViT, Multimodal Multi-Token Prediction, integrated vision-language pre-training and supervised fine-tuning, and joint reinforcement learning across more than 30 task categories.They also include optimized infrastructure for large-scale multimodal RL, expanded toolchains, framework integration, ecosystem development, and ImageMining, a vision-centric deep-search benchmark.Source: §1 · p. 1
• 30.7 on ImageMining, 51.9 on BrowseComp-VL, 72.9 on MMSearch [18], and 78.2 on SimpleVQA [7] demonstrate multimodal tool-use results.The overview also reports effectiveness in practical agentic settings, including chatbot-style environments such as Z.ai and framework-based scenarios such as Claude Code and OpenClaw.Source: §1 · p. 2
• 75.7 on AndroidWorld [30] and 62.3 on OSWorld [44] are GUI-agent results; Claw evaluations report 87.0/80.7 on PinchBench, 57.7/75.0 on ClawEval [46], and 57.6 on ZClawBench.Source: §1 · p. 2
• 94.8 on Design2Code [31] outperforms Claude Opus 4.6; text-only coding preserves GLM-5-Turbo’s base-model capability and surpasses it on CC-Backend (22.8), CC-Frontend (68.4), and CC-RepoExploration (72.2) [49].Source: §1 · pp. 1–2
• The report identifies perception as foundational to higher-level multimodal capability and hierarchical optimization as often more effective for agentic competence than monolithic end-to-end training.It also emphasizes clear task specification, reliable verification, and carefully controlled evaluation for constructing, assessing, and optimizing end-to-end agent tasks.Source: §1 · p. 2

2.1 CogViT Vision Encoder

• CogViT uses two-stage pretraining to balance visual representation learning with cross-modal alignment.Stage one distills semantic and texture features from SigLIP2 [39] and DINOv3 [32] using masked image modeling and a quality-aware data mixture; stage two uses NaFlex [39], a 64K batch with sigmoid-based SigLIP loss, and an 8-billion bilingual image-text corpus.Source: §2.1 · pp. 2–3, Figure 1 · p. 2

2.2 Multimodal Multi-Token Prediction

• GLM-5V-Turbo adopts MMTP’s shared <|image|> token, preserving visual positional information while improving training stability and system scalability.On a 0.5B model, this design achieves lower training loss and more stable convergence than directly passing visual embeddings, while avoiding their propagation across pipeline-parallel stages.Source: §2.2 · p. 3, Figure 2 · p. 3

2.3 Broad training across perception, reasoning, and agent capability

• Pretraining mixes text and multimodal data across knowledge, OCR, coding, GUI, video, tool-use, grounding, and other tasks, with particular emphasis on multimodal coding.The authors aim to strengthen the model’s native processing of multimodal context and align visual understanding with code generation for agentic tasks.Source: §2.3 · p. 4
• Joint RL over more than 30 task categories yields gains across perceptual, reasoning-heavy, and agentic tasks.Compared with SFT, improvements are 4.8% on RefCOCO-avg, 3.2% on PointBench [6], +5.6% on MVBench [24], +7.7% on SUNRGBD, +4.2% on OCRBench [25], +7.7% on CharXiv, +1.8% on MMMU_Val [47], MMMU_Pro [48], MathVista [26] and LogicVista [42], +4.9% on OSWorld [43], +0.2% on CC-Backend [49], and +3.5% on MMSearch [19].Source: §2.3 · p. 4
• Multi-task RL shows weaker cross-domain interference than SFT and can stabilize narrow-distribution optimization through collaborative training and transfer of reasoning strategies.The authors also caution that capabilities underrepresented in RL may decline after post-training, and describe related proxy tasks, such as single-turn UI-to-code, as potentially useful for improving more complex multi-turn coding.Source: §2.3 · p. 4

2.4 Multimodal RL at Scale

• The multimodal RL stack is redesigned around unified tasks and rewards, asynchronous pipeline stages, runtime memory management, and topology-aware partitioning and load balancing.For variable-length visual inputs, partitioning is moved into data loading and combined with joint bin-packing; the system reduces GPU communication buffer overhead by about 7 GB in practice.Source: §2.4 · pp. 4–5

3.1 Multimodal Toolchain Expansion

• Its expanded multimodal toolchain supports long-horizon tasks through repeated switching among search, annotation, screenshotting, and webpage-reading tools, grounding execution in visual understanding.For website reproduction, it can explore screenshots and page interactions before using native UI-to-code capability; it can also crop media assets before embedding them.Source: §3.1 · pp. 5–6
• 30.0 on MMSearch-Plus [35] is nearly an eightfold improvement over GLM-4.6V; GLM-5V-Turbo also scores 51.9 on BrowseComp-VL and 30.7 on ImageMining.The paper states that it matches or exceeds Kimi K-2.5 and Claude Opus 4.6 in these categories.Source: §3.1 · p. 6

3.2 Integration with External Agent Frameworks: Claude Code and AutoClaw

• Claude Code supports terminal and local-file-system work, while AutoClaw provides browser and GUI automation, with GLM-5V-Turbo serving as the vision-language controller in a perception–planning–execution loop.Offloading execution logic to these frameworks lets the model focus on high-dimensional reasoning and autonomous task resolution across digital interfaces.Source: §3.2 · p. 7

3.3 ImageMining: A Self-Collected Vision-Centric Deep Search Benchmark

• ImageMining evaluates “think with image, deep search with image” through agentic visual search, where performance depends on multi-step tool use and precise on-image operations.The benchmark comprises 217 curated cases from seven domains and tests search breadth across sources alongside depth of visual reasoning.Source: §3.3 · p. 7
• Its automated training-data pipeline uses visual transitions between reasoning hops and specialized OCR Search data to encourage image parsing, entity isolation, and localized cropping before search.The pipeline covers knowledge discovery, QA reconstruction, and quality filtering; its “Visual Jump” constraint prevents discovery from relying only on textual shortcuts or parametric knowledge.Source: §3.3 · p. 7

3.4 Multimodal Deep Research and Content Creation

• GLM-5V-Turbo conducts multimodal deep research through iterative gathering, evidence consolidation, and long-form synthesis, reading visual webpages, charts, and documents alongside text.It extracts textual and visual evidence, including table regions and screenshots, to use information distributed across layouts and visual artifacts.Source: §3.4 · pp. 7–8
• The system transforms gathered materials into interleaved text-image reports, slide decks, and document-style write-ups while preserving links between conclusions and visual evidence.Users can provide complex source materials for restructuring into slides or interleaved interpretations.Source: §3.4 · p. 8

3.5 Official Skills

• Official skills support native GLM-5V-Turbo capabilities, expose it as a MaaS API tool for OpenClaw, AutoClaw, and Claude Code, and add five skills based on GLM-OCR and GLM-Image.A unified master skill is provided to help users understand, install, and use the skills.Source: §3.5 · p. 8

Lens 1: Perception remains foundational to higher-level multimodal capability

• Perceptual errors in fine-grained and spatial understanding often propagate into reasoning, decisions, and execution, so perception continues to bound higher-level multimodal capability.Multimodal coding and grounding provided useful perceptual-learning proxies; paired image–SVG data helped downstream STEM problem solving, grounding-focused RL improved GUI-agent performance, and critic data targeting misread details improved GUI observation and reduced recurring perception failures.Source: §4 · p. 9

Lens 2: Agent capability can be more efficiently built through hierarchical optimization

• Hierarchical optimization can develop agent capability more efficiently and stably than concentrating optimization primarily on high-level, long-horizon tasks.The GUI-agent hierarchy spans element perception, grounding, single-step action prediction, and trajectory-level action prediction in both SFT and RL; lower-level tasks are easier to construct, annotate, and verify under limited resources.Source: §4 · pp. 9–10

Lens 3: The key to constructing, evaluating, and optimizing end-to-end long-horizon tasks lies in clear task specification, reliable outcome verification, and controlled evaluation procedures

• End-to-end long-horizon tasks become useful optimization and evaluation targets when specification, outcome verification, and evaluation procedures are jointly made clear and reliable.Open-ended goals and ambiguous execution boundaries make outcomes difficult to compare and reuse as feedback; task definition can draw on multiple constraints, while verification itself needs structure.Source: §4 · p. 10
• Vision2Web [14] instantiates this approach by combining richer task specifications with workflow-based verification for end-to-end visual website development.Tasks may include PRDs, mockups, reference pages, and assets; dependent verification steps support system comparison and failure attribution, while separating functional correctness from visual consistency signals.Source: §4 · p. 10

5 Evaluation

• GLM-5V-Turbo performs strongly across multimodal coding and agent tasks while remaining solid on text-only coding, without materially eroding its underlying coding performance.The evaluation covers multimodal coding, tool use, GUI agents, text-only coding, and Claw benchmarks.Source: §5 · pp. 10–11, Figure 4 · p. 11, Figure 5 · p. 11
• It performs strongly on UI-to-code, visual website development, multimodal search, and visually grounded QA, and is highly competitive on AndroidWorld and WebVoyager GUI-agent benchmarks.These GUI results indicate that visual understanding transfers to grounded interaction and action.Source: §5 · pp. 10–11, Figure 4 · p. 11
• In Claw, one representative vision-enabled agent framework, native on-screen perception and action support strong execution results on PinchBench, ClawEval [46], and ZClawBench, extending gains beyond isolated benchmarks.The model perceives on-screen content and acts on it more effectively when integrated into Claw frameworks.Source: §5 · pp. 10–11

Agentic strategy emergence

• Hand-crafted or filtered cold-start trajectories constrain exploration, while greater trajectory diversity can help RL find nearby improvements without yet enabling autonomous strategy discovery.A deeper challenge is discovering richer structures such as sub-agent decomposition, multi-agent collaboration, and flexible hierarchical decisions.Source: §6 · pp. 11–12

Multimodal context management

• Visual observations, especially video, consume context rapidly, and dropping or text-centric compression can lose details needed for later reasoning, planning, or verification.Long-horizon agents need multimodal-native context and memory that preserve visual details such as layout, spatial relations, and temporal change.Source: §6 · p. 12

Model and harness

• An agent’s effective capability boundary is jointly shaped by its model and harness, whose optimal design shifts as model capabilities evolve.Task decomposition, tool-use policies, memory, and verification workflows affect performance, so apparent model limitations may instead reflect harness choices.Source: §6 · p. 12

• The contributor names are ordered in reverse alphabetical order by first name.Source: §7 · p. 13
• The section lists the core contributors, including Ziyang Pan, Zhen Yang, Yuting Wang, and others.Source: §7 · p. 13
• It also lists additional contributors, tech leads Wenyi Hong and Xiaotao Gu, and academic advisors Peng Zhang, Debing Liu, and others.Source: §7 · p. 13

• The section presents typical qualitative demonstrations from multiple scenarios.Source: A Demo Cases · p. 18

A.1 In Combination with Agent Systems and Skills

• With OpenClaw and the official glmv-stock-analyst3 skill, GLM-5V-Turbo gathers information from multiple sources to produce an NVIDIA stock report.The report includes technical and fundamental analysis, analyst sentiment, and an action plan.Source: Figure 6 · p. 18
• With Claude Code and the official glmv-web-replication4 skill, GLM-5V-Turbo explores a website through a GUI, collects assets, and recreates it in HTML.The requested recreation targets high visual fidelity and functional completeness.Source: Figure 7 · p. 19
• With Claude Code and the official glmv-prd-to-app5 skill, GLM-5V-Turbo uses a PRD and project contents to design and implement a website.The case specifies implementation in the working directory ./act_workspace.Source: Figure 8 · p. 19

Multimodal Coding

• From a high-level design request, GLM-5V-Turbo builds a multi-page e-commerce website with dark-mode styling, interactive buttons, and a dynamic one-page checkout.The example includes a welcome page, shopping interface, parallax brand story, shipping calculation, and address suggestions.Source: Figure 9 · p. 20
• From a mobile mood-tracking app image, GLM-5V-Turbo recreates the interface in executable web code and mocks plausible follow-up pages and interactions in a consistent style.Source: Figure 10 · p. 21
• Given a webpage screenshot, GLM-5V-Turbo reconstructs the page in HTML and retrieves its image assets, combining visual understanding, asset collection, and UI recreation.Source: Figure 11 · p. 21
• GLM-5V-Turbo generates a visually organized research-paper website with interleaved text and figures, and a presentation-ready PowerPoint summarizing a paper’s motivation, method, architecture, and findings.The examples use GLM-5: from Vibe Coding to Agentic Engineering and Attention Is All You Need, respectively.Source: Figure 12 · p. 22, Figure 13 · p. 23

A.3 Multimodal Deep Research

• GLM-5V-Turbo applies image-material collection to an Apple Wearables report, with selected images sourced from the official z.ai website and each image’s original source cited.The request specifies English output, authoritative sources, and high image quality; the case is presented as an illustrated report interleaving images and text.Source: Figure 14 · p. 24

A.4 Document-Based Writing

• Figure 15 illustrates document-based writing with a Chinese Beijing travel guide (103 pages) and accompanying commentary introducing must-visit attractions in Beijing.Source: Figure 15 · p. 25

A.5 OCR and Document Parsing

• Examples show multilingual word recognition and transcription of a physics textbook page, including its text, table, and figures, in Markdown.The OCR example identifies recognized words and their language types.Source: Figure 17 · p. 26, Figure 16 · p. 26

A.6 Visual Search and Reasoning

• The examples use image information and multimodal search tools to answer a novel-related question and compare hotel options by total price for specified dates.The hotel query also requests reviews and experience suggestions for three hotels.Source: Figure 18 · p. 27, Figure 19 · p. 27

A.7 Visual Recognition and Grounding

• Video examples track basketball players per second and objects in a crime-themed video with bounding boxes and globally consistent labels.The basketball prompt requests JSON tracking results; the other prompt requests tracks for described objects.Source: Figure 20 · p. 28, Figure 21 · p. 28
• Grounding and search examples identify people and names in an image and locate GPU circuit-board components for comparison with the H100.The prompts request boxes around the people and components.Source: Figure 22 · p. 28
• Educational-scene examples locate students’ handwritten answers and identify misspelled or incorrectly used words and phrases.The prompts request bounding boxes for answers and identification of writing errors.Source: Figure 23 · p. 29
• The model outputs JSON 3D bounding boxes for furniture or a potted plant, with center coordinates, dimensions in meters, and rotation angles in radians.Each box uses nine values: three center coordinates, three sizes, and three rotation angles.Source: Figure 24 · p. 29

A.8 Spatial Reasoning

• A spatial-reasoning example asks the model to count fingers and mark each finger’s image position in [[x,y]] format.Figure 25 illustrates counting together with positional output.Source: Figure 25 · p. 30