PaperlayerView exact revision on arXiv

Source-linked AI summary

Seed1.5-VL Technical Report

Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, Jingji Chen, Jingjia Huang, Kang Lei, Liping Yuan, Lishu Luo, Pengfei Liu, Qinghao Ye, Rui Qian, Shen Yan, Shixiong Zhao, Shuai Peng, Shuangye Li, Sihang Yuan, Sijin Wu, Tianheng Cheng, Weiwei Liu, Wenqian Wang, Xianhan Zeng, Xiao Liu, Xiaobo Qin, Xiaohan Ding, Xiaojun Xiao, Xiaoying Zhang, Xuanwei Zhang, Xuehan Xiong, Yanghua Peng, Yangrui Chen, Yanwei Li, Yanxu Hu, Yi Lin, Yiyuan Hu, Yiyuan Zhang, Youbin Wu, Yu Li, Yudong Liu, Yue Ling, Yujia Qin, Zanbo Wang, Zhiwu He, Aoxue Zhang, Bairen Yi, Bencheng Liao, Can Huang, Can Zhang, Chaorui Deng, Chaoyi Deng, Cheng Lin, Cheng Yuan, Chenggang Li, Chenhui Gou, Chenwei Lou, Chengzhi Wei, Chundian Liu, Chunyuan Li, Deyao Zhu, Donghong Zhong, Feng Li, Feng Zhang, Gang Wu, Guodong Li, Guohong Xiao, Haibin Lin, Haihua Yang, Haoming Wang, Heng Ji, Hongxiang Hao, Hui Shen, Huixia Li, Jiahao Li, Jialong Wu, Jianhua Zhu, Jianpeng Jiao, Jiashi Feng, Jiaze Chen, Jianhui Duan, Jihao Liu, Jin Zeng, Jingqun Tang, Jingyu Sun, Joya Chen, Jun Long, Junda Feng, Junfeng Zhan, Junjie Fang, Junting Lu, Kai Hua, Kai Liu, Kai Shen, Kaiyuan Zhang, Ke Shen, Ke Wang, Keyu Pan, Kun Zhang, Kunchang Li, Lanxin Li, Lei Li, Lei Shi, Li Han, Liang Xiang, Liangqiang Chen, Lin Chen, Lin Li, Lin Yan, Liying Chi, Longxiang Liu, Mengfei Du, Mingxuan Wang, Ningxin Pan, Peibin Chen, Pengfei Chen, Pengfei Wu, Qingqing Yuan, Qingyao Shuai, Qiuyan Tao, Renjie Zheng, Renrui Zhang, Ru Zhang, Rui Wang, Rui Yang, Rui Zhao, Shaoqiang Xu, Shihao Liang, Shipeng Yan, Shu Zhong, Shuaishuai Cao, Shuangzhi Wu, Shufan Liu, Shuhan Chang, Songhua Cai, Tenglong Ao, Tianhao Yang, Tingting Zhang, Wanjun Zhong, Wei Jia, Wei Weng, Weihao Yu, Wenhao Huang, Wenjia Zhu, Wenli Yang, Wenzhi Wang, Xiang Long, XiangRui Yin, Xiao Li, Xiaolei Zhu, Xiaoying Jia, Xijin Zhang, Xin Liu, Xinchen Zhang, Xinyu Yang, Xiongcai Luo, Xiuli Chen, Xuantong Zhong, Xuefeng Xiao, Xujing Li, Yan Wu, Yawei Wen, Yifan Du, Yihao Zhang, Yining Ye, Yonghui Wu, Yu Liu, Yu Yue, Yufeng Zhou, Yufeng Yuan, Yuhang Xu, Yuhong Yang, Yun Zhang, Yunhao Fang, Yuntao Li, Yurui Ren, Yuwen Xiong, Zehua Hong, Zehua Wang, Zewei Sun, Zeyu Wang, Zhao Cai, Zhaoyue Zha, Zhecheng An, Zhehui Zhao, Zhengzhuo Xu, Zhipeng Chen, Zhiyong Wu, Zhuofan Zheng, Zihao Wang, Zilong Huang, Ziyu Zhu, Zuquan Song

arXiv:2505.07062v1cs.CVcs.AI

TL;DR

Seed1.5-VL addresses gaps in general-purpose multimodal understanding by developing a vision-language model with diversified data synthesis, large-scale multimodal training, and scalable training innovations. It achieves state-of-the-art performance on 38 of 60 public benchmarks and demonstrates integrated reasoning and agent capabilities, while remaining limited in fine-grained visual perception and complex reasoning.

  • Problem

    Current VLMs lack human-level generality, especially for spatial understanding, counting, visual inference, gameplay, and low-level perceptual concepts with limited high-quality annotations.

  • Method

    Seed1.5-VL combines diversified data synthesis, trillion-token multimodal pre-training, human feedback, verifiable rewards, and asymmetric-training optimizations.

  • Results

    38 of 60 evaluated public benchmarks achieve state-of-the-art results, alongside reported capabilities in visual reasoning, OCR, grounding, 3D spatial understanding, video understanding, computer interaction, and gameplay.

  • Takeaways & Limitations

    Seed1.5-VL demonstrates broad multimodal reasoning, perception, and interaction capabilities across benchmark and dissimilar-task settings.

  • Takeaways & Limitations

    Seed1.5-VL struggles with fine-grained perception, including irregular or occluded object counting, subtle image differences, and complex spatial relationships.

Abstract

from arXiv · show

We present Seed1.5-VL, a vision-language foundation model designed to advance general-purpose multimodal understanding and reasoning. Seed1.5-VL is composed with a 532M-parameter vision encoder and a Mixture-of-Experts (MoE) LLM of 20B active parameters. Despite its relatively compact architecture, it delivers strong performance across a wide spectrum of public VLM benchmarks and internal evaluation suites, achieving the state-of-the-art performance on 38 out of 60 public benchmarks. Moreover, in agent-centric tasks such as GUI control and gameplay, Seed1.5-VL outperforms leading multimodal systems, including OpenAI CUA and Claude 3.7. Beyond visual and video understanding, it also demonstrates strong reasoning abilities, making it particularly effective for multimodal reasoning challenges such as visual puzzles. We believe these capabilities will empower broader applications across diverse tasks. In this report, we mainly provide a comprehensive review of our experiences in building Seed1.5-VL across model design, data construction, and training at various stages, hoping that this report can inspire further research. Seed1.5-VL is now accessible at https://www.volcengine.com/ (Volcano Engine Model ID: doubao-1-5-thinking-vision-pro-250428)

1 Introduction

Seed1.5-VL targets gaps in general-purpose multimodal understanding by combining diversified data synthesis, large-scale multimodal pre-training, and preference- and reward-based post-training. It achieves broad benchmark coverage while maintaining a compact architecture intended for efficient interactive use.

  • Motivation: Current VLMs remain limited in 3D spatial understanding, counting, imaginative visual inference, and interactive gameplay, partly because rich vision-language annotations are scarce.Heterogeneous multimodal data also complicates training pipelines, parallelism, and evaluation.
  • Approach: Seed1.5-VL uses diversified synthesis pipelines for OCR, visual grounding, counting, video understanding, long-tail knowledge, visual puzzles, and games.The report also describes training and infrastructure methods developed across multiple stages.
  • Evaluation: 38 of 60 public benchmarks achieve state-of-the-art performance, including 21 of 34 vision-language, 14 of 19 video, and 3 of 7 GUI-agent benchmarks.The evaluation spans visual reasoning, grounding, counting, video understanding, and computer usage.
  • Efficiency: 532 million vision-encoder parameters and 20 billion active language-model parameters support a compact design aimed at reducing inference costs and computational demands.The report connects this efficiency with interactive applications, API access, and smoother chatbot use.
  • Report scope: The report covers architecture, image and video encoding, data curation, pre-training, post-training, scalable infrastructure, evaluation, limitations, and future research directions.Its stated purpose is to share development experience across model design, data construction, and training.

2 Architecture

Seed1.5-VL combines a native-resolution vision encoder with an MLP adapter and LLM, then pre-trains the encoder through masked modeling, contrastive learning, and omni-modal learning. Its video pathway dynamically allocates temporal sampling and spatial resolution to balance information and computation.

  • Architecture: The model consists of a vision encoder, MLP adapter, and LLM, with image patches converted into visual tokens for multimodal processing.Input images are resized to multiples of 28 × 28 pixels, segmented into 14 × 14 patches, and concatenated across images.
  • Vision encoder: Fixed-resolution vision encoders can discard fine-grained information in high-resolution images, videos, and OCR tasks.Prior adaptations alleviate this limitation partially but still rely largely on fixed-resolution architectures and position-encoding changes.
  • Vision encoder: Seed-ViT is a 532-million-parameter encoder designed for native-resolution feature extraction and achieves zero-shot classification performance comparable to the 6-billion-parameter InternVL-C.The architecture includes flexible image dimensions, 2D RoPE, 2×2 feature pooling, and a two-layer MLP before the LLM.
  • ViT pre-training: ViT pre-training proceeds through masked image modeling with 2D RoPE, native-resolution contrastive learning, and omni-modal pre-training.These stages target visual geometry and structure, image-text alignment, and unified representations across video, audio, and captions.
  • Video encoding: Dynamic Frame-Resolution Sampling jointly adjusts video frame rate and spatial resolution to balance temporal information, visual detail, and computational cost.Sampling rates vary by task from 1 FPS by default to 2 FPS for detailed temporal information and 5 FPS for counting or motion tracking, with timestamp tokens and an 81,920-token video budget.

3 Pre-training

Seed1.5-VL pre-training combines a 3-trillion-token multimodal corpus with targeted data curation and synthesis for capabilities including OCR, charts, tables, grounding, and rare visual knowledge. The section also examines how training loss scales with tokens and predicts downstream metrics, while emphasizing trade-offs in balancing common and rare concepts.

  • The pre-training corpus contains 3 trillion diverse, high-quality source tokens categorized by target capabilities.
  • Generic Image-Text Pairs & Knowledge Data: Rare-visual knowledge improves when common-species samples are capped, but aggressively reducing common-species exposure harms common-species recognition.Max1k-46M improves rare-species performance, whereas Max100-15M further enhances rare-species memorization while adversely affecting common-species recognition.
  • Optical Character Recognition (OCR): The OCR dataset exceeds 1 billion samples and includes over 200 million synthesized text-intensive images spanning documents, scene text, tables, charts, and flowcharts.Synthetic data uses diverse fonts and scripts, with augmentations designed to improve robustness to visual distortions.
  • Synthetic charts and rendered tables expand structured visual training, producing over 100 million chart examples and over 50 million table images.Chart data combines open-source and generated examples, while table images are rendered from HTML, LaTeX, and Markdown text.
  • Scaling Laws: Training loss for OCR and grounding datasets follows a power-law relationship with training tokens, while related downstream metrics appear approximately log-linear with sub-category loss locally.The reported pre-training metrics are not directly comparable to final results obtained after reinforcement learning.

4 Post-training

Seed1.5-VL’s post-training combines supervised fine-tuning with reinforcement learning to develop instruction-following and reasoning abilities. The process uses curated multimodal data, human preferences, and verifiable rewards in iterative training.

  • Post-training overview: Post-training combines Supervised Fine-tuning (SFT) and Reinforcement Learning (RL) to strengthen instruction-following and reasoning abilities.The SFT model is iteratively improved by distilling learnings from the RL model until the prompt pool is exhausted and performance metrics converge.
  • Supervised Fine-tuning: SFT uses General Instruction data for concise, accurate responses and Long Chain-of-Thought data for detailed reasoning.The multimodal SFT corpus contains approximately 50,000 samples and is trained with an in-house text-only dataset and LongCoT data.
  • Supervised Fine-tuning: The SFT data pipeline expands coverage through a capability taxonomy, crowdsourcing, research-community samples, clustering, and targeted downsampling.The construction process includes approximately 13,000 crowdsourced instruction-tuning samples and 30,000 additional high-quality samples from the research community.
  • Reinforcement Learning from Human Feedback: RLHF trains a reward model from list-wise multimodal preference data collected through human annotation and heuristic synthesis.Human annotations compare candidate responses with a 5-scale rating system, while synthetic data uses prompts with clear ground truths and format constraints.
  • Reinforcement Learning from Human Feedback: The reward model acts as a generative classifier that predicts preference indicators for two responses given a prompt.The method evaluates both response orderings to reduce positional bias and uses iterative updates to maintain annotation consistency as standards evolve.
  • Hybrid Reinforcement Learning: Hybrid reinforcement learning combines human-feedback optimization with verifiable rewards for tasks whose solutions can be checked by rules or external executors.The visual RLVR tasks include STEM problems, with answer verification based on transformed symbolic expressions matched against ground truths.

5 Training Infrastructure

Seed1.5-VL’s infrastructure addresses multimodal training inefficiencies caused by heterogeneous data and an asymmetric vision-language architecture. The system combines parallelism, workload balancing, data-loading, fault-tolerance, and hybrid reinforcement-learning components for scalable training.

  • Infrastructure optimizations: Training optimizations include hybrid parallelism, workload balancing, parallelism-aware data loading, robust training, efficient attention, checkpointing, kernel fusion, and communication overlap.The pretraining phase consumed 1.3 million GPU hours, normalized to H800 GPU hours.
  • Multimodal training challenges: VLM training is challenging because visual and language data are heterogeneous and the small vision encoder is paired with a much larger language model.Existing sequential unimodal frameworks can treat the vision encoder as preprocessing or disaggregate it from the language model, producing imbalance.
  • Workload balancing: A greedy workload-balancing algorithm assigns images to GPUs according to computation intensity and performs group-wise balancing within groups of 128–256 GPUs.Images are sorted by required FLOPS, then assigned to the GPU with the lowest accumulated computation intensity.
  • Data loading: A parallelism-aware data loader reduces redundant reads and preprocessing by coordinating sample loading across GPUs in non-data-parallel and pipeline-parallel groups.For example, only one GPU within a pipeline-parallel group loads the data before it is shared.
  • Fault tolerance: The robust training framework detects hardware or software faults, resumes from the last successful checkpoint, and uses ByteCheckpoint for efficient saving and recovery.This provides fault tolerance during large-scale training.
  • Reinforcement-learning infrastructure: Hybrid reinforcement learning combines human-feedback and verifier-feedback training in a verl-based framework with controllers for role dataflow and model parallelism.Verifiers run in process-based services to isolate faults, while the system reuses pretraining optimizations for actor and critic updates.

6 Evaluation

Seed1.5-VL shows strong performance across visual, video, multimodal reasoning, document, agent, and out-of-distribution evaluations, while retaining limitations in fine-grained perception and complex reasoning. Its agent evaluations show broad performance across interfaces and games, alongside reported scaling potential and practical user-preference gains.

  • Vision Task Evaluation: Seed1.5-VL performs strongly on multimodal reasoning and document understanding, reaching SOTA on MathVista at 85.6 thinking, TextVQA at 84.2 non-thinking, and DocVQA at 96.9 non-thinking.It remains competitive on several benchmarks where Gemini 2.5 Pro leads, while setting new SOTA results on multiple document and chart tasks.
  • Vision Task Evaluation: Seed1.5-VL achieves state-of-the-art results across grounding and counting benchmarks, including RefCOCO-avg at 91.6 and CountBench at 93.7 thinking.It also surpasses Grounding DINO-L on LVIS-MG, with Seed1.5-VL at 73.8 non-thinking versus 54.4 F1-score for Grounding DINO-L.
  • Video Task Evaluation: Seed1.5-VL achieves SOTA performance across several short-video and streaming benchmarks, although its Video-MMMU and MMVU scores trail top models such as Gemini 2.5 Pro.For long videos, it supports a 128K-token context covering up to 640 frames.
  • Multimodal Agent: Seed1.5-VL outperforms leading baselines on GUI tasks and games, including 87.2% on WebVoyager, 870.6 in 2048, and 1414.0 in Hex-Frvr.Its GUI results exceed OpenAI CUA and Claude 3.7 Sonnet on several benchmarks, while gameplay performance remains higher as interaction rounds increase.
  • Generalization and Limitations: The model achieves the second-highest overall score in one state-of-the-art comparison, with gaps partly attributed to its approximately 20B active parameters and nonsaturated training loss.Its usefulness rate is 62.6%, compared with 57.4% for Gemini 2.5 Pro 0325, and it demonstrates OOD performance comparable to Gemini Pro 2.5 and OpenAI o1.

7 Conclusion and Next Steps

Seed1.5-VL combines broad multimodal capabilities with strong public-benchmark performance despite its moderate size. The report also identifies remaining challenges and documents the architecture, data, training, infrastructure, and evaluation practices behind the model.

  • 38 out of 60 evaluated public benchmarks achieved state-of-the-art results, including a score of 77.9 on MMMU.
  • Seed1.5-VL generalizes beyond benchmark tasks to Rebus puzzles, handwritten-code correction, computer interaction, and gameplay.
  • Scaling analysis shows no sign of performance saturation, making increased model parameters and training compute a promising immediate direction.
  • The report identifies robust 3D spatial reasoning, hallucination mitigation, and complex combinatorial search as limitations shared by contemporary VLMs.
  • The report details Seed1.5-VL’s architecture, data synthesis pipeline, training methodology, framework innovations, and internal evaluation design.

8 Contributions and Acknowledgments

This material combines qualitative capability examples with acknowledgments and author-list information. The examples include visual reasoning and puzzle solving, while the acknowledgments identify the authors’ listing convention and internal aliases.

  • Acknowledgments: Some listed names refer to the authors’ internal aliases at the company.
  • Contributions: Qualitative examples cover visual reasoning, visual puzzles, geometry problems, and object counting across figures 9 to 18.
  • Contributions: Figure 9 presents a visual-centric thinking strategy that emerged during LongCoT RL.

A.2 Reasoning Cases: Geolocation Prediction

Seed1.5-VL demonstrates multimodal reasoning across geolocation, Rebus, word, and visual-pattern puzzles, combining visual analysis with logical inference and knowledge retrieval. The cases also expose occasional incorrect answers or reasoning, especially in challenging spatial tasks.

  • Geolocation Prediction: The geolocation response analyzes stadium architecture, signage, vegetation, and urban infrastructure before predicting Chicago, United States.It reports latitude 41.8316 for the predicted location.
  • Rebus Puzzles: The model solves Rebus puzzles by combining visual clues with word meanings, although one example has logical reasoning but an incorrect final answer.The reported correct answer for the third Rebus example is “travel overseas.”
  • Emoji Movie Identification: Seed1.5-VL identifies movie titles from emoji combinations by linking icons to plot information.One example connects a tropical island and volleyball to “Cast Away” through the character Wilson.
  • Word Games: The model extracts hidden capital-city names from phrases and integrates OCR, reasoning, and knowledge retrieval in word games.The examples identify London, Beijing, Tokyo, Rome, Berlin, Moscow, Paris, and Madrid.
  • Visual Pattern Recognition: For visual pattern puzzles, the model combines row elements or unions symbol positions to select answers 7 and D.The first puzzle uses a union rule; the second integrates an arrow stem with a grid structure.
  • Image Comparison: The cases include image-difference localization using bounding-box instructions, but the supplied passages do not report the resulting differences.The prompt requires integer coordinates from 0 to 999 with valid box ordering.

A.13 OCR Parsing and Document Understanding

Seed1.5-VL is presented as handling document-oriented multimodal tasks including fault-tolerant training retrieval, OCR-based extraction, multilingual receipt parsing, and structured diagram generation. The examples emphasize instruction following alongside visual text understanding.

  • Training Cost: The RL phase requires 60k GPU hours, compared with 24k GPU hours for reward-model training.Thus, the RL phase takes longer to train in the reported comparison.
  • Fault-Tolerant Training: The robust training framework detects faults and recovers training from the last successful checkpoint using ByteCheckpoint.This example follows an instruction to skip the first sentence of the referenced section.
  • OCR Parsing: The model extracts dish names, quantities, and prices from an image into a markdown table.The example includes bilingual dish names and structured quantity and price columns.
  • OCR Parsing: Seed1.5-VL extracts multilingual text from a long receipt.The passage identifies the example as multilingual receipt extraction without reporting an accuracy metric.
  • Diagram Understanding: The model generates Mermaid code for the South region of a tournament bracket and connects rounds through a South Winner node.The code represents first-round matchups, second-round winners, Elite 8 links, and the regional winner.

A.16 Image-conditioned Creative Writing

Seed1.5-VL generates creative text conditioned on image content, producing a romantic poem that interprets clasped hands through imagery of intimacy, memory, and commitment.

  • Image-conditioned Creative Writing: The model generates an English romantic poem inspired by a picture of intertwined hands.The poem is titled “Fingers Entwined.”
  • Image-conditioned Creative Writing: The poem uses touch, synchronized heartbeats, and intertwined hands to convey connection and enduring commitment.Its imagery also invokes shared memories and a future built together.

A.17 Failure Cases: 3D Spatial Imagination

The spatial-reasoning cases show successful manipulation of dice and block structures, but also document failures in robust 3D spatial reasoning and faithful representation of atypical network diagrams.

  • Spatial Reasoning Cases: The dice example predicts 5 on top after downward and rightward movements, based on the stated initial face configuration.The response argues that these movements do not change the top face in the described sequence.
  • Spatial Reasoning Cases: The block-structure example maps column heights 1, 2, 3, 2 to answer D.The response compares this height distribution with the candidate views.
  • Failure Cases: Seed1.5-VL, Gemini 2.5 Pro, and OpenAI o4-mini all fail to produce robust reasoning for the shown 3D spatial tasks.In one case, Seed1.5-VL gives incorrect reasoning despite a correct final answer.
  • Failure Cases: All tested models generate standard-residual DOT code instead of faithfully representing a non-standard FFN residual connection.The case concerns an atypical residual path in the displayed network structure.

A.19 Failure Cases: Combinatorial Search I

The case study examines a prime-digit multiplication puzzle and shows that solving it requires searching among many combinations. The reported reasoning reaches a candidate solution but does not reliably satisfy the puzzle’s constraints.

  • The puzzle assigns each digit one of the prime values 2, 3, 5, or 7 and asks for a two-digit and three-digit number.
  • The reported reasoning tests combinations such as 325 and 77, using intermediate multiplication results to check prime-digit constraints.
  • The proposed conclusion is 325 × 77 = 25025, although an intermediate result contains 0, which violates the stated prime-digit rule.
  • 45 = 1024 combinations make the search space appear to exceed reliable reasoning capabilities for Seed1.5-VL, Gemini 2.5 Pro, and OpenAI o4-mini.
  • A separate example shows the same models failing to count all squares in an image, indicating difficulty with exhaustive visual enumeration.

B.1 Internal Benchmark Structure

The internal benchmark organizes integrated vision and capability evaluation hierarchically, from broad capability groups to fine-grained categories. Its reported metrics cover instruction following, perception, reasoning, documents, translation, GUI tasks, and autonomous driving.

  • The benchmark has two top-level nodes, 11 level-2 categories, 43 level-3 categories, and 29 level-4 categories.The hierarchy supports increasingly fine-grained capability analysis during model development.
  • Document & Diagram Understanding: Document and diagram results include 91.3% for Summarization and 98.0% for Reasoning over Document/Diagram and Open Knowledge.
  • Translation: Translation scores are 93.8% for Minor Languages Translation and 87.3% for Translation between Chinese and English.
  • Integrated capability: Integrated-task results include 96.2% for Mobile & Computer Screen Understanding, 96.0% for multi step GUI Reasoning, and 99.3% for VLN & Autonomous Driving.
  • Vision capability: Reported results include 100.0% for Visual Atomic instruction and Indoor Directional Reasoning, and 99.3% for Satellite Image Matching.
  • Vision capability: Perception results include 98.0% for Hard Indoor Deduplication Counting, 97.3% for Same Room Detection, and 100.0% for ARC-AGI Vision (in-house).

B.2 Comprehensive Comparisons on internal benchmarks

Seed1.5-VL is compared with eight prominent model families on an internal benchmark and ranks second overall. The comparison reports stronger performance than Llama 4 Maverick and a top-five ranking dominated by thinking models.

  • Seed1.5-VL ranks second overall among comparisons with Gemini, GPT, Claude, Qwen, Llama, InternVL, StepFun, and GLM.
  • Seed1.5-VL significantly outperforms Llama 4 Maverick on the internal benchmark despite comparable reported active-parameter scale and shared MoE architecture.
  • Thinking models predominate among the top five, consistent with the benchmark’s focus on integrated model capabilities.
  • Newer releases from the same provider generally outperform earlier iterations, including GPT-4o-Latest over GPT-4o-0513 and Gemini 2.5 over Gemini 2.0.

B.3 Capabilities and Benchmark Tasks

Seed1.5-VL is evaluated across 60 public benchmarks spanning ten capability areas, using zero-shot instruction-tuned models and benchmark-specific prompts. The evaluation includes multimodal reasoning, visual understanding, grounding, counting, video, and GUI-agent tasks.

  • The evaluation covers 60 public benchmarks across ten capabilities, including reasoning, visual question answering, documents, grounding, counting, spatial understanding, video, and GUI agents.
  • All benchmarks are evaluated zero-shot with an instruction-tuned model, while Seed1.5-VL’s thinking mode is activated by an added preamble.
  • Prompt templates replace placeholders with benchmark questions, answer options, image or video embeddings, and labels for grounding tasks.
  • Multimodal reasoning: MathVision prompts require step-by-step solutions enclosed in one boxed answer, with multiple-choice responses restricted to a single option letter.
  • Grounding and counting: Grounding prompts require bounding-box outputs, while counting prompts require pointing to objects before reporting their number.
  • Video understanding: Streaming video evaluation asks models to monitor a video continuously and decide whether it is the right time to respond at each frame.
Loading 2505.07062v1…