Seed1.5-VL Technical Report

• The report identifies gaps in human-level generality, especially 3D spatial understanding, counting, imaginative visual inference, and interactive gameplay, alongside scarce annotations and multimodal-data complexity.These issues complicate data pipeline design, parallel training strategies, and evaluation protocols.Source: §1 · p. 4
• Seed1.5-VL uses diversified synthesis pipelines for OCR, grounding, counting, video, and long-tail knowledge, and trains on trillions of multimodal tokens before post-training with human feedback and verifiable rewards.Post-training data synthesis also targets visual puzzles and games to strengthen general reasoning.Source: §1 · p. 4
• To improve large-scale multimodal training throughput, the report introduces hybrid parallelism and vision-token redistribution, alongside a customized data loader and system-level optimizations.The techniques target imbalance between the vision encoder and language model, GPU workload balance, and I/O bottlenecks under 3D parallelism.Source: §1 · p. 4
• 38 of 60 public benchmarks achieve state-of-the-art performance, including 21 of 34 vision-language, 14 of 19 video, and 3 of 7 GUI-agent benchmarks.The model is also deployed in an internal chatbot to monitor real-world and out-of-distribution performance in dynamic, interactive environments.Source: §1 · p. 4
• Seed1.5-VL combines a 532-million-parameter vision encoder with a language model of 20 billion active parameters to reduce inference costs and computational demands.The report describes this compact design as suitable for interactive applications and API access.Source: §1 · p. 4

2 Architecture

• Seed1.5-VL’s architecture comprises a vision encoder, an MLP adapter, and an LLM, with adjacent 2×2 feature patches average-pooled before a two-layer MLP processes them.The encoder natively supports dynamic image resolutions and uses 2D RoPE; the authors favor its image compression for high-resolution representation with fewer tokens.Source: §2 · p. 5

2.1 Vision Encoder

• Seed-ViT is a 532-million-parameter native-resolution encoder with zero-shot classification performance comparable to 6-billion-parameter InternVL-C.The design addresses fine-detail losses associated with fixed-resolution encoders, particularly for high-resolution images, video, and OCR.Source: §2.1 · pp. 5–6

2.1.1 Architecture

• Images are resized to the nearest multiple of 28 × 28 pixels, split into 14 × 14 patches, and batched with attention masks that prevent cross-image attention.The encoder applies 2 × 2 average pooling to output patch embeddings before passing them through the MLP adapter to the LLM.Source: §2.1.1 · p. 6

2.1.2 ViT Pre-training Stage

• Seed-ViT pretraining proceeds through masked image modeling with 2D RoPE, native-resolution contrastive learning, and omni-modal pretraining.The pipeline maintains the same architecture through ViT pretraining and VLM stages and uses the full range of intended training data. MIM masks 75% of patches and uses teacher CLIP features as reconstruction targets; contrastive learning aligns pooled image and text embeddings, while omni-modal training aligns video-frame, audio, and caption embeddings.Source: §2.1.2 · p. 7

2.2 Video Encoding

• Dynamic Frame-Resolution Sampling varies frame rate by task, using 1 FPS by default, 2 FPS for detailed temporal tasks, and 5 FPS for counting or motion tracking.Timestamp tokens anchor frames on the video timeline to improve temporal awareness across variable frame rates.Source: §2.2 · pp. 7–8
• Video frames receive one of six token allocations from 640 to 128 under an 81,920-token-per-video budget, trading frame resolution against the number of frames.If a long video still exceeds the encoding limit at 128 tokens per frame, uniform sampling reduces the frame count while retaining representation of the full video.Source: §2.2 · p. 8

3.1.1 Generic Image-Text Pairs & Knowledge Data

• Limiting common-species representation improved rare-species recognition, but the strongest restriction hurt common-species recognition, motivating balanced coverage of common and rare visual concepts.The comparison used Random-46M, Max1k-46M, and Max100-15M; the authors also duplicate alt-texts from underrepresented domains to rebalance the corpus.Source: §3.1.1 · pp. 8–9, Table 3 · p. 9

3.1.2 Optical Character Recognition (OCR)

• OCR training combines over 1 billion samples with more than 200 million synthetic text-intensive images, over 100 million chart examples, and over 50 million rendered table images.The dataset covers documents, scene text, tables, charts, and flowcharts; filtered VQA pairs generated from OCR outputs and images further improved textual understanding in images.Source: §3.1.2 · pp. 9–10

3.1.3 Visual Grounding & Counting

• Grounding and counting data combine bounding-box, point, and count annotations, with coordinates normalized to [0, 999] to support predictions across image resolutions.Sources include filtered public datasets and automatic annotation pipelines; the resulting data includes about 200 million multi-object grounding samples, 170 million pointing instructions, and 8 million counting samples.Source: §3.1.3 · pp. 10–11

3.1.4 3D Spatial Understanding

• Single-image 3D data targets relative depth sorting, absolute depth estimation, and 3D grounding through generated depth relationships, annotated depth maps, and reformulated question-answer pairs.The sources include 2 million internet images for relative depth, 18 million absolute-depth instruction pairs, and 770K 3D-grounding instruction pairs.Source: §3.1.4 · p. 11

3.1.5 Video

• Video pre-training covers captioning, QA, action recognition and grounding, temporal grounding, and streaming, including frame-by-frame proactive responses and synchronized commentary.The data combines public and internal sources and supports timestamped responses to video streams.Source: §3.1.5 · pp. 11–12

3.1.6 STEM; 3.1.7 GUI

• STEM data spans educational image comprehension and problem solving, while GUI data trains perception, grounding, and multi-step planning across web, app, and desktop environments.STEM resources include 3.2 million educational grounding samples and over 100 million K12 exercises; GUI trajectories include observations, intermediate thoughts, and actions.Source: §3.1.6 · p. 12, §3.1.7 · p. 12

3.2 Training Recipe

• Training proceeds from adapter-only alignment with the vision encoder and language model frozen, to full-parameter multimodal training, then a balanced mixture with new domains and longer sequences.Stage 1 uses 3 trillion tokens and a small text-only share (e.g., 5%); stage 2 increases sequence length from 32,768 to 131,072. The reported alternative stage-0 strategy performed worse.Source: §3.2 · p. 13

3.3 Scaling Laws

• Most subcategory training losses follow a power law with training tokens, while OCR loss and ChartQA and InfographicVQA top-1 accuracy show an approximately log-linear relationship locally.The authors caution that the downstream relationship may hold only within a local neighborhood because evaluation metrics are bounded.Source: Figure 3 · p. 14, §3.3 · pp. 14–15
• Figure 3(e,f) detail an estimated relationship between training grounding loss and RefCOCO performance.Performance prediction remains an active research area.Source: §3.3 · p. 15
• Prior work has used sigmoid functions to model LLM performance from loss [37] [151] or compute [101].The passage presents performance prediction as an active research area.Source: §3.3 · p. 15

4 Post-training

• Post-training alternates SFT and RL, distilling RL learnings into SFT until prompts are exhausted and metrics converge; four rejection-sampling update rounds have improved the current iteration.The iterative updates collect correctly answered challenging prompts, verify them with RL-stage verifiers, and filter undesirable patterns such as repetition and overthinking.Source: §4 · p. 15, §4.5 · p. 20

4.1.1 SFT Data Construction

• SFT combines concise general-instruction data with toggleable LongCoT data and builds about 13,000 crowdsourced examples plus 30,000 preference-filtered open-source samples.The construction process increases prompt complexity through self-instruct synthesis and manual verification, excluding examples beyond model capacity to mitigate hallucinations.Source: §4.1 · p. 16, §4.1.1 · p. 16

4.1.2 Training Recipe

• SFT trains the combined multimodal, text-only, and LongCoT corpus for two epochs with the vision encoder frozen and other parameters trainable.Training uses a 131,072-token sequence length, batch size 16 times the sequence length, AdamW, 10% warm-up, and cosine decay from 2 × 10^-5 to 2 × 10^-6.Source: §4.1.2 · pp. 16–17

4.2.1 Preference Data

• RLHF preference data combines balanced, human-rated response rankings with synthetic list-wise preferences that prioritize correct, well-formatted answers.Synthetic preferences also include image-captioning pairs intended to reduce hallucinations; human annotators identify response issues and explanations.Source: §4.2 · p. 17, §4.2.1 · p. 17

4.2.2 VLM as a Reward Model

• The reward model compares responses as a generative classifier, while RL prompts are filtered for balanced capability coverage and reward-model discrimination.The approach evaluates both response orderings to mitigate positional bias and uses reward-score variance to exclude prompts where the reward model is insufficiently discriminative.Source: §4.2.2 · p. 17, §4.2.3 · p. 18

4.3.1 Visual STEM

• RLVR uses over one million visual STEM problems, filtering for SFT accuracy above 0% and at most 75% before checking answers against symbolic ground truths.Multiple-choice items are converted to open-ended questions, and the verifier matches predicted answers as SymPy expressions.Source: §4.3 · p. 18, §4.3.1 · p. 18

4.3.2 Visual Perception and Reasoning

• Other visual RLVR tasks verify grounding and pointing with IoU, visual instructions with regular expressions, and over 20,000 synthetic puzzles with answer matching.For “Spot the Differences,” the model describes differences and localizes them with bounding boxes, trained on inpainted images and systematically modified SVG pairs.Source: §4.3.2 · p. 19

4.4 Hybrid Reinforcement Learning

• Hybrid RL mixes reward-model-scored general prompts with verifier-scored tasks, applies format and KL controls, and trains a shared critic across both reward sources.Although RL training uses LongCoT responses, regular responses also improve; the recipe samples 4,096 roll-outs per episode and uses 8 gradient steps per episode.Source: §4.4 · p. 20

5.1 Large-Scale Pre-training

• Pretraining consumed 1.3 million GPU hours normalized to H800 and used hybrid parallelism, workload balancing, parallelism-aware loading, and kernel, checkpointing, and communication optimizations [13] [173].The report says these optimizations were developed to accelerate and stabilize pretraining.Source: §5.1 · p. 21, §5.1.4 · p. 21

5.1.1 Hybrid Parallelism

• Hybrid parallelism separates the vision encoder/adaptor strategy from the LLM’s 4-D parallelism to improve efficiency and avoid pipeline-level imbalance [30].The encoder and adaptor use ZeRO data parallelism [109]; the LLM combines expert parallelism [65] [123], interleaved pipeline parallelism [50] [93] [94], ZeRO-1 data parallelism [109], and context parallelism [77].Source: §5.1.1 · p. 21

5.1.2 Workload Balancing

• Workload balancing greedily assigns images by descending computation intensity to the least-loaded GPU, with group-wise balancing across groups of 128–256 GPUs.Restricting balancing to evenly sized groups reduces data-redistribution overhead.Source: §5.1.2 · p. 21

5.1.3 Parallelism-Aware Data Loading

• The parallelism-aware data loader avoids redundant reads, filters unnecessary images before GPU transfer, and overlaps IO and computation through prefetching.Within a pipeline-parallel group, one GPU loads data and broadcasts metadata to other ranks.Source: §5.1.3 · p. 21

5.1.4 Fault Tolerance

• MegaScale [57] detects training faults and resumes from the last successful checkpoint; ByteCheckpoint [136] supports efficient checkpoint saving and resuming.Source: §5.1.4 · pp. 21–22

5.2 Post-Training Framework

• Post-training used a verl-based [122] framework for RLHF and RLVF, consuming 60k GPU hours for RL and 24k for reward-model training.Actor and critic training uses 3-D parallelism [50] [93] [109] [123], while vLLM [67] generates rollouts; ByteCheckpoint [136] supports checkpoint saving and resuming.Source: §5.2 · p. 22

6.1.1 Vision Encoder as a Zero-shot Classifier; 6.1.2 Vision Task Evaluation

• Seed-ViT averaged 82.5 zero-shot accuracy across six datasets, while Seed1.5-VL achieved top results on reasoning and general visual-question benchmarks including MathVista, RealWorldQA, and SimpleVQA.Seed-ViT used 9% of InternVL-C-6B’s parameters; Seed1.5-VL scored 85.6 on MathVista and 78.4 on RealWorldQA in thinking mode, and 63.4 on SimpleVQA in thinking mode.Source: §6.1.1 · p. 22, Table 5 · p. 22, §6.1.2 · p. 23

6.1.2 Vision Task Evaluation

• Seed1.5-VL set document-understanding highs, led all listed grounding and counting benchmarks, and exceeded prior VLMs on depth estimation and multi-view reasoning.Reported results include 96.9 on DocVQA, 91.6 on RefCOCO-avg, and 59.0 on All-Angles Bench in non-thinking mode; thinking mode regressed on SUN-RGBD relative to non-thinking mode.Source: §6.1.2 · pp. 23, 25, Table 6 · p. 24

6.1.3 Video Task Evaluation

• Seed1.5-VL achieved state-of-the-art results on short-video, streaming-video, and video-grounding benchmarks, while trailing top models on Video-MMMU and MMVU.Long-video evaluation used a 128K-token context window of up to 640 frames; the report identifies expanding that window as future work.Source: §6.1.3 · p. 25, Table 7 · p. 26

6.2 Multimodal Agent

• Seed1.5-VL outperformed OpenAI CUA and Claude 3.7 Sonnet on GUI grounding and several computer- and browser-use benchmarks, and outperformed prior models across multiple games.It scored 60.9 on ScreenSpot Pro, 36.7% on OSWorld, and 870.6 in 2048; its gameplay performance remained higher as interaction rounds increased.Source: Table 8 · p. 26, §6.2 · pp. 27–28, Table 9 · p. 27

6.3.1 Motivation and Design Principles

• The Chinese-focused internal suite addresses English-heavy public benchmarks, multimodal benchmark saturation, and rule-based evaluation limitations, spanning over 100 tasks and more than 12,000 samples from varied sources and domains.LLM-judge accuracy averages above 95% for multiple-choice or simple-answer questions and exceeds 90% for open-ended questions; data are deduplicated, and task types and sources are periodically refreshed.Source: §6.3.1 · pp. 28–29

6.3.2 Comparison with State-of-the-arts

• 61.6 was the leading internal-benchmark score from Gemini 2.5 Pro; Seed1.5-VL ranked second overall and led OOD, Agent, and Atomic Instruction Following, with weaknesses in knowledge, reasoning, code, and captioning/counterfactual tasks.The report attributes part of the gap to its approximately 20B active parameters, noting training loss had not saturated after 3 trillion tokens; it also reports better performance than Llama 4 Maverick on this benchmark.Source: Table 10 · p. 29, §6.3.2 · pp. 29–30

6.3.3 Out-of-distribution Generalization

• On internal OOD evaluation, Seed1.5-VL performed comparably to Gemini 2.5 Pro and OpenAI o1; a separate search-enabled usefulness test scored it at 62.6%.Examples include solving a rebus, correcting code from a whiteboard image, and converting an unfamiliar diagram into Mermaid code under a spatial constraint.Source: §6.3.3 · p. 30

6.4 Limitations

• Despite broad benchmark strengths, Seed1.5-VL remains limited in fine-grained visual perception and complex reasoning.Source: §6.4 · p. 30
• Seed1.5-VL can miscount irregular, similar-colored, or occluded objects, miss subtle image differences, and misinterpret complex spatial relationships or visual prompts.Errors include misidentifying content or quantity in specified regions and localization mistakes.Source: §6.4 · p. 31
• The model struggles with human-simple puzzles and mazes, while combinatorial search challenges many VLMs; visual chain-of-thought and code-use or other external tools are proposed directions.The report also notes that VLMs can produce incorrect inferences on visual puzzles and that programmatic approaches may address combinatorial search more readily.Source: §6.4 · p. 31
• Seed1.5-VL, like most VLMs, is limited in 3D spatial reasoning, including object manipulation and reasoning about 3D projections; image generation may support visual chain-of-thought.The report presents image-generation integration as a potential direction for future research.Source: §6.4 · p. 31
• Reasoning errors can arise from perceptual mistakes or weak logical deduction, and planning or complex instructions may lead Seed1.5-VL to overlook conditions or make unfounded assumptions.Source: §6.4 · p. 32
• Internal evaluations found difficulty ordering continuous actions or before-and-after object states, while multi-image reasoning degrades when clues have strong logical interdependencies.Source: §6.4 · p. 32
• Hallucination remains a challenge for VLMs: when visual input conflicts with language-model prior knowledge, models may prioritize the prior and override perceived visual information.Source: §6.4 · p. 32

7 Conclusion and Next Steps

• Seed1.5-VL achieves state-of-the-art results on 38 of 60 evaluated public benchmarks, including 77.9 on MMMU.The report identifies MMMU as a key indicator of multimodal reasoning ability.Source: §7 · p. 32
• Seed1.5-VL generalizes beyond its training data to visual reasoning, handwritten-code correction, computer interaction, and gameplay.Examples include solving Rebus puzzles and interpreting and correcting code from whiteboard images; the report calls for further exploration of these emergent abilities.Source: §7 · p. 32
• Scaling analysis shows no sign of performance saturation, while robust 3D reasoning, hallucination mitigation, and combinatorial search remain challenges.Immediate directions include increasing model parameters and training compute, unifying capabilities with image generation to potentially enable visual Chain-of-Thought, and developing robust tool use.Source: §7 · p. 32
• The report documents Seed1.5-VL’s architecture, data synthesis, training methodology, training-framework innovations, and internal evaluation design to support future research.The authors situate these advances within prior community work, including Transformer and Vision Transformer architectures.Source: §7 · p. 32

A Qualitative examples

• Qualitative examples illustrate visual reasoning, puzzle solving, geometry, crowded-scene counting, OCR, 3D spatial understanding, video event localization, and image-conditioned creative writing.Figures 25–28 present failure cases involving spatial imagination, hallucination, and combinatorial search.Source: A Qualitative examples · p. 47

A.1 Reasoning Cases: Visual Reasoning

• Figure 9 demonstrates a visual-centric thinking strategy that emerged during LongCoT RL.The report presents it as one of the model’s visual reasoning examples.Source: A Qualitative examples · p. 47, Figure 9 · p. 48

A.2 Reasoning Cases: Geolocation Prediction

• The geolocation example identifies Chicago, United States, and estimates coordinates at 41.8316° N, 87.6348° W after reasoning from stadium and urban cues.The figure describes the approach as analyzing visual details before recognizing a location.Source: §A.2 · p. 49, Figure 10

A.3 Visual Reasoning: Solving Rebus Puzzles

• For a rebus showing cracked “FAST,” the model interprets the cracks as “break” and combines that cue with FAST; the figure says its first two rebus answers were correct.Source: §A.3 · p. 50, Figure 11 · p. 50
• For the POT-and-eight-circles rebus, it derives “Potato”; for TRAVEL above seven Cs, it reasons “travel the seven seas,” though the correct answer is “travel overseas.”Source: §A.3 · p. 50, Figure 11 · p. 50

A.4 Visual Reasoning: Emoji Quiz

• Emoji clues lead the model to “Cast Away” through the island and volleyball, and to “Rocky” through Italian heritage, a stallion, and boxing.Source: §A.4 · p. 51, Figure 12 · p. 51
• It identifies “Independence Day” from emojis of a UFO, landmarks, cityscapes, and explosions associated with the film’s alien attacks on cities.Source: §A.4 · p. 51, Figure 12 · p. 51

A.5 Visual Reasoning: Word Game I

• In the word game, it finds eight capital cities hidden across four phrases: London and Beijing, Tokyo and Rome, Berlin and Moscow, and Paris and Madrid.The figure characterizes the task as combining letter recognition, capital-city retrieval, and reasoning.Source: §A.5 · p. 52, Figure 13 · p. 52

A.6 Visual Reasoning: Word Game II

• CHEWY satisfies the five-letter word puzzle by excluding letters from zero-match guesses, combining candidates from two-match guesses, and checking the result against every guess.The candidate set is C, H, E, W, Y.Source: §A.6 · p. 53

A.7 Visual Reasoning: Visual Pattern Recognition

• The model identifies missing visual patterns by combining row elements, answering 7 in one puzzle and D in another.The examples apply union or integration patterns to the third element in each row.Source: §A.7 · p. 54, Figure 15 · p. 54

A.8 Visual Puzzles: Find the Differences

• The model marks four differences in the first image with bounding boxes: the girl’s braids, the painter’s hat, the sketchpad, and palette colors.The response uses coordinates on a 0–999 scale, with each box specified by top-left and bottom-right coordinates.Source: §A.8 · p. 55

A.9 Geometry

• Figure 17 shows two examples of Seed1.5-VL solving geometry problems that require multi-step reasoning.Source: Figure 17 · p. 56

A.10 Counting in a complex scene

• 27 cherry tomatoes are counted after the model points to their locations in the image.The figure describes the approach as detecting queried objects first and then counting them.Source: §A.10 · p. 57, Figure 18 · p. 57

A.11 Spatial Understanding: Depth Sorting

• Figure 19 illustrates the model estimating objects’ depth ordering from a single image.Source: Figure 19 · p. 58

A.12 Video Temporal Grounding

• The model segments a video into timestamped key events and localizes the woman’s jumps at 6–8 seconds and 16–22 seconds.The event summary includes preparation, a run and jump, walking away, and a replay.Source: §A.12 · p. 58, Figure 20 · p. 58

A.13 OCR Parsing and Document Understanding

• The report says Seed1.5-VL uses MegaScale’s robust training framework to handle hardware and software faults during training.Source: §A.13 · p. 59
• The RL phase requires 60k GPU hours, compared with 24k GPU hours for reward-model training.Source: §A.13 · p. 59
• After detecting a fault, the framework resumes training from the last successful checkpoint, using ByteCheckpoint [155] for efficient saving and resuming.Source: §A.13 · p. 59
• Figure 21 demonstrates text extraction, query understanding, and answer retrieval from a document, including following an instruction to skip its first sentence.The reported response uses MegaScale’s robust training framework to handle hardware and software faults.Source: Figure 21 · p. 59, §A.13 · p. 59

A.14 Multilingual OCR Parsing

• Figure 22 shows the model extracting multilingual text from a long receipt.Source: Figure 22 · p. 60

A.15 Generate Code for a Diagram of Novel Format

• The model generated Mermaid code for the South-region bracket, covering matchups from the first round through the South Winner.The code lays out the opening matchups, second-round pairings, Elite 8, and final South Winner node.Source: §A.15 · p. 61, Figure 23

A.16 Image-conditioned Creative Writing

• The model responds to a picture-inspired romantic-poem prompt with “Fingers Entwined,” an example of image-conditioned generation.The figure describes the output as a poem semantically related to the input image.Source: §A.16 · p. 62, Figure 24 · p. 62
• The poem portrays clasped hands as intimate connection, pairing traced palms with gleaming rings and shared promises.It uses touch and the rings as imagery for stories and promises shared between the couple.Source: §A.16 · p. 62
• It extends the moment into a shared future, recalling first meeting and imagining years together, synchronized heartbeats, and lasting love.The poem connects memory of the couple’s beginnings with the future they will hold together.Source: §A.16 · p. 62

A.17 Failure Cases: 3D Spatial Imagination

• For the dice prompt, Seed1.5-VL concluded that the top remained 5, reasoning that the specified rolls did not affect the top face.The figure characterizes the reasoning in a Seed1.5-VL case as incorrect despite a correct final answer.Source: §A.17 · p. 63, Figure 25 · p. 63
• In the block-view prompt, the response identified four column heights of 1, 2, 3, and 2 cubes from left to right.It used that height distribution to assess the candidate views.Source: §A.17 · p. 63
• The block-view response selected option D, whose stated heights of 1, 2, 3, and 2 match the identified structure.The response rejected A, B, and C as mismatches.Source: §A.17 · p. 63
• Seed1.5-VL, Gemini 2.5 Pro, and OpenAI o4-mini all failed to produce correct reasoning in two 3D spatial examples; Seed1.5-VL’s reasoning was incorrect even when its final answer was correct.The cases illustrate a limitation in robust 3D spatial reasoning.Source: Figure 25 · p. 63

A.18 Failure Cases: Hallucination (Knowledge Prior)

• With a non-standard FFN residual connection, Seed1.5-VL, Gemini 2.5 Pro, and OpenAI o4-mini all generated DOT code for a standard residual connection instead.The figure presents this as a tendency to fall back on a familiar connection pattern rather than represent the atypical residual path faithfully.Source: Figure 26 · p. 64
• The non-standard FFN residual example demonstrates failure to generate faithful DOT code for an atypical network structure.The diagram specifically depicts the unusual residual path.Source: Figure 26 · p. 64

A.19 Failure Cases: Combinatorial Search I

• The prompt specifies that each circle represents 2, 3, 5, or 7 and that puzzle #6 has only one solution.Source: §A.19 · p. 65
• The response narrows the units digits to 5 and 7 because their product, 35, ends in the permitted prime digit 5.It frames these as likely units digits before testing candidate numbers.Source: §A.19 · p. 65
• After trying combinations, the response tests the three-digit number 325 with the two-digit number 77.Source: §A.19 · p. 65
• The model response concludes that 325 × 77 = 25025 solves the puzzle, although the product contains 0, which is outside the permitted prime-digit set.The response acknowledges an intermediate zero but still presents this as the solution despite the stated digit constraint.Source: §A.19 · p. 65
• Seed1.5-VL, Gemini 2.5 Pro, and OpenAI o4-mini are shown struggling with a prime-digit puzzle whose 4^5 = 1024 combinations exceed reliable search capabilities.The puzzle asks for a two-digit and a three-digit number with every digit drawn from {2, 3, 5, 7}.Source: Figure 27 · p. 65

A.20 Failure Cases: Combinatorial Search II

• The three VLMs also fail to correctly count all squares in an image, illustrating difficulty with exhaustive visual enumeration.The figure characterizes the arrangement as a complex configuration of simple shapes.Source: Figure 28 · p. 66

B.1 Internal Benchmark Structure

• The benchmark has two top-level nodes—vision and integrated capability—and 11 level-2, 43 level-3, and 29 level-4 categories for increasingly fine-grained capability assessment.This categorization helps identify deficiencies, gauge improvements over prior iterations, and guide future development.Source: §B.1 · p. 67
• Capability-level accuracy is defined as the alignment rate between LLM-as-a-judge assessments and human evaluations.The benchmark structure is presented in Table 11.Source: Table 11 · p. 67
• Reported alignment rates include 100.0% for ARC-AGI Vision (in-house), 91.3% for Summarization, 98.0% for Reasoning over Document/Diagram and Open Knowledge, and 93.8% for Minor Languages Translation.Source: §B.1 · p. 68

B.2 Comprehensive Comparisons on internal benchmarks

• Seed1.5-VL significantly outperforms Llama 4 Maverick, despite having a comparable size to its reported 17 billion active parameters.Llama 4 Maverick is reported to use a Mixture-of-Experts architecture.Source: §B.2 · p. 69
• Newer releases from the same provider generally score higher, as GPT-4o-Latest beats GPT-4o-0513 and Gemini 2.5 surpasses Gemini 2.0.The comparison includes different releases from some model families to assess progress over time.Source: §B.2 · p. 69
• Seed1.5-VL ranks second overall among comparisons spanning eight prominent model families.Figure 29 orders models by overall internal-benchmark performance.Source: Figure 29 · p. 69, §B.2 · p. 69
• Thinking models predominate among the top five, which the authors attribute to the benchmark’s focus on integrated model capabilities.Figure 29 distinguishes thinking models with red bars and non-thinking models with blue bars.Source: Figure 29 · p. 69, §B.2 · p. 69

B.3 Capabilities and Benchmark Tasks

• The suite covers 60 public benchmarks across ten capabilities: multimodal reasoning, general visual question answering, document/chart understanding, grounding/counting, spatial understanding, short/long/streaming video, video grounding, and GUI agents.The benchmark inventory includes tasks such as MMMU, DocVQA, RefCOCO, LongVideoBench, Charades-STA, and OSWorld.Source: §B.3 · p. 70

B.4 Evaluation Prompts

• All benchmarks use 0-shot evaluation with an instruction-tuned model; a thinking-mode preamble encloses reasoning in <think> tags before the answer.Benchmark-specific templates fill in questions, options, image or video embeddings, and labels where relevant.Source: §B.4 · p. 71
• MMMU prompts preserve original image placement and require a final answer in a specified option format.The prompt template requests an answer of the form “Final answer: [the correct option].”Source: §B.4 · p. 71
• MathVision prompts require boxed answers to support precise parsing, and the same prompt is used to evaluate Seed1.5-VL and Gemini-2.5-Pro.The prompt was designed to reduce verifier parsing issues, including answers with units or option values instead of option letters.Source: §B.4 · p. 71
• Grounding and counting prompts request point coordinates or bounding boxes, while spatial prompts specify camera parameters and 3D-box coordinates, dimensions, and Euler angles.For absolute depth estimation, the prompt asks for the photographer-to-object distance in meters.Source: §B.4 · pp. 73–74
• Video prompts vary by task: StreamingBench proactive evaluation asks whether to respond at each frame, Video-MMMU open-ended prompts prohibit intermediate reasoning, and temporal grounding asks for start and end seconds.The streaming prompt requires continuous monitoring; Video-MMMU also has a separate multiple-choice format.Source: §B.4 · pp. 74, 76, 77