Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervision

1 Introduction

• Prevailing text-only autoregressive objectives treat vision as passive input, encouraging models to discard details unnecessary for coarse text generation and creating an information bottleneck for dense perception [Liu et al., 2023, Assran et al., 2023].Source: §1 · p. 2
• Youtu-VL’s VLUAS paradigm predicts visual tokens alongside text, shifting from “vision-as-input” to “vision-as-target” to retain fine-grained visual details.A learned visual codebook expands the textual lexicon into a unified multimodal vocabulary; the Synergistic Vision Tokenizer fuses high-level semantics with low-level geometry to produce discrete visual codes.Source: §1 · pp. 2–3, Figure 2 · p. 2
• Youtu-VL supports dense vision prediction and text-based tasks such as segmentation, depth estimation, grounding, and detection within a standard VLM without task-specific additions.For detection and grounding, it generates bounding boxes using an axis-specific vocabulary with absolute pixel coordinates; for segmentation and depth, it predicts from native logits without extra task-specific heads or embeddings.Source: §1 · pp. 2–3
• Youtu-VL achieves competitive performance on general multimodal and vision-centric tasks, supporting its use as a foundation for generalist visual agents.Source: §1 · p. 3

2 Architecture and Methodology

• Youtu-VL combines a vision encoder, a projector into a unified vision-language token space, and an LLM; its training pipeline adds visual-token prediction without auxiliary decoders or task-specific tokens.Stage 3 trains a vision tokenizer to build the visual codebook, and Stage 4 applies multi-label next-token prediction to vision tokens.Source: §2 · p. 3
• The vision encoder processes images at native resolution, while the projector concatenates adjacent 2 × 2 patch features to reduce token count to one-quarter before mapping them into the LLM input space.The encoder uses SigLIP-2 [Tschannen et al., 2025] with spatial RoPE and window attention, inserting global attention every eight layers.Source: §2 · pp. 3–4

2.1 Vision-Language Unified Autoregressive Supervision Paradigm

• The vision tokenizer fuses language-aligned SigLIP-2 features [Tschannen et al., 2025] with spatially structured DINOv3 features [Siméoni et al., 2025], then quantizes them into a 150,000-entry, 768-dimensional codebook.It combines the fused representation with the original structural features before applying Index Backpropagation Quantization.Source: §2.1 · pp. 4–5
• The tokenizer excludes pixel-wise ℓ1 loss, balances perceptual and adversarial fidelity with codebook regularization, and achieves 97.74% codebook utilization on the composite dataset.The learned visual indices are merged with text tokens to form the unified vocabulary.Source: §2.1 · p. 5
• VLUAS predicts language and visual tokens autoregressively, using continuous visual embeddings as context and discrete image-code tokens as targets to preserve input cues while enabling dense supervision.The objective weights text and image prediction losses, with λ empirically set to 0.5.Source: §2.1 · pp. 4–5

2.2 Vision-Centric Predictions from the standard VLM

• Text-prediction tasks use axis-specific X/Y coordinate tokens and absolute pixel coordinates for localization, with task formulations covering grounding, detection, pose estimation, polygon segmentation, and counting.Detection supports single- or multi-class queries, including “detect all”; polygon outlines are capped at 20 points, and counting can use direct numeric prediction or detect-then-count.Source: §2.2 · p. 6
• For dense tasks, Youtu-VL derives predictions directly from vision-token logits without auxiliary modules, using category tokens for segmentation and discretized depth bins for depth estimation.Category-token logits are aggregated, reshaped into a spatial grid, upsampled, and argmaxed to produce pixel-level maps; depth bins support linear or logarithmic quantization.Source: §2.2 · p. 6
• NTP-M supervises each vision patch with multi-hot targets and independent Bernoulli token probabilities, enabling patches to represent multiple labels, tasks, and granularities.Its loss averages valid positives separately from top-k relevant negatives and masks indices with missing annotations to address class imbalance and incomplete labels.Source: §2.2 · p. 7

3.1 Training Recipe

• The four-stage recipe progresses from language-backbone training on approximately 10T text tokens to multimodal pre-training and task adaptation, using visual reconstruction and LNTP-M losses in Stages 3–4.Stage 3 trains the LLM backbone, vision encoder, and projector end-to-end on approximately 1.8T mixed tokens; Stage 4 adapts the full model on approximately 0.6T multimodal instruction tokens, including synthetic short CoT data.Source: §3.1 · pp. 8–9

3.2.1 Vision-Centric Data

• Vision-centric training data combines text-prediction tasks such as grounding, detection, counting, classification, and pose estimation with dense segmentation and depth labels.Open-world data pipelines synthesize detection, segmentation, and depth examples from raw and labeled data, including pseudo-labeling and depth quantization.Source: §3.2.1 · pp. 9–10, Figure 5 · p. 10

3.2.2 Image Caption and Knowledge Data

• Image-caption data curation filters a roughly 5-trillion-token raw corpus, then uses concept-balanced sampling, rare-class mining, and knowledge-injected recaptioning before deduplication.The purified dataset contains approximately 1 trillion tokens; recaptioning aims to produce detailed, visually grounded descriptions.Source: §3.2.2 · pp. 10–11

3.2.3 Optical Character Recognition (OCR) Data; 3.2.4 Science, Technology, Engineering, and Mathematics (STEM) Data

• OCR data synthesis addresses brevity, prompt monotony, and missing reasoning, while STEM curation filters, verifies, and expands visually grounded reasoning questions.The OCR pipeline creates verbose QA and rendered synthetic samples; STEM question expansion generates diverse queries from images and answers, yielding a corpus with rigorous reasoning chains and robust visual grounding.Source: §3.2.3 · p. 12, §3.2.4 · pp. 12–13

3.2.5 Graphical User Interface (GUI) Data; 3.2.6 Pure Text Data

• GUI pre-training combines single-turn UI grounding with long-horizon desktop, mobile, and web trajectories, while pure-text mid-training data helps preserve language abilities.GUI data undergoes ensemble-based quality filtering and privacy and access-control checks; top-scoring trajectories receive human annotation, while other validated data supports continual pre-training.Source: §3.2.5 · pp. 13–14, §3.2.6 · p. 14

3.3 Analysis of Pre-training Paradigm

• VLUAS avoids the early performance saturation seen with text-dominant supervision, sustaining a steeper learning trajectory and widening its advantage through Stage 4.Across 27 multimodal benchmarks, the text-dominant model plateaus early in Stage 3, while VLUAS retains superior data efficiency even with improved Stage 4 data.Source: Figure 8 · p. 14, §3.3 · pp. 14–15

3.4 Analysis of Data Scaling

• Across 2.4T training tokens, average performance on 27 multimodal benchmarks rises monotonically from approximately 0.43 to over 0.74.Evaluation error follows neural scaling laws, with Stage 3 α ≈0.102 and Stage 4 α ≈0.079.Source: Figure 9 · p. 15, §3.4 · p. 15

3.5 Visual Representation Analysis

• PCA visualizations show that vision-token supervision gives Youtu-VL-4B clearer semantic structure and sharper object separation than its unsupervised counterpart.Without vision supervision, Youtu-VL-4B has representation capabilities comparable to Qwen3-VL; adding supervision improves its visual representations.Source: Figure 10 · p. 16, §3.5 · p. 16

4.1.1 Training Recipe

• Supervised fine-tuning expands the context window from 16K to 32K tokens and uses AdamW [Loshchilov and Hutter] with cosine decay from 2 × 10−5 to 2 × 10−6.The recipe uses a 5% warmup ratio.Source: §4.1.1 · pp. 16–17

4.1.2 Data Curation

• SFT data curation mines quality- and alignment-scored, taxonomy-balanced examples and rewrites open-source prompts into detailed paragraph-level responses and reasoning steps.The rewrite-and-expand pipeline uses a powerful VLM to analyze visual details, increasing the information density of original data by an order of magnitude.Source: §4.1.2 · p. 17

4.2 Reinforcement Learning

• Reinforcement learning proceeds through perception tasks, STEM reasoning, and general VQA to improve fine-grained visual perception, problem-solving, and task generalization.Perception RL covers grounding, detection, segmentation, and counting; STEM RL uses text-only and multimodal data with verifiable rewards; General RL includes OCR, chart understanding, and instruction following.Source: §4.2 · pp. 17–18

4.2.1 Data Curation

• RL data curation categorizes examples by domain, prioritizes verifiable objective samples, retains consensus-validated responses, and discards samples all eight SFT-model responses answer correctly.Domains include vision-centric tasks, STEM, OCR, chart understanding, general VQA, and complex instruction following; filtering also removes hard-to-verify or guessable prompts.Source: §4.2.1 · p. 18

4.2.2 Reward System

• Task-specific rewards use structured-prediction metrics for vision tasks, rule-based and verifier checks for STEM, string matching or edit distance for OCR and charts, and LLM checks for VQA and instruction following.Auxiliary rewards penalize dominant-language mismatches and excessive n-gram repetition, following Youtu-LLM.Source: §4.2.2 · pp. 18–19, Table 1 · p. 19

4.2.3 Training Recipe

• RL uses 32,768-token contexts, 6,144 rollouts per stage, mini-batches of 1,536, four update steps, and a 1 × 10−6 policy learning rate.The recipe also uses clipping [0.20, 0.24], soft overlong punishment, no KL penalty, and FP16 training; variance-aware sampling rejects low-variance or all-nonpositive-reward groups, while consistent sampling admits groups only when K(q) ≤ τK. The cited setup follows DAPO [Yu et al., 2025] and Youtu-LLM, and removes the KL penalty following Qi et al. [Qi et al., 2025].Source: §4.2.3 · pp. 19–20

5.1 Vision-Centric Tasks

• Youtu-VL scores 90.4% δ1 on NYUv2 depth estimation, 89.1% PCKh@0.5 on MPII pose estimation, and 88.6% accuracy on CountBench.It also scores 89.3% Top-1 on ImageNet-ReaL; depth is predicted directly without task-specific fine-tuning or additional inference.Source: §5.1 · pp. 21, 23
• Youtu-VL averages 91.8% across RefCOCO splits and reaches 47.1% mAP on COCO detection without extra heads or task-specific inference modifications.For detection, this exceeds GiT’s 46.7% mAP, while remaining below UFO’s 48.9% and VisionLLM-v2’s 56.7%.Source: §5.1 · p. 21, Table 2 · p. 22
• Youtu-VL achieves 54.2 mIoU on ADE20k and 80.7% mIoU on RefCOCO validation through direct dense prediction and grounding-then-segmentation, without task-specific decoders.On COCOStuff it scores 52.5% mIoU versus SAN’s 45.7%, and its segmentation supports open category prompts.Source: §5.1 · p. 21, Table 2 · p. 22

5.2 General Multimodal Tasks

• Youtu-VL consistently outperforms or matches leading comparably sized models across multimodal tasks, scoring 83.9 on MMBench English and 83.6 on Chinese.Its performance on MMVet is comparatively lower, leaving room to improve on complex instruction-style VQA requiring compositional reasoning and response calibration.Source: §5.2 · p. 23, Table 3 · p. 24
• Youtu-VL scores 25.7% on VisuLogic, 88.9% on VLMs Are Blind, and 56.5% on MathVerse, while attaining around 60 on MMMU.Its reasoning is competitive, especially on image-grounded inference, but knowledge-intensive and context-heavy academic benchmarks retain a modest performance gap.Source: Table 3 · p. 24, §5.2 · p. 25
• Youtu-VL reaches about 79% on CharXivDQ and 44% on CharXivRQ, with competitive results across OCR-heavy benchmarks and multi-image and real-world understanding.It scores 74.6 on RealWorldQA; on HallusionBench it scores 59.1%, with the evaluation reporting reduced hallucination and stronger grounding in visual evidence.Source: §5.2 · pp. 25–26
• Youtu-VL achieves state-of-the-art performance on ScreenSpot Pro and OSWorld, including a 38.8 success rate on OSWorld.The benchmarks test static GUI grounding and dynamic multi-turn interaction in a live Ubuntu environment.Source: §5.2 · p. 26
• On text-only benchmarks, Youtu-VL scores 76.9 on IFEval, 76.8 on MMLU-Redux, and 39.8 on GPQA-Diamond, with headroom relative to leading language models.The paper identifies language capability improvement as a future objective.Source: §5.2 · p. 26

Conclusion

• VLUAS shifts VLM optimization from “vision-as-input” to “vision-as-target,” jointly predicting fine-grained visual details and high-level linguistic semantics.The paper frames this as a way to mitigate the text-dominant optimization bias of traditional architectures.Source: §6 · p. 27
• The unified objective enables a standard VLM to execute diverse vision-centric tasks without task-specific decoders or auxiliary heads.The conclusion describes this as end-to-end high-fidelity perception within a single generalist transformer.Source: §6 · p. 27
• Youtu-VL remains limited by visual granularity on low-resolution inputs, sensitivity to sensor intrinsics and training-distribution diversity, and weaknesses in complex mathematical reasoning and dense knowledge retrieval.These constraints limit high-precision perception, zero-shot robustness on out-of-distribution geometry-aware tasks, and high-level cognition.Source: §6 · p. 27

Contributions and Acknowledgments

• The paper thanks all contributors, including those not listed, and says names within each group are in no particular order.Source: Contributions and Acknowledgments · p. 27

Core Contributors

• The Core Contributors subsection provides a roster of named contributors.The roster names Zhixiang Wei, Yi Li, Zhehan Kan, Xinghua Jiang, Zuwei Long, Shifeng Liu, Hongze Shen, Wei Liu, Xiaoyu Tan, Haojia Lin, Yubo Zhu, Qianyu Li, Di Yin, Haoyu Cao, Weibo Gu, Xin Li♠, Yinsong Liu, Deqiang Jiang, Xing Sun†, and Yunsheng Wu.Source: Core Contributors · p. 27

A.1.1 Visual Grounding

• RefCOCO grounding prompts the model to return a queried object's bounding box, while open-world prompts support flexible and Chinese phrasing.The target is typically provided as a keyword or phrase.Source: §A.1.1 · p. 37

A.1.2 Object Detection

• COCO detection predicts categories without a preset category, parses outputs into labels and boxes, scores boxes by area, and merges 1×–4× predictions with NMS at IoU 0.7.Open-world detection supports multiple requested categories; “detect anything” can miss less prominent categories.Source: §A.1.2 · pp. 38–39

A.1.3 Image Classification

• ImageNet-ReaL classification asks for a single word or short phrase naming the dominant object, counted correct when it matches the ground-truth label.Source: §A.1.3 · p. 40

A.1.4 Object Counting

• TallyQA and CountBench require a single numeric count that exactly matches ground truth, using images upsampled by 2× or 2.5×.Source: §A.1.4 · p. 41

A.1.5 Semantic Segmentation

• Semantic-segmentation evaluation uses dataset-specific prompts and image scaling, emits class references and RLE masks, and applies softmax at temperature 0.2 with DenseCRF refinement.For open-world segmentation, direct prediction is faster but lower quality than grounding or detection followed by segmentation; the combined approach can improve background accuracy.Source: §A.1.5 · pp. 41, 42, 44, 46

A.1.6 Referring Expression Segmentation

• Referring-expression segmentation has an unevaluated polygon mode; the recommended grounding-then-segmentation pipeline crops padded boxes, segments foreground and background, and evaluates with cIoU.The crop uses 1.2 padding and is resized with its shorter side set to 1280; DenseCRF is used except for polygon outputs.Source: §A.1.6 · pp. 46–47

A.1.7 Depth Estimation

• NYUv2 depth evaluation uses preset depth IDs, linear quantization over 0–10 meters into 1–1000, dequantization at test time, and threefold image resizing.The prompt precedes the image; predictions are upsampled before argmax and then resized to the original image size, with invalid depths excluded.Source: §A.1.7 · p. 48
• Cityscapes and DDAD follow NYUv2 testing criteria but quantize depth to 1–1000 over 0–80 m and 0.05–120 m, respectively, without resizing.Cityscapes also masks invalid regions along the left and bottom; both dataset prompts request a depth estimate and return a depth map as an RLE string.Source: §A.1.7 · p. 49
• Open-world depth defaults to log-uniform quantization of 0.5–100 m into 1–1000, ignores out-of-range values, and requires de-quantization for real depths.Inputs are resized to a 2000-pixel focal length; prompts may precede or follow the image, and the approach uses standard VLMs without additional structures, though it relies on substantial training data and depth-anything capability still needs improvement.Source: §A.1.7 · pp. 49–50

A.1.8 Human Pose Estimation

• MPII validation uses PCKh@0.5 on ground-truth person-box crops, selecting the predicted pose whose keypoint center is closest to the ground-truth center for one-to-one evaluation.The prompt requests person instances with bounding boxes and 16 keypoints in standard MPII ordering, encoded with dedicated coordinate and visibility tokens.Source: §A.1.8 · p. 50

Implementation Details

• Fine-grained benchmarks use dynamic high resolution with 1,280 minimum and 65,536 maximum patches, while less resolution-sensitive benchmarks retain the 64-patch default.The high-resolution group includes DocVQA, MMMU, MMBench, and RealWorldQA; the default-resolution group includes MathVista, OCRBench, ScienceQA, and HallusionBench.Source: §A.2 · p. 51

Evaluation settings

• The VLMEvalKit-based evaluation framework combines exact matching with LLM judging when answers cannot be extracted or are ambiguous.The authors enhanced the framework to address limitations of exact-match evaluation on some multiple-choice benchmarks.Source: §A.2 · p. 51

Prompt Specifics

• Complex reasoning and real-world analysis benchmarks use step-by-step prompting, including specialized reasoning prompts for VisuLogic, mathematical benchmarks, MMMU, and RealWorldQA.VisuLogic requests a boxed final answer; MathVerse, MathVista, and LogicVista elicit structured geometric or mathematical derivations; MMMU prompts encourage detailed reasoning, and RealWorldQA uses step-by-step reasoning for spatial and physical understanding.Source: §A.2 · pp. 51–52
• HallusionBench and POPE disable chain-of-thought and require concise Yes/No answers, while document and OCR-related tasks require a short phrase or single-word response.The constraints target concise outputs for hallucination testing and facilitate metric calculation for DocVQA, InfoVQA, TextVQA, ChartQA, and AI2D.Source: §A.2 · pp. 52–53

B Extended Experiments and Comparison

• The standard multimodal LLM achieves competitive or state-of-the-art performance in segmentation and grounding comparisons across five representative settings, without extra modules, heads, or task embeddings.The comparisons cover vision specialist models, CLIP-based vision–language models, vision generalist architectures, multimodal LLMs with task-specific additions, and the standard multimodal LLM.Source: B Extended Experiments and Comparison · p. 56

B.1 Comparison With Dense Prediction Methods

• Youtu-VL achieves superior performance across dense prediction benchmarks without complex architectural additions.Table 4 covers semantic segmentation, depth estimation, and referring segmentation, comparing Youtu-VL with specialist, CLIP-based, generalist, and multimodal models.Source: §B.1 · p. 56, Table 4 · p. 57
• 54.2 mIoU on ADE20K exceeds GiT’s 47.8 and SAN’s 32.1, while Youtu-VL handles fine-grained segmentation unlike standard multimodal LLMs.Youtu-VL achieves results across all five semantic segmentation datasets.Source: §B.1 · p. 56, Table 4 · p. 57
• 90.4 on NYUv2 exceeds DepthLLM’s 87.6, and fine-tuning yields 92.7 δ1 versus SwinMTL’s 92.1.Youtu-VL processes the image once, whereas DepthLLM draws each point and runs inference multiple times. UniDepthv2 performs higher on NYUv2, but DDAD results are close: 88.2 versus 87.6.Source: §B.1 · p. 56
• 80.7 cIoU on RefCOCO exceeds GLaMM’s 79.5 and UniPixel’s 80.5.UniPixel requires an extra SAM decoder, while UFO scores 80.0 with extra mask-token embeddings and polygon prediction scores 74.5; Youtu-VL requires no extra architectural modifications.Source: §B.1 · pp. 56–57, Table 4 · p. 57

B.2 Comparison on Localization Tasks

• Youtu-VL secures leading visual-grounding results across all RefCOCO series splits.Source: §B.2 · p. 57
• Youtu-VL achieves superior localization performance without complex architectural additions, with detection, counting, and grounding jointly establishing leading performance in a standard VLM framework.Table 5 covers COCO 2017 validation, CountBench, TallyQA-Simple and TallyQA-Complex, and the RefCOCO series.Source: §B.2 · p. 57, Table 5 · p. 58
• 47.1 mAP on COCO validation rises to 48.0 mAP after detection-only SFT from Stage 3, bridging the gap with specialized architectures.The 47.1 mAP result is for Youtu-VL on the COCO validation split; 48.0 mAP is after fine-tuning exclusively on detection data starting from Stage 3.Source: §B.2 · p. 57, Table 5 · p. 58
• Youtu-VL exhibits remarkable counting performance on CountBench, TallyQA-simple, and TallyQA-complex.Source: §B.2 · p. 57, Table 5 · p. 58

• Figures 12–13 show qualitative COCO object-detection examples.Source: Figure 12 · p. 59, Figure 13 · p. 60
• Figure 14 shows a qualitative open-world detection example from Objects365.Source: Figure 14 · p. 61
• Figure 15 shows a qualitative detect-then-count example using a photo taken by a Youtu-VL Team author.Source: Figure 15 · p. 62
• Figures 16–17 show qualitative semantic-segmentation examples from ADE20k.Source: Figure 16 · p. 63, Figure 17 · p. 64
• Figures 18–19 show RefCOCO visual-grounding examples, Figure 20 shows RefCOCO referring-expression segmentation, and Figures 21–22 show NYUv2 depth-estimation examples.Source: Figure 18 · p. 65, Figure 19 · p. 66, Figure 20 · p. 67

Human Pose Estimation

• Figure 23 shows a qualitative example of human pose estimation from a generated image (HY).Source: Figure 23 · p. 69