Source-linked AI summary
Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervision
Zhixiang Wei, Yi Li, Zhehan Kan, Xinghua Jiang, Zuwei Long, Shifeng Liu, Hongze Shen, Wei Liu, Xiaoyu Tan, Haojia Lin, Yubo Zhu, Qianyu Li, Di Yin, Haoyu Cao, Weibo Gu, Xin Li, Yinsong Liu, Deqiang Jiang, Xing Sun, Yunsheng Wu, Mingkong Tang, Shuangyin Liu, Lexiang Tang, Haodong Lin, Junru Lu, Jiarui Qin, Lingfeng Qiao, Ruizhi Qiao, Bo Ke, Jianfeng He, Ke Li, Yangning Li, Yunhang Shen, Mengdan Zhang, Peixian Chen, Kun Yin, Bing Liu, Yunfei Wu, Huang Chen, Zhongpeng Cai, Xiaotian Li
TL;DR
Existing VLMs often lose fine-grained visual information because training treats vision as a passive input under text-dominant objectives. Youtu-VL introduces VLUAS, which predicts visual tokens alongside language and extends this supervision to vision-centric tasks in a standard architecture. The framework achieves competitive performance on general multimodal and vision-centric tasks, while retaining limitations in visual granularity, geometry-aware robustness, and advanced cognition.
Problem
Current VLM training treats visual signals as passive inputs under text-only autoregressive supervision, limiting retention of fine-grained visual information.
Method
Youtu-VL introduces VLUAS, integrating visual tokens into the autoregressive prediction stream and supporting vision-centric tasks without task-specific additions.
Results
Youtu-VL achieves competitive performance on both general multimodal tasks and vision-centric tasks.
Takeaways & Limitations
A standard VLM architecture can jointly model visual details and linguistic content while natively performing diverse vision-centric tasks.
Takeaways & Limitations
Performance remains constrained by visual granularity on low-resolution inputs, geometry-aware out-of-distribution robustness, and complex mathematical reasoning and dense knowledge retrieval.
Abstract
from arXiv · showhide
Despite the significant advancements represented by Vision-Language Models (VLMs), current architectures often exhibit limitations in retaining fine-grained visual information, leading to coarse-grained multimodal comprehension. We attribute this deficiency to a suboptimal training paradigm inherent in prevailing VLMs, which exhibits a text-dominant optimization bias by conceptualizing visual signals merely as passive conditional inputs rather than supervisory targets. To mitigate this, we introduce Youtu-VL, a framework leveraging the Vision-Language Unified Autoregressive Supervision (VLUAS) paradigm, which fundamentally shifts the optimization objective from ``vision-as-input'' to ``vision-as-target.'' By integrating visual tokens directly into the prediction stream, Youtu-VL applies unified autoregressive supervision to both visual details and linguistic content. Furthermore, we extend this paradigm to encompass vision-centric tasks, enabling a standard VLM to perform vision-centric tasks without task-specific additions. Extensive empirical evaluations demonstrate that Youtu-VL achieves competitive performance on both general multimodal tasks and vision-centric tasks, establishing a robust foundation for the development of comprehensive generalist visual agents.
1 Introduction
Youtu-VL addresses fine-grained visual-information loss by replacing text-dominant supervision with VLUAS, treating visual tokens as prediction targets. It extends this unified objective to vision-centric tasks within a standard VLM architecture and reports competitive performance across general multimodal and vision-centric evaluations.
- Motivation: Current VLMs treat visual signals as passive inputs and optimize mainly autoregressive text generation, creating an information bottleneck for dense perception.The paper attributes this limitation to text-dominant optimization that encourages discarding visual details unnecessary for coarse-grained text generation.
- VLUAS: The Synergistic Vision Tokenizer fuses semantic concepts with geometric structures into discrete visual codes for dense semantic visual supervision.The framework constructs a unified multimodal vocabulary through a learned visual codebook.
- VLUAS: VLUAS shifts supervision from text-only objectives to unified autoregressive prediction of visual and linguistic tokens.Youtu-VL expands the textual lexicon with a learned visual codebook and treats visual signals as supervisory targets.
- Vision-Centric Predictions: Youtu-VL supports dense vision prediction and text-based vision tasks in a standard VLM architecture without task-specific additions.Its vision-centric scope includes segmentation, depth, grounding, and detection, with direct prediction mechanisms for both streams.
- Empirical Performance: Extensive evaluations show competitive performance on both general multimodal and vision-centric tasks, supporting Youtu-VL as a foundation for generalist visual agents.The reported result is a broad cross-task conclusion rather than a single benchmark score.
2 Architecture and Methodology
Youtu-VL unifies visual and linguistic supervision by representing images as discrete tokens predicted autoregressively, while supporting vision-centric tasks through a standard VLM architecture.
- Architecture: Youtu-VL combines a vision encoder, vision-language projector, and LLM within a standard architecture.The projector maps visual features into a unified vision-language token space.
- Vision Tokenizer: The synergistic tokenizer fuses semantic and structural features before quantization to preserve high-level meaning and fine spatial detail.SigLIP-2 supplies language-aligned semantics, while DINOv3 supplies structural information; cross-attention produces the fused representation.
- Vision Tokenizer: Perceptual, adversarial, vector-quantization, and entropy objectives produce a visual codebook with 97.74% utilization on the composite dataset.The tokenizer deliberately excludes pixel-wise ℓ1 reconstruction to avoid texture shortcuts.
- VLUAS: VLUAS extends the textual lexicon with visual tokens, treating visual signals as prediction targets alongside language.The unified vocabulary is built from discrete image codes and textual tokens.
- VLUAS: The VLUAS objective jointly reconstructs visual details and linguistic content, addressing the text-dominant optimization bias.The image loss directly supervises visual-token prediction in the autoregressive stream.
- Vision-Centric Predictions: Vision-centric tasks use coordinate-token text prediction, token-based dense prediction, and multi-label autoregressive supervision within the standard architecture.Absolute pixel coordinates support localization, while dense prediction aggregates token logits into spatial maps and multi-label loss handles multiple targets.
3 Pre-training
Youtu-VL uses a four-stage pre-training recipe that progresses from language foundation building to multimodal foundation pre-training and versatile task adaptation. Its unified visual supervision supports continued scaling, visual representation quality, and broad visual-language capability.
- Training recipe: Stages 1 and 2 establish the language backbone with pure text, followed by multimodal foundation pre-training in Stage 3 and versatile task adaptation in Stage 4.The stages progress from language preparation to multimodal and task-oriented training.
- Training recipe: Stage 3 mixes image-caption and vision-centric data with high-quality text, while Stage 4 trains on multimodal instructions spanning VQA, OCR, STEM, GUI, detection, segmentation, grounding, and pose estimation.Stage 3 uses approximately 1.8T tokens, and Stage 4 uses approximately 0.6T tokens.
- Data construction: The data pipelines construct open-world vision-centric data through parallel detection-segmentation and depth-estimation branches, using grounding, data binding, depth models, quantization, and augmentation.These processes support object detection, semantic segmentation, and depth estimation.
- Data construction: The STEM corpus is built through quality filtering, synthesis and consistency verification, and visual-grounded question expansion to improve reasoning details, fidelity, and query diversity.The resulting corpus is described as high-fidelity, visually grounded, and diverse in query perspectives.
- Scaling and analysis: VLUAS scaling follows neural scaling laws with exponents α ≈0.102 in Stage 3 and α ≈0.079 in Stage 4, while vision-token supervision produces clearer semantic structures and sharper object separation.The visual representation analysis compares Youtu-VL-4B with and without vision supervision.
4 Post-training
Post-training refines Youtu-VL through supervised fine-tuning and a three-stage reinforcement-learning framework. The framework progresses from fine-grained perception to STEM reasoning and general multimodal generalization, using curated and verifiable rewards to stabilize optimization.
- Supervised fine-tuning: Supervised fine-tuning extends the context window from 16K to 32K tokens and uses AdamW with cosine learning-rate decay after a 5% warmup.The learning rate decays from 2 × 10−5 to 2 × 10−6.
- Supervised fine-tuning: The supervised fine-tuning dataset combines mined pre-training data with rewritten and expanded open-source examples to improve instruction following, reasoning, and multimodal task coverage.A VLM scores and filters mined samples, while open-source examples receive paragraph-level descriptions and reasoning steps.
- Reinforcement learning: The three reinforcement-learning stages target fine-grained visual perception, complex STEM reasoning, and broader generalization across OCR, chart understanding, instruction following, and general VQA.Perception RL covers grounding, detection, segmentation, and counting before reasoning and general RL.
- Reinforcement learning: RL data curation categorizes tasks, filters for verifiability, validates responses by model consensus, and calibrates complexity by discarding samples solved correctly by every generated response.The process prioritizes deterministic ground-truth labels and samples that remain challenging for the SFT model.
- Reward design: Task-specific rewards compare structured vision predictions with deterministic annotations using metrics such as IoU or mAP, while STEM rewards use strict output formats for automated answer extraction.The reward design provides verifiable optimization targets across different task domains.
- Reward design: Auxiliary language-consistency and repetition-detection rewards penalize mixed-language outputs and excessive n-gram repetition, respectively.These rewards target code-switching and degenerate looping behavior.
- Training stability: Reward-variance-aware and consistent sampling remove weakly discriminative or excessively policy-drifting rollout groups to improve policy-optimization stability.The sampling strategies address low-advantage gradients, non-positive rewards, and mismatch between training and inference engines.
5 Evaluation
Youtu-VL performs competitively across vision-centric and general multimodal evaluations, including dense prediction, grounding, reasoning, OCR, and hallucination-sensitive tasks. Its results support a unified visual agent capable of broad visual understanding while retaining some task-specific performance gaps.
- Evaluation Scope: 30 vision-centric and 45 general multimodal benchmarks form the evaluation suite, spanning perception, reasoning, OCR, and GUI operations.The evaluation pipeline unifies assessment across dozens of task types and reports category averages where results are available for all compared models.
- Vision-Centric Tasks: 91.8% average grounding score across RefCOCO splits exceeds InternVL-3.5-4B’s 89.4% and is close to Seed1.5-VL’s 91.6%.
- Vision-Centric Tasks: 80.7% mIoU on RefCOCO val demonstrates high-quality referring-expression segmentation without extra mask tokens, decoders, or task-specific heads.
- Vision-Centric Tasks: 90.4% δ1 on NYUv2 and 87.6% δ1 on DDAD show depth estimation close to specialist performance, without additional inference or task-specific fine-tuning.
- General Multimodal Tasks: General multimodal results are consistently strong across VQA, reasoning, OCR, and hallucination benchmarks, though knowledge-intensive and context-heavy tasks retain modest gaps.Examples include 83.9 on MMBench English, 56.5% on MathVerse, roughly 60 on MMMU, and about 79% on CharXivDQ.
6 Conclusion
Youtu-VL reframes VLM optimization from vision-as-input to vision-as-target through VLUAS, supporting unified prediction of visual details and language. The framework enables a standard VLM architecture to perform diverse vision-centric tasks, while remaining limited by representation granularity, specialized geometry robustness, and complex reasoning.
- VLUAS shifts optimization from passive visual conditioning to generative supervision over visual and linguistic content.This addresses the text-dominant bias described for traditional architectures.
- Youtu-VL enables a standard VLM architecture to execute diverse vision-centric tasks without task-specific decoders or auxiliary heads.
- The framework’s current visual representation granularity bottlenecks high-precision tasks on low-resolution inputs.
- Sensitivity to sensor intrinsics and training-distribution diversity limits zero-shot robustness for specialized geometry-aware tasks in out-of-distribution environments.
- Complex mathematical reasoning and dense knowledge retrieval remain areas requiring further optimization despite strong general perception.
Contributions and Acknowledgments
The acknowledgments recognize contributors’ support and state that contributor names within each group are listed without a particular ordering.
- Acknowledgments: The authors thank all contributors, including those not listed in the paper, for their support and efforts.
- Acknowledgments: Contributor names within each group are listed in no particular order.
- Contributors: The contributor list includes the named authors and marks some affiliations or roles with symbols.
A.1 Vision-Centric Benchmarks and Prompts
The appendix specifies zero-shot evaluation prompts and output formats for vision-centric tasks, covering localization, detection, classification, counting, segmentation, and depth estimation. It also documents open-world prompting and post-processing choices for dense outputs.
- Evaluation Setup: All vision-centric benchmark results use zero-shot evaluation without few-shot examples or chain-of-thought reasoning.
- Visual Grounding: Visual grounding prompts require localizing a queried object and returning its bounding-box coordinates, including flexible open-scene and Chinese prompt variants.
- Object Detection: COCO detection parses autoregressive outputs into object categories and boxes, assigning each predicted box a confidence score equal to its area for mAP computation.
- Object Detection: Open-world detection supports multiple requested categories and an optional detect-anything mode, which may miss less prominent categories.
- Image Classification: ImageNet-ReaL classification requires a single word or short phrase naming the dominant object category, matched against the ground-truth label.
- Object Counting: Counting benchmarks require an exact numeric count, with input images upsampled by a factor of 2 or 2.5.
- Semantic Segmentation: ADE20K segmentation outputs pixel-level masks as run-length encoded strings, reshaped to the raw image size for use.Temperature-scaled softmax and DenseCRF refine predictions when evaluating without background.
- Depth Estimation: NYUv2 depth estimation quantizes depths from 0–10 meters into 1–1000, then dequantizes predictions while excluding invalid depths.The protocol also uses category IDs, resizing, and prompt placement before the image.
A.2 General Multimodal Benchmarks
The appendix describes benchmark-specific evaluation, resolution, judging, and prompting procedures for general multimodal tasks. It combines dynamic high-resolution inputs, hybrid answer judging, selective chain-of-thought prompts, and constrained outputs tailored to each benchmark.
- Evaluation Framework: Evaluations use a refined VLMEvalKit framework with LLM-based judging when exact matching fails or outputs are ambiguous.
- Resolution Settings: Dynamic high-resolution inputs use benchmark-dependent patch limits, reaching a maximum of 65,536 patches for fine-grained tasks.
- Prompting Strategies: Complex reasoning and real-world analysis benchmarks selectively use chain-of-thought prompts, while hallucination and OCR tasks use concise output constraints.
- Reasoning Benchmarks: VisuLogic prompts require step-by-step reasoning followed by a boxed final answer.
- Reasoning Benchmarks: MathVerse, MathVista, and LogicVista use chain-of-thought prompting for structured geometric and mathematical derivations.
- Reasoning Benchmarks: MMMU configurations append standard chain-of-thought instructions after answer options to encourage detailed reasoning.
- General Perception: RealWorldQA, MMBench, and CV-Bench use step-by-step instructions, whereas MME-RealWorld uses a system prompt and answer-generating suffix.
- Constrained Outputs: HallusionBench and POPE enforce yes-or-no answers, while document and OCR benchmarks require a short phrase or single word for metric calculation.
B Extended Experiments and Comparison
Youtu-VL is evaluated against five representative vision-centric settings, spanning specialist models, CLIP-based models, vision generalists, augmented multimodal LLMs, and a standard multimodal LLM. Despite using no extra modules, heads, or task embeddings, it achieves competitive or state-of-the-art performance across diverse settings.
- Extended Experiments and Comparison: The evaluation compares Youtu-VL with five model settings, including specialist, CLIP-based, vision generalist, augmented multimodal, and standard multimodal architectures.The standard multimodal setup uses no extra modules, heads, or task embeddings.
- Extended Experiments and Comparison: Youtu-VL achieves competitive or state-of-the-art performance across diverse vision-centric settings despite its architectural simplicity.The results support using a standard MLLM as a universal predictor when equipped with appropriate supervision and training.
B.1 Comparison With Dense Prediction Methods
Youtu-VL performs strongly across dense prediction tasks, including semantic segmentation, depth estimation, and referring expression segmentation, without requiring complex architectural additions. It achieves competitive results against specialized and augmented alternatives while retaining a standard VLM architecture.
- Dense Prediction: Youtu-VL demonstrates strong performance across dense prediction benchmarks without requiring complex architectural additions.The evaluated tasks include semantic segmentation, depth estimation, and referring segmentation across multiple datasets.
- Semantic Segmentation: Youtu-VL achieves 54.2 mIoU on ADE20K, exceeding GiT’s 47.8 mIoU and SAN’s 32.1 mIoU.Unlike standard multimodal LLMs marked as inapplicable, Youtu-VL handles fine-grained dense prediction directly.
- Depth Estimation: Youtu-VL achieves 90.4 on NYUv2, compared with 87.6 for DepthLLM, while requiring only one image pass.After fine-tuning, it reaches 92.7 δ1 versus 92.1 for SwinMTL; UniDepthv2 remains higher on NYUv2, while DDAD results are close at 88.2 versus 87.6.
- Referring Expression Segmentation: Youtu-VL demonstrates strong performance in referring expression segmentation on RefCOCO.The section reports this capability alongside broader dense prediction results.
- Referring Expression Segmentation: Youtu-VL reaches 80.7 cIoU on RefCOCO val, compared with 79.5 for GLaMM and 80.5 for UniPixel.UniPixel requires an extra SAM decoder, while UFO requires extra mask token embeddings for retrieval.
B.2 Comparison on Localization Tasks
Youtu-VL achieves strong localization performance within a standard VLM framework, covering detection, visual grounding, and counting without complex architectural additions. Detection performance reaches 47.1 mAP on COCO and 48.0 mAP after detection-only supervised fine-tuning.
- Localization Tasks: Youtu-VL achieves superior localization performance without complex architectural additions through detection, counting, and grounding capabilities.These capabilities establish leading performance within a standard VLM framework.
- Object Detection: 48.0 mAP is achieved on the COCO validation split after supervised fine-tuning exclusively on detection data from Stage 3, versus 47.1 mAP otherwise.The 47.1 mAP result is reported for the COCO validation split, while detection-only fine-tuning raises it to 48.0 mAP.
- Visual Grounding: Youtu-VL secures leading results across all RefCOCO visual grounding splits.The localization comparison also evaluates counting tasks on CountBench, TallyQA-Simple, and TallyQA-Complex.
C Case Study
The case studies provide qualitative examples of Youtu-VL’s vision-centric capabilities across detection, open-world detection, counting, segmentation, grounding, referring segmentation, and depth estimation. These examples span COCO, Objects365, ADE20k, RefCOCO, and NYUv2.
- Detection: Qualitative examples cover object detection on the COCO dataset and open-world detection on Objects365.The figures include two COCO object-detection examples and one Objects365 open-world-detection example.
- Counting: A qualitative example demonstrates detect-then-count on a photograph taken by the Youtu-VL Team.The case study includes a dedicated figure for this capability.
- Segmentation: Qualitative examples cover semantic segmentation on ADE20k.Two figures present semantic-segmentation examples from the ADE20k dataset.
- Grounding: Qualitative examples cover visual grounding and referring expression segmentation on RefCOCO.Two figures show visual grounding, and one shows referring expression segmentation.
- Depth Estimation: Qualitative examples cover depth estimation on NYUv2.Two figures present depth-estimation examples from the NYUv2 dataset.
Human Pose Estimation
The paper presents a qualitative human pose estimation example using a generated image. The example is part of a broader set of demonstrations of Youtu-VL’s visual capabilities.
- The human pose estimation example uses a generated image as the visual input.
- The qualitative examples also cover creative visual-language generation, including composing a Chinese poem for a snowy sunrise.
- Additional examples demonstrate mathematical reasoning, chart understanding, fine-grained perception, formula recognition, and GUI interaction.