PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision as Unified Multimodal Generation

Xiaoyang Han, Jianhua Li, Kewang Deng, Zukai Chen, Xuanke Shi, Sihan Wang, Boxuan Li, Linyan Wang, Siyi Xie, Xin You, Jinsheng Quan, Zhongang Cai, Haiwen Diao, Ziwei Liu, Lei Yang, Dahua Lin, Quan Wang

arXiv:2607.06560v1cs.CV

TL;DR

Computer vision lacks a unified approach spanning heterogeneous tasks without task-specific architectures. SenseNova-Vision casts these tasks as text, image, or mixed multimodal generation and matches leading task-specialized systems across four perception families.

  • Problem

    Computer vision remains organized around specialist systems with task-specific architectures, losses, decoding rules, and evaluation protocols across heterogeneous outputs.

  • Method

    SenseNova-Vision converts diverse vision annotations into instruction-response targets and trains one unified multimodal model to generate text, images, or mixed outputs without task-specific heads.

  • Results

    SenseNova-Vision matches leading task-specialized systems across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry.

  • Takeaways & Limitations

    Unified multimodal generation provides a single framework for expressing broad computer vision capabilities within general-purpose foundation models.

  • Takeaways & Limitations

    The authors identify stronger in-context learning, video extension, and larger corpora and models as future directions.

Abstract

from arXiv · show

We formulate computer vision as unified multimodal generation, where heterogeneous visual tasks are expressed in the native text and image generation spaces of a unified multimodal model, without task-specific architectures. Under this formulation, SenseNova-Vision uses natural-language instructions and optional visual prompts to specify tasks, target regions or views, and decoding conventions, and generates responses as text for symbolic outputs, images for dense spatial predictions, or mixed text-and-image outputs for compositional tasks. To support large-scale training, we convert diverse computer vision annotations into instruction-response examples compatible with these generation spaces, resulting in the SenseNova-Vision Corpus, a computer-vision instruction-response corpus spanning text, image, and mixed targets. Starting from an off-the-shelf pretrained unified multimodal model, SenseNova-Vision is trained primarily on this corpus, with auxiliary multimodal data used as a capability-preserving mixture, and requires no task-specific prediction heads or architectural modifications. The resulting model covers a broad range of vision tasks, including detection, OCR, keypoint estimation, segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation, while supporting language-defined variants that combine category, color, region, and other visual cues. Experiments show that a single unified model can match leading task-specialized systems across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. These results suggest unified multimodal generation as a scalable route for integrating computer vision capabilities into general-purpose foundation models. The model and corpus are publicly available.

1 Introduction

SenseNova-Vision formulates heterogeneous computer vision as unified multimodal generation within a single unified multimodal model, using native text, image, and mixed targets without task-specific heads. It introduces an instruction-response corpus that enables this formulation to cover structured understanding, dense prediction, segmentation, and multi-view visual geometry.

  • Unified formulation: SenseNova-Vision casts heterogeneous computer vision tasks into native text, image, and mixed generation spaces within one unified multimodal model, without task-specific heads.Natural-language instructions specify tasks, targets, output schemas, and decoding conventions.
  • Unified formulation: Text generation expresses symbolic outputs such as categories, spatial references, OCR strings, and camera parameters, while image generation represents dense predictions.Dense outputs include masks, depth maps, surface normals, and point maps.
  • Training corpus: The SenseNova-Vision Corpus converts heterogeneous computer vision annotations into instruction-response examples spanning structured understanding, dense geometry, segmentation, and multi-view visual geometry.Targets are decodable as boxes, masks, depth maps, surface normals, point maps, and camera poses.
  • Training corpus: Built from the off-the-shelf UMM Bagel, SenseNova-Vision is trained primarily through the unified multimodal generation interface illustrated in Figure 3.The approach uses the corpus to express diverse visual task outputs through a shared interface.
  • Contributions: The paper presents unified multimodal generation as a route for making computer vision a native generative capability of unified foundation models rather than isolated task-specific systems.The contributions include the formulation, the large-scale instruction-response corpus, and decodable text, image, and mixed targets.

2 Related Works

Prior work unifies vision through shared sequence formats, pretrained visual representations, promptable perception, image generation, and multimodal language-vision models. SenseNova-Vision builds on unified multimodal models by expressing heterogeneous computer-vision annotations as native text, image, and mixed generation targets.

  • Unified vision models: Early unified vision models broadened sequence-based interfaces from object detection to diverse task families, but remained dependent on shared formats and task-specific encoding rules.Pix2Seq, Pix2Seq v2, UniTAB, OFA, and Uni-Perceiver exemplify this progression, whose interfaces are less natural for dense maps and structurally diverse outputs.
  • Visual foundation models: Visual foundation models provide reusable representations and promptable or in-context mechanisms for recognition, dense prediction, geometry transfer, segmentation, and image-to-image perception.MAE and DINOv2 provide pretrained features, while SAM, Painter, and SegGPT support prompt- or example-conditioned visual tasks.
  • Generative dense prediction: Diffusion and image-generation models recast dense prediction as spatially aligned visual generation, covering monocular depth, surface normals, and multiple perceptual tasks.Marigold, Lotus, Lotus-2, and FE2E adapt generative priors for geometry, while DICEPTION uses conditional image generation in a shared RGB output space.
  • Multimodal language-vision models: MLLMs extend language generation to structured grounding and dense spatial outputs through boxes, coordinates, regions, points, discrete tokens, and logits.Kosmos-2, Shikra, Ferret, and Rex-Omni address grounding, while Text4Seg and DenseMLLM represent dense outputs in discrete spaces.
  • Unified multimodal generation: Recent unified multimodal models natively generate text and images, motivating a unified computer-vision formulation with decodable native text, image, and mixed targets.SenseNova-SI studies spatial-intelligence supervision, whereas this work focuses on converting heterogeneous computer-vision annotations into benchmark-decodable generation targets.

3 Data

Section 3 presents a protocol that converts heterogeneous computer-vision annotations into instruction-response examples in text, image, or mixed text-image generation spaces. It organizes these examples into the four-family SenseNova-Vision Corpus, including the released SN-VC-50M collection with generated or curated targets for incomplete supervision.

  • Source families: SN-VC organizes converted supervision into structured understanding, dense geometric prediction, segmentation, and multi-view visual geometry.These families cover tasks such as detection, OCR, keypoints, depth, surface normals, masks, point maps, and camera poses.
  • Corpus construction: SN-VC is built from public images and annotations, with additional targets generated or curated when direct conversion is impossible or supervision is incomplete.The released SN-VC-50M subset contains 50 million converted computer-vision supervision examples.
  • Data protocol: The data protocol represents each sample with visual inputs, a natural-language instruction, and a decodable target response.Instructions specify task intent, output schema, and decoding conventions; responses recover benchmark-compatible labels, coordinates, masks, dense maps, or camera parameters.
  • Instruction-response conversion: The protocol selects single images, prompted images, or ordered image sets as inputs and deterministically renders text, image, or mixed outputs according to task requirements.Text-oriented tasks use normalized schemas, image-oriented tasks render masks or geometric maps, and mixed tasks place components in a fixed order.

4 Training

SenseNova-Vision adapts Bagel-7B-MoT through mixed-task supervised fine-tuning on the SenseNova-Vision Corpus and auxiliary multimodal data. Training interleaves text and visual supervision while using high-resolution image conditioning and multi-view scene samples for spatial and geometric tasks.

  • Mixed-task fine-tuning: SenseNova-Vision adapts the pretrained Bagel-7B-MoT checkpoint instead of training a multimodal model from scratch.The adaptation targets benchmark-compatible computer vision abilities while mitigating degradation in image understanding, instruction following, and image generation.
  • Mixed-task fine-tuning: Supervised fine-tuning combines the SenseNova-Vision Corpus with auxiliary multimodal data spanning VQA, text-to-image, and image-to-image tasks.The mixture supports structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry.
  • Mixed-task fine-tuning: A weighted joint-sampling strategy interleaves converted vision samples and auxiliary multimodal samples, producing mixed text and visual supervision within the same optimization process.Mini-batches may contain multiple task categories, while text-form outputs and visual targets are trained together despite different objectives.
  • High-resolution and multi-view training: For fine spatial conditioning, SigLIP2 input resolution is retained up to 980 pixels for understanding and image-conditioned generation, especially for segmentation.The high-resolution pathway preserves conditioning when generation depends on an input image.
  • High-resolution and multi-view training: Multi-view training randomly selects at most 10 views per scene because of memory constraints, with view alignment and normalization applied for point-map reconstruction.Camera pose estimation uses a dedicated set of special vocabulary entries repurposed from the base model.
  • Training setup: SFT freezes the VAE visual encoder while updating other modules and connectors, using AdamW with a learning rate of 2.5 × 10−5 and no weight decay.Training uses 32K–36K tokens per rank, a maximum 32K context window, dropout rates of 0.05 for text and 0.1 for ViT and VAE input tokens, and 50K steps.

5 Experiments

Experiments show that SenseNova-Vision performs strongly across structured visual understanding, dense geometry, segmentation, and multi-view geometry while using unified text, image, and mixed outputs. Qualitative probes further show that recombining cross-modal supervision enables language-defined task variants beyond explicit training protocols, while largely preserving pretrained multimodal abilities.

  • Structured visual understanding: SenseNova-Vision achieves strong performance on coordinate-level structured prediction, including dense, long-tailed, small-object, referring, and OCR localization benchmarks.These tasks require serialized generation of long object lists, stable ordering, and precise coordinates.
  • Dense geometric prediction: SenseNova-Vision produces strong depth and normal estimation without task-specific prediction heads, outperforming generation-based baselines on several benchmarks and remaining competitive with geometry-specialized models.The evaluation tests dense geometric maps generated directly as image outputs.
  • Segmentation: SenseNova-Vision is competitive among unified segmentation and multimodal baselines, though specialized models remain stronger on several generic and referring segmentation metrics.The tasks evaluate language-conditioned target selection and benchmark-compatible mask generation.
  • Multi-view visual geometry: SenseNova-Vision performs strongly among generalist geometric approaches, especially on ETH3D reconstruction and camera pose estimation, but trails feed-forward geometric models on several metrics.The evaluation tests cross-view alignment and view-specific outputs for reconstruction and pose estimation.
  • Generalist comparisons and capability retention: SenseNova-Vision retains core multimodal abilities after mixed-task fine-tuning, scoring 79.0 on MMVP versus 83.3 for Bagel and 0.85 on GenEval versus 0.82 for Bagel.It performs strongly against other generalist models across structured, semantic, depth, segmentation, and dense-prediction benchmarks.
  • Cross-modal task recombination: Qualitative probes show that unified generation recombines recognition, localization, grounding, OCR, and rendering into language-defined mask tasks with text or image spatial cues.The model extracts nearly one hundred dense objects, grounds textual points or boxes, follows free-form color descriptions, and segments queried words or letters, though outputs remain qualitative and imperfect.

6 Conclusion · Appendix · A SenseNova-Vision Corpus Construction Details

SenseNova-Vision presents unified multimodal generation as a computer-vision formulation and trains one model by converting heterogeneous annotations into text, image, and mixed generation targets. The appendix details the corpus protocol, while the conclusion identifies in-context learning, video extension, and scaling as future directions.

  • 6 Conclusion: Unified multimodal generation is proposed as a computer-vision formulation analogous to GPT-style generative modeling in NLP.
  • 6 Conclusion: The SenseNova-Vision Corpus converts heterogeneous computer-vision annotations into text, image, and mixed text-and-image generation targets.
  • 6 Conclusion: This conversion enables large-scale training of a single unified multimodal model across structured visual understanding, dense geometric prediction, and segmentation.
  • 6 Conclusion: Stronger in-context learning could reduce task-domain boundaries by specifying new visual tasks through examples, prompts, or mixed demonstrations.
  • 6 Conclusion: Extending unified multimodal generation from images to video could incorporate temporal dynamics and web-scale video supervision into foundation-model training.
  • 6 Conclusion: Scaling the corpus and model capacity, together with deeper improvements, is identified as a future-work direction.
  • Appendix: Each corpus training example contains one or more visual inputs, a natural-language instruction, and a target response.
  • A SenseNova-Vision Corpus Construction Details: Target responses are represented as text, an image, or a mixed text-and-image output, with the appendix documenting composition, processing, prompts, and release resources by task family.

A.1 Structured visual understanding

Structured visual understanding covers detection, OCR, layout, GUI grounding, referring detection, and keypoint estimation through instruction-response data. The pipeline unifies spatial annotations with normalized coordinates, data engines, and task-specific prompts, and releases generated or converted examples.

  • Task coverage: Structured visual understanding spans bounding-box, point, visual-prompt, referring, OCR, layout, GUI, and keypoint detection.Bounding-box, point, and visual-prompt detection datasets are integrated, while other tasks use separately constructed datasets.
  • Spatial representation: Detection-related coordinates are normalized by image width and height, rounded to three decimals, clipped to [0.000, 0.999], and shared across boxes, points, and keypoints.This representation reduces discrepancies caused by differing image resolutions and annotation formats.
  • Data construction: Data engines generate large-scale, high-quality annotations for challenging scenarios, multi-task learning, and visual-spatial correspondences.These engines are used to improve training-data diversity and quality.
  • Prompt design: Each sample pairs an input image and task instruction with a textual target containing semantic labels and spatial annotations such as boxes, points, polygons, or keypoints.OCR samples additionally include recognized text, and the <image> placeholder marks the visual input location.
  • Prompt design: Task-specific instruction templates are randomly selected from pools during dataset construction, with the complete template collection summarized in Table 9.A template is selected for each distinct task.
  • Released datasets: Generated or converted training examples produced by the data engines are released because most source datasets are publicly available.The released examples support structured visual-understanding training.

A.2 Dense geometric prediction · A.3 Segmentation

Dense geometric prediction is converted into standardized image-to-image generation using real and MoGe-2 pseudo-labeled data, while segmentation unifies binary-target and comprehensive panoptic-mask outputs. Both sections define task-specific data representations, processing pipelines, and prompts for multimodal generation.

  • A.2 Dense geometric prediction: Dense geometric prediction uses synthetic datasets for dense supervision and MoGe-2 to generate pseudo labels from sparse or imperfect LiDAR annotations.Additional related-domain datasets use the same MoGe-2 procedure to strengthen generalization and real-world performance.
  • A.2 Dense geometric prediction: Depth data are filtered within 0.1 m to 80 m, discard annotations exceeding 1 percent invalid pixels, and use a 10 percent threshold for MoGe-2 pseudo labels.Depth supervision uses inverse-depth maps converted to standardized three-channel RGB images.
  • A.2 Dense geometric prediction: Surface-normal training excludes images with sky coverage above 10 percent and stores valid X, Y, and Z normal components in RGB channels.Filtering is applied only to MoGe-2 pseudo labels.
  • A.2 Dense geometric prediction: Dense geometric prediction is formulated as an image-to-image task following InternVL’s prompt-construction paradigm.The released resources are training examples generated by MoGe-2, including datasets with sparse or no annotations.
  • A.3 Segmentation: Segmentation datasets are divided into binary-mask tasks for single foreground targets and panoptic-mask tasks assigning dense semantic and instance labels across the image.The binary group includes referring, reasoning, and interactive segmentation; the panoptic group includes generic and GCG segmentation.
  • A.3 Segmentation: Binary segmentation annotations use white foreground pixels, (255, 255, 255), and black background pixels, (0, 0, 0).Generic and GCG segmentation instead decomposes instance annotations into individually colored masks blended over a black background.
  • A.3 Segmentation: Segmentation prompts specify the target, output format, and required interaction, with binary-mask tasks storing mask file paths through a dedicated Binary_mask field.The conversion pipeline supports reconstruction of binary-mask datasets, while generated and curated generic and GCG resources are released.

A.4 Multi-view visual geometry

Multi-view visual geometry training combines scene-level reconstruction and camera-pose data represented as RGB images, depth maps, point maps, and camera trajectories. The pipeline uses dense-depth filtering, metadata-rich prompts, and publicly released preprocessing resources to support reconstruction and pose estimation.

  • Data Composition: Multi-view reconstruction and camera-pose estimation draw on closely related geometric datasets, with LingBot-Depth completing sparse depth data to generate dense point maps.Both tasks require dense geometric supervision, and the adopted datasets are summarized in Table 16.
  • Data Processing: Each reconstruction scene contains an image sequence, depth maps, and camera trajectories convertible to point maps in a shared coordinate frame.Inputs are filtered to retain only views whose depth maps contain no more than two-thirds invalid pixels.
  • Prompt Design: Reconstruction examples are generated online, with scene-level dataset metadata embedded directly into the prompt.The prompt structure is designed specifically for multi-view reconstruction rather than relying on masking strategies.
  • Released Datasets: Camera-pose examples can be assembled from public multi-view datasets with camera trajectories, while released preprocessing supports sparse and dense reconstruction data.Sparse datasets are preprocessed with LingBot-Depth, and dense datasets can be reconstructed using the released conversion pipeline.

A.5 SN-VC-50M Release Summary

The SN-VC-50M release covers four task families, 73 dataset-task entries, and 10 task types across structured understanding, segmentation, dense geometry, and multi-view geometry.

  • Release scope: SN-VC-50M spans 73 dataset-task entries across 10 task types and four task families.The release groups resources into structured visual understanding, segmentation, dense geometric prediction, and multi-view visual geometry.
  • Release scope: 18.9M frames support structured visual understanding, 1.3M frames support segmentation, 17.3M frames support dense geometric prediction, and 12.5M frames support multi-view visual geometry.These four frame counts correspond to the released task families.
  • Release format: Released training examples retain image file paths rather than redistributing raw RGB images from public datasets.This design avoids duplicating raw RGB images from public datasets.

B Additional Qualitative Results

Additional qualitative examples demonstrate SenseNova-Vision’s breadth across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry, using text, image, and mixed outputs. They also show that one model follows varying task instructions while maintaining decodable outputs under the prescribed protocols.

  • Additional qualitative results: Qualitative examples span structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry across text, image, and mixed text-and-image tasks.The examples complement the main quantitative evaluation and illustrate the breadth of SenseNova-Vision.
  • Additional qualitative results: The same model follows different task instructions while keeping outputs decodable under the protocols described in Appendix A.This demonstrates instruction-conditioned behavior across the showcased task types.
  • Additional qualitative results: Additional figures present qualitative results for structured visual understanding, dense geometric prediction, and segmentation.These examples are shown in Figures 14–16.
  • Additional qualitative results: Figure 17 provides additional qualitative results for multi-view reconstruction.The figure compares input views with reconstruction outputs.

C Exploratory Results on Challenging and Language-Defined Vision Tasks

This section presents exploratory qualitative tests of language-defined vision tasks beyond the main quantitative evaluation. The results span composed dense prediction, specialized segmentation formats, fine-grained visual grounding, and challenging geometric scenes.

  • Overview: The exploratory suite evaluates language instructions that specify both the vision objective and expected output format across challenging and composed tasks.Examples include composed dense prediction, specialized mask formats, fine-grained segmentation, and dense geometry.
  • Composed Dense Prediction: Under fixed image and instruction inputs, different text-image CFG settings can produce depth-like, normal-like, segmentation-like, or compositionally consistent predictions.Figure 18 compares qualitative outputs under different classifier-free guidance strengths.
  • Specialized Segmentation Formats: Hexadecimal-color segmentation generally localizes target regions, but generated masks can deviate from requested RGB values or assign incorrect colors.The task combines semantic segmentation with language-specified color formatting.
  • Specialized Segmentation Formats: Dense instance segmentation requires structured text to enumerate instances and assign colors, yet outputs may use detection-style records, over-predict instances, or include non-target regions.The requested format interleaves instance labels, RGB colors, and masks.
  • Fine-Grained Visual Grounding: In visual-grounded segmentation, insufficient caption color slots can merge dense same-category objects, while manually adding sufficient entries improves instance-level separation.The visual prompt specifies the target category and the model segments all same-category instances.
  • Challenging Dense Geometry: Dense geometric predictions remain ambiguous on visual illusions and transparent or reflective surfaces when appearance cues conflict with physical geometry.Reflections, textures, and distorted patterns can mislead depth or normal predictions, especially around reflective layers.
Loading 2607.06560v1…