PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action

Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, Aniruddha Kembhavi

arXiv:2312.17172v1cs.CVcs.AIcs.CL

TL;DR

Multimodal models face substantial challenges in jointly handling diverse inputs, outputs, and training signals. Unified-IO 2 uses a shared token space, one encoder-decoder transformer, from-scratch multimodal pretraining, and instruction tuning across 120 datasets. It achieves state-of-the-art GRIT performance and strong results across more than 35 benchmarks, while remaining less reliable for depth, video, and niche abilities.

  • Problem

    Building models that parse and produce many modalities remains difficult, especially when training broad-coverage generative systems from scratch.

  • Method

    Unified-IO 2 tokenizes modalities into a shared semantic space, processes them with one encoder-decoder transformer, and trains from scratch with multimodal denoising plus instruction tuning.

  • Results

    Unified-IO 2 achieves state-of-the-art performance on GRIT and strong results across more than 35 benchmarks spanning multiple modalities and tasks.

  • Takeaways & Limitations

    A single autoregressive model can understand and generate image, text, audio, and action while following free-form multimodal instructions.

  • Takeaways & Limitations

    The model is less reliable for depth, video, and niche abilities, and image-generation quality does not match stable diffusion models.

Abstract

from arXiv · show

We present Unified-IO 2, the first autoregressive multimodal model that is capable of understanding and generating image, text, audio, and action. To unify different modalities, we tokenize inputs and outputs -- images, text, audio, action, bounding boxes, etc., into a shared semantic space and then process them with a single encoder-decoder transformer model. Since training with such diverse modalities is challenging, we propose various architectural improvements to stabilize model training. We train our model from scratch on a large multimodal pre-training corpus from diverse sources with a multimodal mixture of denoisers objective. To learn an expansive set of skills, such as following multimodal instructions, we construct and finetune on an ensemble of 120 datasets with prompts and augmentations. With a single unified model, Unified-IO 2 achieves state-of-the-art performance on the GRIT benchmark and strong results in more than 35 benchmarks, including image generation and understanding, natural language understanding, video and audio understanding, and robotic manipulation. We release all our models to the research community.

1. Introduction

UNIFIED-IO 2 addresses the difficulty of building broad multimodal systems with one model trained from scratch across diverse data and tasks. It combines unified multimodal processing with training and instruction-tuning methods, achieving strong results across more than 35 datasets.

  • Motivation: Multimodal models must perceive, communicate, act, and reason across vision, language, sound, and action.The paper motivates modalities as potentially mutually supervisory during training.
  • Challenges: Adding modalities amplifies challenges in data sourcing, quality, bias management, architecture, training stability, and instruction tuning.These difficulties already affect single-modality language models and increase with each additional modality.
  • Research Gap: Training broad-coverage generative multimodal models from scratch remains an open challenge.Prior work commonly extends pretrained language models or supports only limited modality combinations.
  • Approach: UNIFIED-IO 2 encodes text, images, audio, video, and interleaved sequences while producing text, actions, audio, images, and dense or sparse labels.The 7-billion-parameter model is pretrained from scratch on diverse multimodal data and instruction-tuned on more than 120 datasets covering 220 tasks.
  • Training: A multimodal mixture-of-denoisers objective and dynamic packing address diverse self-supervision and variable sequence lengths.Dynamic packing provides a 4x increase in training throughput.
  • Results: More than 35 datasets show state-of-the-art GRIT performance and strong results across vision-language, image generation, video, natural language, audio, and embodied AI tasks.The model also follows free-form instructions, including novel ones.

2. Related Work

Prior multimodal systems broaden task and modality coverage through adapters, expert modules, diffusion components, or pretrained language models. UNIFIED-IO 2 extends this direction with wider multimodal generation and a single model trained from scratch.

  • Pretrained-Language-Model Systems: Many multimodal systems adapt pretrained language models using vision encoders, modality-specific decoders, or modular frameworks.These designs commonly map modality features into language-model inputs or combine specialized components.
  • Multimodal Generation: Earlier multimodal-generation models produce images through VQ-GAN tokens, diffusion features, or fused language-and-image generators.UNIFIED-IO 2 also uses VQ-GAN while additionally supporting text, image, and audio generation.
  • Scope: UNIFIED-IO 2 pushes modality and task breadth toward a single any-to-any generative model.CoDi achieves similar any-to-any generation using independently trained diffusion models with aligned embedding spaces.
  • Training Regime: Unlike many prior from-scratch models, UNIFIED-IO 2 supports complex free-form generations and text instruction following.The comparison concerns models that are typically not designed for free-form text, image, or sound generation.
  • Training Regime: Training from scratch avoids a costly preliminary language-model pretraining stage and fits simultaneous learning across co-occurring modalities.The paper contrasts this with learning modalities one at a time.

3. Approach

UNIFIED-IO 2 uses a single encoder-decoder transformer and shared token space to process and generate diverse modalities. Its training combines multimodal denoising, dynamic packing, and architectural changes intended to address instability and efficiency challenges.

  • Unified task representation: UNIFIED-IO 2 encodes images, text, audio, action, and other inputs or outputs as tokens in a shared representation space for one encoder-decoder transformer.The model can use modality-specific encoders and decoders while retaining a unified transformer core.
  • Unified task representation: Inputs including text, images, audio, and multimodal history are converted into embeddings, concatenated, and decoded into discrete text, image, or audio tokens.The architecture also supports image or audio history as contextual input.
  • Architecture and stability: Adding video and further modalities makes standard training increasingly unstable, with the reported video-inclusive mixture producing pronounced gradient and loss problems.Image-only training is stable, while adding image and text slightly increases gradient norms but remains stable before video is introduced.
  • Architecture and stability: The proposed stability measures include query-key normalization, float32 attention logits, and freezing the pretrained ViT and AST during pretraining.These choices target saturated attention weights, numerical instability, and instability from jointly updating modality encoders.
  • Training objective: The multimodal mixture of denoisers combines masked denoising and causal generation paradigms for text, images, and audio while using modality and paradigm tokens.Dynamic masking is introduced to preserve causal generation without decoder-side information leakage during image and audio denoising.
  • Efficiency optimizations: Dynamic packing rearranges variable-length multimodal examples so multiple examples share transformer sequences, improving training efficiency for heterogeneous data.Packing occurs around the encoder-decoder stage, allowing modality encoders and decoders to operate on unpacked data.

4. Multimodal Data

Unified-IO 2 curates diverse open-source multimodal data for pre-training and instruction tuning. The data span text, video, audio, 3D, embodiment, prompting, augmentation, and multimodal task demonstrations.

  • Unified-IO 2 trains from scratch on curated open-source multimodal data for both pre-training and instruction tuning.
  • Pre-training Data: Video samples are built by extracting frames, audio spectrograms, and transcripts, then randomly selecting modalities, objectives, and masks for the input-target pair.
  • Pre-training Data: The pre-training mixture includes NLP, video and audio, 3D and embodiment, and augmented annotation data.Video and audio comprise 25% of the mixture, while 3D and embodiment and augmentation each comprise 1%.
  • Instruction Tuning Data: Instruction tuning combines 220 tasks from over 120 datasets, including open-ended and synthetic tasks for broader multimodal skill coverage.
  • Instruction Tuning Data: Figure 8 illustrates one model producing outputs for captioning, instruction following, editing, detection, segmentation, surface normals, and image-based audio generation.
  • Instruction Tuning Data: The instruction-tuning mixture contains 60% prompted supervised data, 30% carried-over pre-training data, 6% task augmentation, and 4% free-form text.The carried-over pre-training data is included to avoid catastrophic forgetting.

5. Experiments

UNIFIED-IO 2 is evaluated without task-specific finetuning across a broad range of multimodal understanding and generation tasks. It achieves state-of-the-art GRIT performance and competitive or superior results across vision-language, generation, audio-video, and 3D tasks, while showing weaknesses in depth estimation and some data-sparse settings.

  • Evaluation scope: UNIFIED-IO 2 is evaluated on over 35 datasets spanning image, video, audio, text, and action tasks without task-specific finetuning.The evaluation uses pre-trained and instruction-tuned models directly.
  • Pre-training evaluation: Comparable or better zero-shot performance is reported on HellaSwag, TIFA, Seed-Bench, and AudioCaps versus specialist or universal multimodal models.These evaluations cover sentence completion, text-to-image generation, spatial and temporal comprehension, and text-to-audio generation.
  • GRIT results: 2.7 points: UNIFIED-IO 2 surpasses UNIFIED-IO overall on GRIT, with gains in localization, categorization, segmentation, and keypoint estimation.The model is evaluated on GRIT’s seven tasks, which require multimodal inputs and sparse or dense outputs.
  • GRIT results: 65.2 vs. 64.5: the 3-billion-parameter UNIFIED-IO 2 model maintains better overall GRIT performance than UNIFIED-IO.Ablations report improvements across average performance and individual tasks with increasing model size.
  • Vision-language results: UNIFIED-IO 2 matches or surpasses generalist vision-language models while also supporting high-quality image and audio generation.It achieves strong VQA, referring-expression, captioning, MMB, SEED-Bench, and POPE results; its CIDEr score is 130.3.
  • Video and audio results: 8.5 points: UNIFIED-IO 2 outperforms BLIP-2 and InstructBLIP on Seed-Bench Temporal, while outperforming MBT on Kinetics-Sounds.The model also shows reasonable audio and video classification, captioning, and video question-answering performance.
  • Other results: UNIFIED-IO 2 performs decently on single-object 3D detection but drops significantly on multi-object 3D detection, where only 1.0% of training data is devoted to 3D detection.The authors suggest combining 2D and 3D detection techniques as a potential solution.
  • Other results: 0.623 RMSE: depth estimation on NYUv2 is weak, improving to 0.423 after task-specific finetuning.The authors attribute the issue to incompatible dense-depth ground-truth scales and per-dataset normalization.

6. Limitation

The authors identify limitations from constrained encoders, limited compute and data diversity, and weaker performance in depth, video, and niche abilities.

  • Model and compute constraints: Base ViT and AST encoders were used because of memory constraints, potentially limiting image and audio performance.The authors state that larger image and audio encoders could substantially improve results.
  • Generation limitations: Image generation is more faithful than stable-diffusion-based methods but does not match their quality, while audio generation is limited to approximately 4 seconds.The audio-duration limit restricts practical applications of the generated outputs.
  • Model and compute constraints: Limited computational resources constrained hyperparameter exploration, and larger batch sizes might improve performance.
  • Data and task coverage: The model is less reliable for depth, video, and niche abilities such as 3D object detection because task variety in these areas is limited.
  • Instruction diversity: Human-written prompts remain insufficiently diverse, and performance decreases on new instruction tasks compared with tasks represented during instruction tuning.

7. Conclusion

UNIFIED-IO 2 unifies multimodal understanding and generation in a single autoregressive model trained from scratch and refined with instruction tuning. The authors report promising results across many tasks and identify decoder-only scaling, larger models, better data, and refined design as future directions.

  • Conclusion: UNIFIED-IO 2 understands and generates image, text, audio, and action as a single autoregressive multimodal model.
  • Conclusion: The model is trained from scratch on broad multimodal data and refined through instruction tuning on a massive multimodal corpus.
  • Conclusion: Architectural changes stabilize multimodal training, while a multimodal mixture-of-denoisers objective uses multimodal training signals.
  • Conclusion: UNIFIED-IO 2 achieves promising results across a wide range of tasks.
  • Future work: Future work will extend the encoder-decoder model toward decoder-only architecture, increase model size, improve data quality, and refine model design.

B.1. Detailed of Unified Task Representation

Unified-IO 2 represents heterogeneous modalities as token sequences for a shared encoder-decoder, while adding modality-specific encoders, decoders, positional schemes, and efficient packing. These design choices address representation, sequence-length, and training-efficiency challenges in the multimodal setting.

  • Unified representations: Text, sparse structures, and actions are encoded as token sequences in a shared representation space, including normalized coordinates and discretized continuous values.The representation uses a 32,000-token BPE vocabulary plus special tokens and 1,000 tokens for continuous values such as points, boxes, camera transformations, and 3D cuboids.
  • Visual representations: Images use ViT features for inputs and discrete VQ-GAN tokens for generation, while dense labels are represented as RGB or grayscale images.Depth, surface normals, and segmentation masks are converted into image-like representations that the model can encode or generate.
  • Position and history encoding: History inputs assign two-dimensional positions to image, audio, and frame-latent tokens, combining rotary encoding with learnable modality embeddings.The implementation uses separate temporal and latent-vector indices for history representations and applies rotary encoding to relative positions.
  • Dynamic packing: Dynamic packing reduces multimodal padding by packing multiple examples after modality-specific preprocessing and masking cross-example attention.The packed implementation uses matrix multiplication with one-hot matrices for TPU execution.
  • Dynamic packing: Packing typically combines two examples into input length 864 and target length 1280, producing roughly a 4x speedup.The gain comes from reduced sequence length and simultaneous processing of two examples, although streaming data cannot be packed reliably.

C. Pre-Training Details

Pre-training combines diverse text, image, video, interleaved, multiview, and embodied data with modality-targeting objectives. The corpus supports self-supervised learning across modalities and includes tasks for visual, 3D, and action understanding.

  • Data mixture: Pre-training samples present modalities as potential targets and balances corpus usage so output modalities are represented across datasets.The distribution is summarized in the pre-training data mixture and its sampling-rate visualization.
  • Data sources: The corpus includes text, image-text pairs, 180M videos, and interleaved image-caption data, with video examples targeting text, audio, or image modalities.Video examples use up to five frames, with earlier frames encoded as history and the final frame as the current input.
  • Interleaved data: Interleaved examples train caption denoising, image-conditioned caption generation, and sequences of captions linked to multiple images through marker tokens.These tasks are designed to teach image-history semantics and image-reference markers.
  • Multi-View: Multiview pre-training uses cross-view completion and Objaverse view synthesis, including in-context examples of views and camera transformations.Both tasks target 3D understanding during pre-training.
  • Agent Trajectory: Embodied pre-training uses ProcTHOR and Habitat trajectories for next-frame, next-position, and next-action prediction from visual observations, history, and actions.The trajectories are subsampled near the target object to form short multimodal prediction contexts.
  • Synthetic: Synthetic tasks add segmentation and sparse-coordinate prediction, including counting artificial image patches.These augment the pre-training signals for spatial outputs and counting.

D. Instruction Tuning Details

Instruction tuning broadens Unified-IO 2's capabilities through a balanced mixture of prompted supervised, open-ended, synthetic, and augmented tasks spanning language, generation, understanding, labeling, and dense prediction.

  • D. Instruction Tuning Details: The instruction-tuning mixture contains 220 tasks from over 120 datasets, with broad task categories sampled evenly and datasets generally sampled by the square root of size.Hand-engineered adjustments downweight noisy datasets and upweight rare tasks.
  • D.1. Natural Language: Natural-language tuning uses the FlanV2 mixture and continues unsupervised NLP pretraining to reduce forgetting during instruction tuning.The mixture also includes data from Muffin, T0-SF, NIV2, CoT annotations, Alpaca, Dolly, Open Assistant, and MDPP.
  • D.2. Image Generation: Image-generation tuning covers image-text pairs, localized narratives, captions, view synthesis, editing, dense controls, segmentation-based generation, and inpainting.Prompts distinguish approximate unsupervised image matches from stylistically specified supervised outputs.
  • D.3. Audio Generation: Audio-generation tuning uses captioned sound, music, and speech datasets plus video next-frame prediction to generate four-second audio segments.The prompts specify target audio characteristics, and inference uses top-p sampling without classifier-free guidance.
  • D.4. Image Understanding: Image-understanding tasks include VQA, tagging, region classification, visual instruction following, image-pair QA, grounded relationships, and captioning.The data combines M3IT, MIMIC-IT, GQA, TallyQA, OK-VQA, A-OKVQA, OCR-VQA, Visual Genome, ScienceQA, VCR, VizWiz, and classification datasets.
  • D.4. Image Understanding: Prompt style controls output behavior, such as requesting short VQA answers and specifying the expected class type for image tagging.The authors report that these hints are critical for preserving longer, more natural responses to user questions.
  • D.5. Image Sparse Labelling: Sparse-labeling tasks train object detection, localization, referring expressions, 3D boxes, camera pose, text detection, and human keypoints.Outputs include normalized coordinates, projected 3D properties, camera pose, text boxes, and visible or missing keypoints.
  • D.6. Image Dense Labelling / D.11. Task Augmentation: Dense-labeling tasks cover surface normals, depth, optical flow, and segmentation, while task augmentation varies category sets, spatial constraints, box properties, and relationships.Augmented outputs include box dimensions, areas, locations, edge distances, centers, intersections, and enclosing boxes.

E.1. Pre-training Visualization

Qualitative and benchmark evaluations examine Unified-IO 2's multimodal generation and language understanding. The pre-trained model produces audio, video-conditioned audio, future frames, and images, while NLP results place it between the cited LLaMA baselines.

  • E.1. Pre-training Visualization: The pre-trained model generates speech from text, music matching video input, future frames from images and actions, and images from prompts.These qualitative samples are presented as evidence of learned multimodal generation capabilities.
  • E.2. NLP Results: NLP evaluation uses zero-shot LM-Eval tasks including HellaSwag, MMLU, ARC, and BoolQ with default prompts plus modality prefixes.Results are reported in Table 14.
  • E.2. NLP Results: UNIFIED-IO 2 is generally ahead of Open LLaMA 3B but behind LLaMA on the evaluated NLP tasks.The comparison is reported as the overall positioning of the model relative to the two cited baselines.

E.3. GRIT Details

UNIFIED-IO 2 achieves strong GRIT localization and segmentation results, but performance varies across VQA subsets and crowded localization settings.

  • UNIFIED-IO 2 is the first unified model to surpass the Masked R-CNN localization baseline and narrow the segmentation gap with SAM.
  • GRIT VQA: 84.6 vs 58.5 on same-source VQA contrasts with 57.7 vs 67.2 on new-source VQA.The difference may reflect question-type differences and answer-style mismatches, including “hair” versus “mane” for an ambiguous question.
  • GRIT localization: UNIFIED-IO 2 struggles with GRIT localization images containing many instances of the target class, particularly under beam search.The authors attribute this to probability mass splitting among similar location tokens and use an EOS probability threshold during inference.
  • GRIT localization: A 0.8 non-maximum-suppression threshold removes duplicate bounding boxes caused by the crowded-image inference adjustment.The same inference procedure is used for localization and initial localization in keypoint and segmentation tasks.

E.4. Multimodal Benchmark Details

Across multimodal benchmarks, UNIFIED-IO 2 shows strong performance in object hallucination, temporal understanding, image generation, audio, video, and robotic manipulation, with task-specific limitations.

  • Vision and video understanding: The XXL model achieves the highest POPE F1 score across all three object-hallucination dimensions and outperforms other 7B models on SEED-Bench.It is slightly better than LLaVA-1.5 13B, while the XL model leads the temporal-understanding split among compared counterparts.
  • Image generation: UNIFIED-IO 2 generates more faithful TIFA images than baselines, whose outputs can be high-quality but fail to fully follow captions.The comparison emphasizes faithfulness rather than image fidelity alone.
  • Audio generation: UNIFIED-IO 2 supports audio generation across environmental sounds, speech, and music, but clips longer than 4.08 seconds can contain discontinuities or changes in tone, speed, or melody.Longer clips are completed by conditioning on previously generated clips.
  • Image generation: 33.77 vs 13.39 FID shows classifier-free guidance can improve qualitative image quality while worsening the computed FID score.
  • Audio generation: AudioCaps evaluation is not directly comparable with other methods because their clips are 10 seconds while UNIFIED-IO 2 generates 4.08 seconds at a time.The evaluation instead samples four 2-second audio segments and uses zero-padded log-mel-spectrogram inputs.
  • Video and audio understanding: 89.3 vs 87.4 vs 38.2 on Kinetics-Sounds shows higher performance with audio-visual input than video-only or audio-only input.The model achieves 79.1 top-1 accuracy on Kinetics after additional Kinetics-400 finetuning, versus 73.8 with instruction tuning only.
  • Embodied manipulation: On VIMA-Bench, UNIFIED-IO 2 outperforms image- or patch-based counterparts across all four evaluation levels despite predicting all actions from the initial observation.It remains behind the object-centric method.

E.9. Other Tasks

Other-task evaluations show coherent tracking, 3D detection, and view synthesis, alongside limitations in multi-object detection and relative camera-transform accuracy.

  • Tracking: UNIFIED-IO 2 tracks small moving objects such as a table-tennis paddle without using specific class labels.The qualitative examples come from single-object tracking on LaSOT.
  • 3D view synthesis: UNIFIED-IO 2 produces coherent image-based 3D view-synthesis results but struggles to represent relative camera transformations.
Loading 2312.17172v1…