PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action

Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, Aniruddha Kembhavi

arXiv:2312.17172v1cs.CVcs.AIcs.CL

TL;DR

Models for rich multimodal interaction require capabilities across modalities, while training across modalities can become unstable. Unified-IO 2 uses one multimodal model and achieves strong results across more than 35 datasets, including state-of-the-art performance on GRIT.

  • Problem

    Developing models for rich multimodal interactions remains challenging because training across added modalities can become unstable.

    Source: §1 · p. 2, §3.2 · p. 4

  • Method

    Unified-IO 2 uses a single multimodal model, pretrained from scratch on diverse data and instruction-tuned across more than 120 datasets spanning 220 tasks.

    Source: §1 · p. 2, §7 · p. 11

  • Results

    Across more than 35 datasets, UNIFIED-IO 2 shows effectiveness across modalities and sets a new state of the art on GRIT, surpassing UNIFIED-IO by 2.7 points.

    Source: §1 · p. 2, §5.2 · p. 9

  • Takeaways & Limitations

    The authors conclude that moving from language models to multimodal models offers new capabilities and opportunities.

    Source: §7 · p. 11

    Image-generation quality remains below Stable Diffusion, while audio generation is capped at approximately 4 seconds, restricting practical application.

    Source: §6 · p. 11

Abstract

from arXiv · show

We present Unified-IO 2, the first autoregressive multimodal model that is capable of understanding and generating image, text, audio, and action. To unify different modalities, we tokenize inputs and outputs -- images, text, audio, action, bounding boxes, etc., into a shared semantic space and then process them with a single encoder-decoder transformer model. Since training with such diverse modalities is challenging, we propose various architectural improvements to stabilize model training. We train our model from scratch on a large multimodal pre-training corpus from diverse sources with a multimodal mixture of denoisers objective. To learn an expansive set of skills, such as following multimodal instructions, we construct and finetune on an ensemble of 120 datasets with prompts and augmentations. With a single unified model, Unified-IO 2 achieves state-of-the-art performance on the GRIT benchmark and strong results in more than 35 benchmarks, including image generation and understanding, natural language understanding, video and audio understanding, and robotic manipulation. We release all our models to the research community.

1. Introduction

Unified-IO 2 is a from-scratch multimodal model trained to understand and generate across modalities, with broad benchmark results and a single model spanning vision, language, audio, video, and action.

1. Introduction

  • Because sensory modalities can supervise one another, multimodal agents could better perceive, communicate, act, and reason, but adding modalities amplifies already substantial training challenges.Those challenges include sourcing and processing data, ensuring quality and managing biases, designing architectures, stabilizing training, and instruction tuning.Source: §1 · p. 2
  • Multimodal systems have augmented pretrained LLMs with modality encoders [5] [46] [119], decoders [14] [96], or modular frameworks [64] [166], while broad generative coverage from scratch remains open.Recent from-scratch work focused on text generation [81] [143] or understanding and generating text and images [123] [125].Source: §1 · p. 2
  • The 7-billion-parameter model handles text, image, audio, video, and interleaved inputs, and produces text, action, audio, images, and sparse or dense labels.Pretraining used 1 billion image-text pairs, 1 trillion text tokens, 180 million video clips, 130 million interleaved image-text examples, 3 million 3D assets, and 1 million agent trajectories; instruction tuning combined 120+ datasets spanning 220 tasks.Source: §1 · p. 2
  • A multimodal mixture-of-denoisers objective combines denoising and generation, while dynamic packing increases throughput 4x and architectural changes target training stability and scalability.The architectural changes include 2D rotary embeddings, QK normalization, and scaled cosine attention on the perceiver resampler.Source: §1 · p. 2
  • Instruction tuning gives tasks clear prompts and adds open-ended and synthetic tasks to broaden instruction and task variety, especially for less common modalities.The tuning corpus combines more than 120 datasets covering 220 tasks across vision, language, audio, and action.Source: §1 · p. 2
  • Across 35+ datasets, UNIFIED-IO 2 sets a new state of the art on GRIT, matches or exceeds many recent VLMs, and outperforms the closest image-generation competitor on faithfulness.The image-generation competitor leverages a pretrained stable diffusion model [154], and the faithfulness metrics are defined in [76].Source: §1 · p. 2

2. Related Work

Prior multimodal systems broadened task and modality coverage, but often focused on understanding rather than complex generation. Unified-IO 2 extends this trend with broader capabilities and a from-scratch training approach.

Related Work

  • A common multimodal-LLM design maps vision-encoder features into LLM embeddings and trains on paired image-language data, often followed by visual instruction tuning.Such models can perform some tasks zero-shot or with in-context examples [109] [132]; instruction tuning uses instructions, visual inputs, and target-text triples [25] [34] [118] [119] [205] [218] [225].Source: §2 · p. 3
  • Researchers expanded these systems to OCR, grounding, retrieval, languages, embodied tasks, video, and audio, but many remained focused on understanding and lacked complex generation.PandaGPT [170] and ImageBind-LLM [69] use ImageBind [56], while ChatBridge [222] uses a universal encoder based on language; ImageBind cannot support sparse image annotation.Source: §2 · p. 3
  • Earlier multimodal-generation systems use VQ-GAN-decoded tokens, diffusion-consumable features, or fused pre-trained models; Unified-IO 2 uses VQ-GAN for text, image, and audio generation.UNIFIED-IO [123], LaVIT [88], OFA, Emu [172], and CM3Leon [210] use VQ-GAN-decoded tokens; GILL [96], Kosmos-G [141], and SEED [53] generate features for diffusion, while JAM [4] fuses pre-trained language and image-generation models.Source: §2 · p. 3
  • Unified-IO 2 extends prior multimodal capabilities and outputs, while CoDi achieves similar any-to-any generation with independently trained diffusion models; Unified-IO 2 has stronger language abilities and performs well on more tasks.The paper describes Unified-IO 2 as pushing the trend of expanding supported tasks and modalities to its limit, with few exceptions.Source: §2 · p. 3
  • Training from scratch distinguishes Unified-IO 2 from prior approaches [114] and avoids costly preliminary language-model pre-training.The paper says those prior approaches typically do not produce complex generations or follow text instructions; scratch training is also presented as a natural fit for learning modalities through co-occurrence.Source: §2 · p. 3

3. Approach

UNIFIED-IO 2 represents multimodal inputs and outputs in a shared token space for one encoder-decoder transformer. Its approach combines modality-specific encodings, architectural changes for stable training, a multimodal denoising objective, and efficiency and optimization techniques.

3.1. Unified Task Representation

  • Images, text, audio, actions, and structured outputs are represented as token sequences in a shared space and processed by one encoder-decoder transformer.Images and audio use pretrained encoders and discrete-token generators; image and audio history can also be compressed into tokens for context.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2. Architecture

  • Without improvements, all-modality XXL loss exploded after 350k steps and next-token accuracy fell at 400k; 2D RoPE, QK normalization, scaled cosine attention, float32 logits, and frozen ViT/AST address instability.ViT and AST are frozen during pretraining and finetuned at the end of instruction tuning.Source: §3.2 · pp. 4–5, Figure 3 · p. 5, Figure 4 · p. 5

3.3. Training Objective

  • The multimodal denoiser combines [R], [S], and [X] paradigms; dynamic masking hides target tokens except during prediction to curb decoder leakage without disrupting causal generation.For image and audio generation, the decoder also uses row-, column-, and convolution-shaped masked sparse attention.Source: §3.3 · p. 5, Figure 5 · p. 5

3.4. Efficient Implementation

  • Dynamic packing rearranges streamed examples and packs them around transformer stages, increasing training throughput almost 4x while allowing modality encoders and decoders to run on unpacked data.Attention is masked to prevent the transformer from cross-attending between examples.Source: §3.4 · p. 6

3.5. Optimizer

  • Adafactor uses a 5,000-step linear warm-up and gradient clipping at 1.0; models train 3.0M steps, split 1.5M each between pretraining and instruction tuning.The optimizer uses β1 = 0.9 and β2 = 1.0 − k − 0.8, where k is the step number.Source: §3.5 · p. 6

4. Multimodal Data

Unified-IO 2 is trained from scratch on curated, open-source multimodal data for pre-training and instruction tuning. Pre-training combines modality-specific sources with randomly constructed prediction targets, while instruction tuning spans 220 tasks from over 120 datasets.

4. Multimodal Data

  • Unlike prior work, Unified-IO 2 is trained from scratch on curated, high-quality, open-source multimodal data for both pre-training and instruction tuning.Source: §4 · p. 6

4.1. Pre-training Data

  • Pre-training draws on NLP (33%), image-text (40%), video-audio (25%), 3D and embodiment (1%), and augmentation (1%) data.Sources include public text and paired-image datasets, video and audio datasets, 3D and embodied environments, generated segmentation examples, and synthetic patch-detection data.Source: §4.1 · pp. 6–7
  • Pre-training randomly selects a modality as the target, then removes or corrupts it and randomly keeps or masks other modalities as inputs.For video examples, sample construction also selects the objective and input mask and adds a prefix token.Source: §4.1 · p. 7, Figure 7 · p. 7

4.2. Instruction Tuning Data

  • Instruction tuning combines supervised tasks with prompts, open-ended tasks, and synthetic tasks to build diverse multimodal skills and support new instructions.The mixture includes 220 tasks from over 120 external datasets.Source: §4.2 · p. 7
  • Instruction tuning covers language, image and audio generation and understanding, video understanding, image and video labeling, and embodied AI.Tasks include image editing, segmentation, video question answering, object detection, depth estimation, tracking, continuous-control manipulation, and PointNav.Source: §4.2 · pp. 7–8
  • The instruction-tuning mixture comprises 60% prompted supervised data, 30% pre-training data, 6% task augmentation, and 4% free-form text.Pre-training data is carried over to help avoid catastrophic forgetting; task augmentation constructs novel tasks from existing sources.Source: §4.2 · p. 8

5. Experiments

Experiments evaluate Unified-IO 2 across modalities without task-specific fine-tuning. Results show strong performance on GRIT and vision-language benchmarks, alongside competitive generation and weaker results on some 3D and depth tasks.

5.1. Pre-training Evaluation

  • On zero-shot generation and comprehension tasks, Unified-IO 2 performs comparably to or better than task-specific specialists and universal multimodal models.Its HellaSwag results indicate language-modeling ability between typical 3B and 7B models; the authors note it saw approximately 250 billion tokens. The evaluations also cover TIFA, AudioCaps, and SEED-Bench.Source: §5.1 · pp. 8–9, Table 2 · p. 9

5.2. GRIT Results

  • Unified-IO 2 achieves state-of-the-art overall on GRIT, surpassing Unified-IO by 2.7 points across a benchmark spanning seven image-understanding tasks.It gains 3 points in localization, 14 in categorization, 2 in segmentation, and 5 in keypoint; on same-source VQA questions it scores 84.6 vs. 81.2. The 3-billion-parameter model also scores 65.2 vs. Unified-IO’s 64.5 overall.Source: §5.2 · p. 9, Table 3 · p. 9

5.3. Generation Results

  • On TIFA, Unified-IO 2 scores close to minDALL-E and about 10 points ahead of generalist models CoDi and Emu, while its FID scores are slightly higher than compared models.Audio generation scores higher than specialist models except the recent latent diffusion model, but the AudioCaps evaluation is not directly comparable because the model generates 4.08-second segments for a benchmark of 10-second clips. On VIMA-Bench, it predicts all action primitives at once from the initial observation and multimodal prompt.Source: §5.3 · pp. 9–10, Table 4 · p. 9

5.4. Vision Language Results

  • Unified-IO 2 scores highest on MMB by 3.8 points and leads SEED-Bench among 7B models, surpassing LLaVa-1.5 13B on both.It also scores 87.7 on POPE; its CIDEr score of 130.3 is ahead of Shikra and InstructBLIP but behind Qwen-VL and mPLUG-Owl2. OK-VQA is an exception, which the authors attribute to less extensive language pre-training.Source: §5.4 · pp. 10–11, Table 5 · p. 10

5.5. Video, Audio and other Results

  • Unified-IO 2 outperforms BLIP-2 and InstructBLIP on SEED-Bench Temporal by 8.5 points and exceeds MBT on Kinetics-Sounds.The section also reports reasonable audio and video classification and captioning, as well as video question answering.Source: Table 6 · p. 10, §5.5 · p. 11
  • Unified-IO 2 performs similarly to Cube-RCNN on single-object Objectron detection, but drops significantly on multi-object 3D detection tasks.On COCO object detection, excluding ‘stuff’ categories, it reaches AP 47.2, AP50 57.7, and AP75 50.0; it struggles with images containing many objects. Depth estimation is weak at RMSE 0.623 on NYUv2, improving to 0.423 with task-specific fine-tuning; the authors cite depth-scale incompatibility across datasets as a challenge.Source: §5.5 · p. 11, Table 7 · p. 11

6. Limitation

Unified-IO 2’s limitations include resource-constrained model and hyperparameter choices, weaker or restricted outputs in some modalities, and reduced reliability on underrepresented tasks and new instructions.

Limitation

  • Memory constraints led the project to use base ViT and AST encoders, while larger encoders could substantially improve image and audio performance.Source: §6 · p. 11
  • Image generation is more faithful than SD-based methods but does not match Stable Diffusion quality, and audio generation is capped at approximately 4 seconds.Source: §6 · p. 11
  • Limited computational resources constrained hyperparameter exploration, and a significantly larger batch size could enhance performance.Source: §6 · p. 11
  • The model is less reliable on depth, video, and niche abilities such as 3D object detection, probably because these areas have limited task variety.Source: §6 · p. 11
  • Data quality and prompt diversity remain limitations, with performance notably decreasing on new instruction tasks compared with trained tasks.Despite considerable efforts, the human-written prompts still fall short in diversity.Source: §6 · p. 11

7. Conclusion · A. Contributions · B. Model Implementation Details

Unified-IO 2 unifies image, text, audio, and action understanding and generation in an autoregressive model trained from scratch and instruction-tuned on multimodal data. The authors highlight architectural stabilization and a multimodal mixture-of-denoisers objective, while identifying future model and data improvements.

7. Conclusion

  • UNIFIED-IO 2 is presented as the first autoregressive multimodal model to understand and generate image, text, audio, and action.The conclusion describes promising results across a wide range of tasks.Source: §7 · p. 11
  • The model was trained from scratch on diverse multimodal data and instruction-tuned on a massive corpus, using stabilization changes and a multimodal mixture-of-denoisers objective.The objective is described as a way to effectively utilize multimodal signals.Source: §7 · p. 11
  • The authors plan to extend the encoder-decoder model to decoder-only, expand model size, improve data quality, and refine model design.Source: §7 · p. 11

A. Contributions

  • Jiasen Lu led the project’s scope, model pipeline, pretraining and instruction-tuning data pipelines, training process, experiments, and paper writing.The listed pipeline work includes image and audio tokenizers, the main architecture, stabilization, and the training objective.Source: A. Contributions · p. 21
  • Christopher Clark co-led infrastructure, instruction tuning, and evaluation; Sangho Lee led pretraining data and sample construction; Zichen Zhang led instruction-tuning data and prompts.Source: A. Contributions · p. 21

B. Model Implementation Details

B.1. Detailed of Unified Task Representation … B.4. Full Model Details

The appendix specifies how modalities are represented and how rotary positions, dynamic packing, and model training are implemented. It details the token formats and processing choices used to handle diverse multimodal inputs and outputs efficiently.

B.1. Detailed of Unified Task Representation

  • 32,000 BPE tokens and 200 denoising markers support text, while navigation uses text actions and manipulation discretizes motion and gripper states.Inputs and outputs are limited to 512 text tokens. Multi-step plans use human-readable steps, skills, and discretized position and rotation parameters, as in VIMA [87].Source: §B.1 · p. 21
  • 1,000 special tokens encode points, boxes, camera transformations, and 3D cuboids, with prompts specifying the requested cuboid output format.Points and boxes use normalized image coordinates; cuboids include projected center, depth, log-normalized dimensions, and continuous allocentric rotation.Source: §B.1 · p. 21
  • 576-token ViT inputs use pretrained visual features; generated images use an Open Images VQ-GAN yielding 32 × 32 discrete tokens, while dense labels become RGB image targets.For segmentation, the model predicts a black-and-white mask for a class and box; instance segmentation localizes each detected box before segmenting it.Source: §B.1 · p. 22
  • 128 AST tokens encode each 4.08-second audio segment; history features are resampled to 32 visual and 16 audio outputs, with per-frame or per-segment latents preserving temporal information.Audio generation uses discrete spectrogram tokens and a HiFi-GAN vocoder; the final history implementation uses separate latent queries for each frame or segment.Source: §B.1 · pp. 22–23

B.2. 2D Rotary Embedding

  • Two-dimensional rotary embeddings encode image, audio, and history positions, while text uses rotary encoding at each layer and modality embeddings identify segments.The model splits attention query and key embeddings across the two spatial coordinates; history positions pair frame indices with perceiver-latent indices.Source: §B.2 · p. 23

B.3. Dynamic Packing

  • Dynamic packing addresses padding inefficiency because multimodal examples often lack modalities, while sorting by modality lengths is difficult with data and framework constraints.A single image-output example can require 1,024 target tokens, padding other examples in the batch even when their outputs use another modality.Source: §B.3 · p. 23, L XL XXL · p. 23

L XL XXL

  • Packing runs image and VAE components before dynamically combining resulting tokens, masks attention across examples, and can process two examples together for roughly 4× speedup.For streaming data, a pool-based heuristic pairs a new example with the largest compatible pooled example; unpaired examples are emitted when the pool reaches 10, a case occurring less than 0.1% of training.Source: L XL XXL · pp. 23–24

B.4. Full Model Details

  • 1.5 million steps with an effective batch size of 512 yielded approximately 1 trillion pretraining tokens; training also used modality patch subsampling and up to four history images or segments.The models used batch sizes of 512 for pretraining and 256 for instruction tuning, with 8-way in-layer and 64-way data parallelism for scaling to 7B.Source: §B.4 · p. 24

C. Pre-Training Details · C.1. Data Sources

Unified-IO 2 pre-training balances target modalities and corpus sampling while drawing on varied text, image, video, interleaved, multi-view, agent-trajectory, and synthetic data. These sources train denoising and generation across modalities, with tasks designed to develop visual, spatial, and action-related capabilities.

C. Pre-Training Details

C.1. Data Sources

  • Text follows MPT-7B; image-text sources include LAION aesthetic for image generation, LAION 400M for text generation, NSFW-filtered LAION images, and web icons and stylized images.Source: §C.1 · p. 24
  • The 180M short-video corpus uses up to five frames, with four in image/audio history and one as image/audio input; text, audio, or image may be targets with other modalities masked or noised.Source: §C.1 · p. 24
  • OBELICS uses image-or-paragraph targets with preceding image context, trained by denoising or generation; constructed image-caption sequences target text in three captioning tasks.OBELICS contexts include up to four prior images for an image target or five for a paragraph target; marker tokens align captions with images.Source: §C.1 · pp. 24–25
  • Multi-view pre-training uses CroCo cross-view completion and Objaverse [40] camera-conditioned view synthesis or camera-motion prediction, augmented with in-context view examples to improve 3D understanding.Source: §C.1 · p. 25
  • Agent trajectories combine ProcTHOR shortest paths and Habitat demonstrations, subsampling recent frames; targets are the next image, coordinates, or action, using visual context and, where applicable, location or action inputs.Source: §C.1 · pp. 25–26
  • Synthetic pre-training pairs Segment Anything point or box prompts with masks, and trains sparse patch-location and patch-count outputs on artificially patched images.Source: §C.1 · p. 26

D. Instruction Tuning Details · E. Experiment Details

The appendix details how Unified-IO 2 constructs its instruction-tuning mixture and expands supervised tasks across image, video, audio, and embodied settings. It describes data sampling, task-specific inputs and outputs, and augmentations that broaden the model’s instruction-following skills.

D. Instruction Tuning Details

  • The instruction mixture samples broad task categories evenly, then samples datasets roughly by square-root size with adjustments for noisy or rare tasks.Natural-language tuning combines supervised sources with continued unsupervised pretraining to retain information learned earlier.Source: D. Instruction Tuning Details · p. 27, Figure 10 · p. 27, §D.1 · p. 27

D.2. Image Generation

  • Image-generation tuning covers text-to-image, view synthesis, editing, dense-control generation, segmentation-based generation, and object-conditioned inpainting.Prompts distinguish noisy or approximate unsupervised captions from supervised styles; inpainting uses object annotations to mask a region and generate the complete image.Source: §D.2 · pp. 27, 29

D.3. Audio Generation; D.9. Audio Understanding

  • Audio tuning covers text-to-audio generation, next-frame prediction, audio tagging and captioning, and audio-visual action classification.Generation uses four-second segments and prior audio as history; audio understanding samples five segments from the recording for classification and captioning.Source: §D.3 · p. 29, §D.9 · p. 30

D.4. Image Understanding

  • Image-understanding tuning spans visual question answering, tagging, region classification, image-pair QA, instruction following, relationship prediction, and captioning.Prompts specify response style, including short answers for VQA and captioning, to support longer, more natural responses to user questions.Source: §D.4 · p. 29

D.5. Image Sparse Labelling; D.6. Image Dense Labelling

  • Image-labeling tasks cover sparse coordinates for detection, localization, referring expressions, 3D boxes, camera pose, text, and keypoints, plus dense normals, depth, flow, and segmentation.For instance segmentation, the model localizes objects of the target class and then segments each returned box; keypoint inference queries people with localized bounding boxes.Source: §D.5 · p. 29, §D.6 · p. 30

D.7. Video Understanding; D.8. Video Sparse Labelling

  • Video-understanding tasks include captioning, action classification and anticipation, question answering, instruction following, object tracking, and temporal action localization.Video inputs use five sampled frames; action localization uses temporal context because some actions require it for correct identification.Source: §D.7 · p. 30, §D.8 · p. 30

D.10. Embodied AI

  • Embodied-AI tuning combines interleaved language and visual prompts with pick-and-place or push actions, continuous-control data, and point-goal navigation.For navigation, the model predicts the next action from a point goal, observations, and prior actions or states; the authors limit this task because of non-causality and image-history sequence length.Source: §D.10 · pp. 30–31

D.11. Task Augmentation

  • Task augmentation diversifies supervised instructions by varying segmentation, detection, relationship prediction, captioning, surface-normal, and embodied-AI outputs.Embodied augmentations add video QA for planning and affordance, plus goal or subgoal image generation from initial observations and language prompts.Source: §D.11 · pp. 31–32

E.1. Pre-training Visualization · E.2. NLP Results

On NLP tasks, Unified-IO 2 is generally ahead of Open LLaMA 3B but behind LLaMA; qualitative pre-training examples show audio, video-frame, and image-generation capabilities, alongside difficulty rendering text from captions.

E.1. Pre-training Visualization

  • The appendix complements quantitative benchmark evaluation with qualitative visualizations from the pre-trained UIO-2XXL model.Source: §E.1 · pp. 32–33

E.2. NLP Results

  • Unified-IO 2 is generally ahead of Open LLaMA 3B but behind LLaMA on the reported NLP results.Table 14 reports NLP task results; the supplied text does not provide individual scores.Source: Table 14 · p. 33, §E.2 · p. 33

E.1. Pre-training Visualization

  • The pre-trained model learns text-to-speech synthesis through video pre-training.Source: §E.1 · p. 33
  • It can synthesize music that matches the video input.Source: §E.1 · p. 33
  • Future-frame prediction samples are generated from an initial input image and action sequence.Source: §E.1 · p. 33
  • Image-generation samples from prompts show that the model has a good understanding of different objects.Source: §E.1 · p. 33
  • The model struggles to generate the correct text from a given caption.Source: §E.1 · p. 33

E.2. NLP Results

  • NLP evaluations use the EleutherAI LM-Eval harness in zero-shot mode with default prompts, adding the [Text] [S] prefix for text generation tasks.Tasks include HellaSwag, MMLU, ARC, and BoolQ; baselines use the same zero-shot setting and default prompts.Source: §E.2 · p. 33

E.3. GRIT Details

On GRIT, Unified-IO 2 passes the Masked R-CNN localization baseline and moves toward closing the segmentation gap with SAM; its VQA scores vary by source subset, and an inference threshold improves crowded-image localization.

E.3. GRIT Details

  • UNIFIED-IO 2 is the first unified model to pass the Masked R-CNN localization baseline and goes a long way toward closing the segmentation gap with SAM.The passage presents these findings as part of the more detailed GRIT results in Table 13.Source: §E.3 · p. 33
  • On GRIT VQA, UNIFIED-IO 2 scores 84.6 vs 58.5 on same-source questions but 57.7 vs 67.2 on new-source questions.Same-source questions come from VQA 2.0 and new-source questions from VG; the passage notes that the ablation questions lack ground-truth annotations and suggests answer-style or training-data differences as possible explanations.Source: §E.3 · p. 33
  • For crowded-class localization, requiring EOS probability over 0.5 improves performance by mitigating premature stopping under beam search, though duplicate boxes can result and are removed with NMS at 0.8.The inference trick is also used for the initial localization step in keypoint and segmentation tasks.Source: §E.3 · pp. 33–34

E.4. Multimodal Benchmark Details … E.8. Embodiment Details

These sections detail benchmark results and evaluation procedures for multimodal image, audio, video, and robotic tasks. They report comparative results alongside qualitative examples and note limitations in generation and evaluation settings.

E.4. Multimodal Benchmark Details

  • POPE results show the largest model achieves the highest F1 score across all three dimensions, while smaller models favor “no” responses, possibly reflecting negative instruction-tuning examples.POPE tests object hallucination through yes-or-no answers.Source: §E.4 · p. 34
  • On SEED-Bench, the XXL model outperforms other 7B vision/video-language models and slightly exceeds LLaVA-1.5 13B; the XL model leads counterparts in temporal understanding.The passage suggests weaker temporal-understanding results for recent video-language models may reflect limited instruction-following capabilities.Source: §E.4 · p. 34

E.5. Image Generation Details

  • For TIFA captions, UNIFIED-IO 2 generates faithful images, while baselines often fail to follow the text despite high fidelity.The baselines mostly use Stable Diffusion decoders trained on large-scale, high-quality image datasets.Source: §E.5 · p. 34, Figure 15 · p. 35
  • On MS COCO text-to-image evaluation, classifier-free guidance yields FID 33.77 versus 13.39 without guidance, despite qualitatively improving image quality.Evaluation uses 30K validation captions, generates eight images per caption, and selects the best by CLIP text-image similarity.Source: §E.5 · p. 36

E.6. Audio Generation Details

  • AudioCaps evaluation is not apples-to-apples with 10-second audio methods: Unified-IO 2 generates 4.08 seconds at a time, so evaluation uses sampled segments, zero-padded log-mel spectrograms, and HiFi-GAN waveforms.Generated and ground-truth audio are compared using Fréchet Audio Distance, Inception Score, and Kullback–Leibler divergence; examples include environmental sounds, speech, and music, with some longer outputs discontinuous.Source: §E.6 · p. 36, Table 17 · p. 36

E.7. Video and Audio Understanding Details

  • Classification and question answering use open-ended answer generation scored by Exact Match; video captioning uses CIDEr.For Kinetics classification, selecting the highest-logit dataset label as a multiple-choice answer provided only a marginal performance boost.Source: §E.7 · pp. 36–37
  • 79.1 top-1 accuracy on Kinetics-400 after five Kinetics-400 finetuning epochs exceeds 73.8 for instruction tuning only; Kinetics-Sounds accuracy is 89.3 audio-visual, 87.4 video-only, and 38.2 audio-only.The model was trained on Kinetics-710, a mixture including Kinetics-400, 600, and 700, rather than directly on Kinetics-400.Source: §E.7 · pp. 36–37

E.8. Embodiment Details

  • Despite predicting all actions from only the initial observation, Unified-IO 2 outperforms image-input baselines on all four VIMA-Bench levels, trailing only the object-centric method [87].The compared policy-learning baselines predict next-step actions; Unified-IO 2 predicts all actions at once. It can also synthesize target image states from an input state image and natural-language prompt.Source: §E.8 · p. 37

E.9. Other Tasks

Unified-IO 2 handles diverse tasks including single-object tracking and image-based 3D view synthesis, but has limitations in multi-object 3D detection and representing relative camera transformations.

E.9. Other Tasks

  • The model tracks small moving objects without using specific class labels, as illustrated by LaSOT [50] examples.The examples include a table tennis paddle.Source: §E.9 · p. 37
  • The model exhibits suboptimal performance on multi-object 3D detection, with qualitative Objectron [3] examples shown in Figure 19.Source: §E.9 · p. 37, Figure 19 · p. 38
  • Image-based 3D view synthesis produces coherent results but struggles to accurately represent relative camera transformations in Objaverse [40] examples.Source: §E.9 · pp. 37–38, Figure 20 · p. 38

Related papers

Loading 2312.17172v1…