PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards More Unified In-context Visual Understanding

Dianmo Sheng, Dongdong Chen, Zhentao Tan, Qiankun Liu, Qi Chu, Jianmin Bao, Tao Gong, Bin Liu, Shengwei Xu, Nenghai Yu

arXiv:2312.02520v2cs.CV

TL;DR

Existing visual in-context frameworks have been limited to particular output modalities, leaving multimodal output as an open goal for visual understanding. The paper unifies quantized visual and text prompts in interleaved representations and models them with a decoder-only sparse transformer; its conclusion highlights the potential of in-context learning across modalities and tasks.

  • Problem

    Existing visual in-context frameworks were limited to particular output modalities, leaving multimodal output for visual understanding as an open question.

    Source: §1 · p. 1

  • Method

    The framework quantizes modality-specific prompts into discrete tokens, embeds them in a shared representational space, and uses an autoregressive transformer with sparse MoEs.

    Source: §1 · p. 1

  • Results

    The authors report that their framework demonstrates the potential of in-context learning across multiple modalities and tasks.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The work presents a unified framework for jointly learning multimodal data in a general token embedding space.

    Source: §5 · p. 8

    The model performs poorly on multiple small objects or uncommon classes and supports only one class per forward pass, requiring repeated inference for multiple categories.

    Source: §9 · p. 10

Abstract

from arXiv · show

The rapid advancement of large language models (LLMs) has accelerated the emergence of in-context learning (ICL) as a cutting-edge approach in the natural language processing domain. Recently, ICL has been employed in visual understanding tasks, such as semantic segmentation and image captioning, yielding promising results. However, existing visual ICL framework can not enable producing content across multiple modalities, which limits their potential usage scenarios. To address this issue, we present a new ICL framework for visual understanding with multi-modal output enabled. First, we quantize and embed both text and visual prompt into a unified representational space, structured as interleaved in-context sequences. Then a decoder-only sparse transformer architecture is employed to perform generative modeling on them, facilitating in-context learning. Thanks to this design, the model is capable of handling in-context vision understanding tasks with multimodal output in a unified pipeline.Experimental results demonstrate that our model achieves competitive performance compared with specialized models and previous ICL baselines. Overall, our research takes a further step toward unified multimodal in-context learning.

1. Introduction

Visual in-context learning has explored both image and text outputs, but prior frameworks were limited to a particular output modality. The paper proposes a unified multimodal approach that quantizes and embeds vision-language prompts, then models interleaved samples for multimodal input and output.

1. Introduction

  • The proposed framework quantizes and embeds vision-language data in a shared space, then performs next-token prediction on interleaved in-context samples.The authors describe this as their first attempt at multimodal in-context learning.Source: §1 · p. 1
  • Modality-specific decoders map prediction tokens back to their intended domains, allowing the design to support multimodal input and output.The approach uses modality-specific quantizers and a unified embedding layer before decoding.Source: §1 · p. 1
  • Earlier visual in-context systems were limited to image outputs, as in SegGPT, or text outputs, as in Flamingo, motivating a unified Image →Image / Text model.The paper frames the gap as enabling multimodal output for visual understanding.Source: §1 · p. 1, Figure 1 · p. 2
  • An autoregressive transformer with Mixture of Experts models the unified representations through next-token prediction for vision-understanding tasks.Semantic segmentation and dense captioning are used as example tasks, with semantic category information as a clue across in-context samples.Source: §1 · p. 2

2. Related Works

Prior visual in-context learning work includes image-inpainting approaches and dedicated segmentation models, while multimodal systems have explored understanding and generation across modalities. The paper positions its work at the intersection: multimodal outputs within an in-context learning framework.

2. Related Works

  • Vision in-context approaches use image inpainting or masked image modeling, with SegGPT focused on segmentation and other work spanning seven tasks.Prompt selection and nearest-neighbor retrieval have also been studied to enhance visual in-context learning.Source: §2 · p. 2
  • The paper distinguishes its focus from prior work confined to specific domains by studying in-context learning across vision and language.The related-work discussion notes that LLM-based vision-language interaction systems have been confined to textual outputs.Source: §2 · p. 2
  • Multimodal understanding and generation systems connect or generate across modalities, but the cited systems are not designed to use multiple prompts for in-context learning.The passage describes joint embedding, composable generation, and multimodal adaptors with diverse decoders.Source: §2 · pp. 2–3
  • Mixture of Experts has been studied for multitask and multimodal learning, and this work applies autoregressive transformers with MoE to vision-language in-context learning.The related work identifies VL-MoE as combining modality-specific MoEs with generative modeling for vision-language pretraining.Source: §2 · p. 3

3. Method

The method organizes visual and linguistic tasks as interleaved in-context prompts, maps modality-specific discrete tokens into unified representations, and uses a decoder-only transformer with sparse experts for generative modeling.

3. Method

  • The framework combines organized vision-language prompts, modality-specific tokenization, unified embeddings, and a decoder-only transformer with sparse MoEs.It uses next-token prediction on interleaved unified representations.Source: §3 · p. 3

3.1. Vision-Language Prompt Design

  • Vision-only prompts cast each input-output image pair as token-space inpainting, while dense-captioning prompts interleave image tokens with category, bounding-box, and caption text.The image representation uses a [BOI] tag; caption text begins with [BOT], and samples are concatenated as in-context tokens.Source: §3.1 · p. 3

3.2. Unified Multimodal Representations.

  • Modality-specific quantizers encode image and text data as discrete tokens, which a shared linear embedding layer maps into a unified space for sequential concatenation.The pipeline uses a VQGAN vector quantizer for images and GPT-2 BPE tokenization for text; the figure depicts the two stages.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Model Architecture and Training Objective

  • A decoder-only transformer performs next-token prediction on interleaved representations, and modality-specific tokenizers decode sampled prediction tokens into their respective domains.Sparse MoE layers replace decoder FFN blocks, using top-k gating to route token embeddings to experts.Source: §3.3 · pp. 4–5
  • The training objective applies cross-entropy to predicted and ground-truth tokens for each in-context pair, alongside an auxiliary loss for optimizing MoE gating.The total loss combines output and auxiliary losses, with λ weighting the auxiliary term.Source: §3.3 · p. 5

4. Experiments

The experiments define class-aware in-context segmentation and captioning tasks, then test prompt, model, and training choices. The final model is compared with generalist and specialist baselines on both tasks.

4.1. Datasets and Benchmarks.

  • The experiments define class-aware segmentation and captioning as tasks using semantic clues; training and evaluation data comprise about 350k class masks and 460k region descriptions.The segmentation validation set has about 15k masks, and the captioning test set has 2k descriptions.Source: §4.1 · p. 5

4.2. Implementation Details.

  • The implementation uses a GPT-small architecture with attribute-routing MoEs, VQ-GAN image tokens at 256 × 256 resolution, and three in-context samples by default.Training runs for 40 epochs unless otherwise specified.Source: §4.2 · p. 6

4.3. Ablation Studies

  • Small objects with a large object scale yield optimal CA-ICL segmentation results, while bbox-text prompts outperform bbox-image prompts and the baseline in CIDEr.The bbox-text prompts also produce captions more precisely aligned with intended regions.Source: §4.3 · pp. 6–7, Table 2 · p. 6
  • Replacing some transformer FFN layers with MoE improves segmentation by 18.74 scores over the all-tasks baseline, but captioning remains 10.8 scores below it.The AG MoE structure further narrows the performance gap between tasks.Source: §4.3 · p. 7, Table 3 · p. 7

4.4. Comparison with State-of-the-art Methods

  • At 256 resolution, the model exceeds SegGPT by 6.92 MIoU and 0.006 in MAE score, and Openflamingo by 0.7 BLEU4 and 25.6 CIDEr.Segmentation remains behind SegGPT at 448 resolution; captioning still has an mAP gap against GRiT.Source: Table 4 · p. 8, §4.4 · p. 8

4.3. Ablation Studies

  • With AG MoE and the multi-task strategy, both tasks exceed the baseline, including a 28.17-point gain in CA-ICL segmentation.GPT-2 small co-training degrades performance; AG MoE alone balances tasks but remains behind single-task performance.Source: §4.3 · pp. 7–8, Table 3 · p. 7

5. Conclusion

The paper presents a unified in-context visual-understanding framework using multimodal quantization, unified embedding, and autoregressive transformers with MoEs for multitask co-training.

5. Conclusion

  • The framework jointly learns multimodal data in a general token embedding space and uses autoregressive transformers with MoEs for stable, balanced multitask co-training.The authors describe this as demonstrating the potential of in-context learning across modalities and tasks.Source: §5 · p. 8
  • Table 5 presents hyperparameters for the GPT-2 baseline and proposed model, including vocabulary settings with or without special category and bbox tokens.Source: Table 5 · p. 9
  • Table 6 analyzes loss choices for class-aware in-context captioning.Source: Table 6 · p. 9
  • Table 7 analyzes in-context samples and category information for the two CA-ICL tasks.Source: Table 7 · p. 9

6. Model Architecture and Configuration

The proposed architecture alternates dense decoder layers with sparse decoder layers, and the configuration description also specifies vocabulary and normalization choices.

6. Model Architecture and Configuration

  • The out-of-domain segmentation test forms per-class mask pools from non-overlapping LVIS classes, following the MS-COCO approach.Source: Figure 7 · p. 9
  • Even-numbered decoder layers are replaced with sparse decoder layers, while dense decoder layers remain the same.Source: §6 · p. 9
  • The model uses a smaller layer norm epsilon of 1e-12 than GPT-2 to ensure stable training.Source: §6 · p. 9
  • The vocabulary is tested in two settings: with or without special category and bbox tokens.Source: §6 · p. 9

7. Additional Quantative Analysis

Additional analyses examine captioning losses, example and category information, inference speed, and out-of-domain captioning with Visual Genome categories.

7. Additional Quantative Analysis

  • Using only Lout outperforms configurations with Lin in one-shot captioning, while 0.5Lin + Lout performs best in two- and three-shot settings.The authors use 0.5Lin + Lout for individual captioning and Lout for co-training.Source: §7 · pp. 9–10
  • The out-of-domain captioning test uses per-category pools from Visual Genome with categories that do not overlap the training categories.Source: Figure 8 · p. 10
  • More in-context examples consistently improve segmentation, but captioning performance does not rise steadily and can worsen without category information.The authors attribute possible captioning ambiguity to varied description styles across examples; they report the best captioning results with class information.Source: §7 · p. 10
  • 2.8, 2.4, 1.9, and 1.4 fps are reported for the model with 0, 1, 2, and 3 in-context examples, respectively.The paper reports 7.7 fps for SegGPT and 0.3 img/s for OpenFlamingo, and describes its model as usable in interactive applications.Source: §7 · p. 10

8. Additional Qualitative Results

Qualitative results describe in-context reasoning for segmentation and captioning, alongside tests of generalization to categories not encountered during training.

8. Additional Qualitative Results

  • With suitable prompts, the model segments instances from the same category as the in-context samples across indoor and outdoor scenarios.Source: §8 · p. 10
  • The captioning examples show generated captions with locations identifying the region of the desired category.Source: §8 · p. 10
  • Out-of-domain tests report dependable segmentation and effective captioning on categories not encountered during training.The tests use non-overlapping LVIS classes for segmentation and Visual Genome categories for captioning.Source: §8 · p. 10

9. Limitation and future work

The model struggles with multiple small objects and uncommon classes because of the training data’s long-tailed class and object-scale distributions, and supports only one class per forward pass. The authors describe broader task applicability and plan to extend the model to additional modalities, tasks, and output settings.

Limitations

  • Long-tailed class and object-scale distributions limit performance on multiple small objects and uncommon classes, while each forward pass supports only one class.Multiple categories require multiple inference passes.Source: §9 · p. 10

Future work

  • The authors suggest data balancing strategies to address the reported weaknesses, including Copy-Paste for per-category instances and resampling for captioning.They also note that SegGPT’s color-mapping strategy might help with multiple categories.Source: §9 · p. 10

Broader applicability

  • The framework can accommodate in-context learning tasks beyond class-aware tasks because it maps quantized multimodal inputs into a unified representation space.Source: §9 · p. 10

Future work

  • Planned extensions include web pages, 3D vision, heat maps, tables, image generation and editing, inpainting, grounding, high-resolution images, and longer outputs.The authors present these as future directions for broadening applicability.Source: §9 · p. 11

Related papers

Loading 2312.02520v2…