PaperlayerView exact revision on arXiv

Source-linked AI summary

Contextual Object Detection with Multimodal Large Language Models

Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, Chen Change Loy

arXiv:2305.18279v2cs.CVcs.AI

TL;DR

MLLMs perform vision-language generation but lack object detection over contextual human-language categories. ContextDET models multimodal context with a visual encoder, pretrained LLM, and visual decoder in a generate-then-detect framework. Experiments show advantages on the CODE benchmark and open-vocabulary detection, while annotation costs and limited training data constrain performance.

  • Problem

    MLLMs support vision-language tasks but lack object detection for identifying and locating objects expressed in human-language vocabulary within interactive contexts.

  • Method

    ContextDET uses a visual encoder, pretrained LLM, and visual decoder in an end-to-end generate-then-detect framework for contextual object words and bounding boxes.

  • Results

    ContextDET shows advantages on the CODE benchmark and open-vocabulary detection, including 2.8%, 4.7%, and 4.2% margins over BARON on novel, base, and all sets, respectively.

  • Takeaways & Limitations

    ContextDET extends MLLMs from textual vision-language outputs to contextual object detection with language-associated bounding boxes.

  • Takeaways & Limitations

    High annotation costs led to less training data than previous MLLM papers, which may limit final performance.

Abstract

from arXiv · show

Recent Multimodal Large Language Models (MLLMs) are remarkable in vision-language tasks, such as image captioning and question answering, but lack the essential perception ability, i.e., object detection. In this work, we address this limitation by introducing a novel research problem of contextual object detection -- understanding visible objects within different human-AI interactive contexts. Three representative scenarios are investigated, including the language cloze test, visual captioning, and question answering. Moreover, we present ContextDET, a unified multimodal model that is capable of end-to-end differentiable modeling of visual-language contexts, so as to locate, identify, and associate visual objects with language inputs for human-AI interaction. Our ContextDET involves three key submodels: (i) a visual encoder for extracting visual representations, (ii) a pre-trained LLM for multimodal context decoding, and (iii) a visual decoder for predicting bounding boxes given contextual object words. The new generate-then-detect framework enables us to detect object words within human vocabulary. Extensive experiments show the advantages of ContextDET on our proposed CODE benchmark, open-vocabulary detection, and referring image segmentation. Github: https://github.com/yuhangzang/ContextDET.

1 Introduction

The paper introduces contextual object detection for understanding and localizing visible objects within human-AI interactive contexts. ContextDET addresses this problem by generating contextual object words with an LLM and then detecting their corresponding visual locations.

  • Research Problem: Contextual object detection targets human-language object vocabulary, informative descriptions, language-query association, and contextual understanding across interactive vision-language tasks.The proposed settings include language cloze testing, visual captioning, and question answering with object detection.
  • Task Settings: The three contextual settings operationalize detection through masked-name completion, caption-associated boxes, and question answering about object names and locations.These scenarios extend beyond traditional detection of a limited set of predefined categories.
  • Research Problem: Standard detectors rely on closed-set classes, while open-vocabulary detectors remain constrained by predefined category scales and cannot reliably handle context-specific names such as “hockey goalie” or “groom.”Their locate-then-classify paradigm is also unsuitable when object categories vary with contextual language.
  • ContextDET: ContextDET is an end-to-end generate-then-detect framework that uses a visual encoder, a pretrained LLM, and a visual decoder to produce contextual object words and bounding boxes.The decoder derives conditional object queries from contextual LLM tokens and combines them with full visual tokens to predict matching scores and boxes.

2 Related Work

The related work situates ContextDET among multimodal language models, open-vocabulary detection, visual grounding, and image captioning. It emphasizes that ContextDET extends language-based visual interaction toward contextual object detection with bounding-box outputs.

  • Multimodal Large Language Models: MLLMs combine language and visual understanding for tasks such as image captioning and visual question answering, but existing systems are limited to textual outputs.ContextDET is positioned as an extension that adds contextual object detection and bounding boxes.
  • Open-Vocabulary Object Detection: Open-vocabulary detection uses image-text knowledge to detect novel classes, but ContextDET applies MLLMs to generate contextually valid object names without restricting detection to predefined base or novel classes.The paper contrasts this approach with CLIP-centered open-vocabulary methods.
  • Visual Grounding: Visual grounding combines language understanding with object localization by predicting the position of an object described by a language query.This provides a related task formulation for connecting language and visual regions.
  • Image Captioning: Image captioning encodes images into feature embeddings and decodes them into descriptive sentences using sequence models such as LSTMs or Transformers.ContextDET incorporates caption-like language generation while additionally associating object words with boxes.

3 Approach

ContextDET is an end-to-end framework that combines visual encoding, multimodal language modeling, and visual decoding to generate contextual object words and locate their corresponding objects. Its generate-then-detect pipeline uses language-derived object tokens as conditional queries for bounding-box prediction and trains with conditional matching.

  • Framework: ContextDET accepts images with human text and produces freeform text with corresponding bounding boxes through a visual encoder, pretrained LLM, and visual decoder.The visual encoder computes visual tokens, the LLM decodes multimodal context, and the visual decoder predicts matching scores and boxes.
  • Visual Encoder: The visual encoder extracts spatial image features, then produces local visual tokens for the LLM and full visual tokens for the visual decoder.Local tokens are formed with adaptive pooling and projection, while full tokens are encoded from flattened spatial features with Transformer layers.
  • Multimodal Context Modeling: The LLM models multimodal context by conditioning autoregressive generation on local visual tokens and task-specific language tokens.The language inputs vary across contextual object detection settings, while generated vocabulary tokens provide human-language outputs.
  • Visual Decoder: ContextDET selects language and generated tokens identified as object words, using their latent embeddings to form conditional object queries for visual decoding.The generate-then-detect design uses contextual words such as ‘bride’, ‘groom’, and ‘dog’ rather than restricting detection to a closed set of predefined classes.
  • Visual Decoder: The visual decoder applies cross-attention between conditional object queries and full visual tokens before predicting matching scores and bounding boxes.Its outputs include binary matching probabilities and box coordinates for each conditional object word.
  • Training Details: Training uses conditional bipartite matching so that only ground-truth boxes matching the queried object words contribute to the loss.The overall objective combines classification, box, language-modeling, and noun-classification losses, while the CODE benchmark adds captions, token indices, and object names to COCO-format data.

4 CODE Benchmark

The CODE benchmark adapts Flickr30k data for contextual object detection, adding masked object names, language-token positions, and name-based evaluation across cloze, captioning, and question-answering settings.

  • Dataset construction: CODE combines Flickr30k images and Flickr30k Entities annotations with object-name position information for contextual detection.Object names in captions are replaced with [MASK] tokens for the cloze-test setting.
  • Dataset construction: The benchmark follows COCO’s data format while adding fields needed to track language tokens and masked object positions.Tokenized object names are recorded with their start and end indices before replacement by [MASK].
  • Dataset characteristics: CODE contains 10,346 unique masked object words, including frequent and rare terms that challenge prediction.Examples of low-frequency words include “player,” “scooty,” and “breadstick.”
  • Evaluation criteria: Contextual cloze predictions are true positives when boxes have high IoU, matching meaning, and the correct masked-word index.The index distinguishes multiple objects with the same name at different positions in a sentence.
  • Benchmark reporting: The benchmark reports cloze classification accuracy and detection AP across cloze, captioning, and question-answering settings.The evaluation presents the same contextual understanding scenario through different text prompts.
  • Evaluation criteria: Average Precision uses COCO-style IoU thresholds from 0.5 to 0.95, while per-name AP is omitted because many names are long-tailed and sparsely represented.The benchmark also reports supplementary top-5 AP because top-1 evaluation can be stringent for synonyms and fine-grained categories.

5 Experiments

Experiments evaluate ContextDET on contextual object detection, open-vocabulary detection, referring image segmentation, and qualitative generalization. Results show gains from contextual multimodal modeling, local visual tokens, larger backbones, and LLMs, while revealing failures with uncommon or occluded objects.

  • Evaluation: ContextDET is evaluated on CODE contextual detection, open-vocabulary detection, referring image segmentation, and qualitative settings using a single-machine implementation.The evaluation includes benchmark, ablation, efficiency, generalization, and extension experiments.
  • Contextual Object Detection: ContextDET outperforms cascade baselines across all contextual object detection settings.The authors attribute this advantage to LLM hidden embeddings that capture relationships between visual and text tokens.
  • Ablation Studies: 10.2 to 13.1 to 13.7 AP@1 is achieved as the vision backbone and language model are strengthened.Replacing ResNet50 with Swin-B raises AP@1 from 10.2 to 13.1, while replacing OPT-2.7B with OPT-6.7B raises it to 13.7.
  • Ablation Studies: 48.7 to 30.9% top-1 accuracy results when local visual tokens are removed, underscoring their role in contextual understanding.The ablation also reports that language-modeling loss barely decreases without the local visual tokens.
  • Open-Vocabulary Object Detection: 2.8%, 4.7%, and 4.2% margins over BARON are reported on the novel, base, and all OV-COCO sets, respectively.ContextDET conditions its visual decoder on class-name embeddings after asking the MLLM whether each class appears in the image.
  • Referring Image Segmentation: 0.63% and 0.45% mIoU gains over PolyFormer are achieved on the referring segmentation validation and test splits.The extension adds a pixel-level segmentation head using full visual tokens and trains with pixel-wise cross-entropy and Dice losses.
  • Qualitative Results: ContextDET qualitatively predicts contextual object words and boxes for open-world names and multi-round question-answering dialogs, but struggles with uncommon terms and occluded objects.Examples include novel names such as Harry Potter, Pikachu, and Messi; reported failures include “earth” and occluded “sheep.”

6 Conclusion

ContextDET highlights the potential of MLLMs for perception tasks beyond question answering, including contextual object detection for human-AI interaction. The authors note that limited training data from costly annotations may constrain performance and identify semi-supervised learning as future work.

  • MLLMs’ potential in perception tasks beyond question answering remains largely unexplored.
  • ContextDET predicts precise object names and locations for human-AI interaction.
  • High annotation costs required less training data than in previous MLLM papers, which may limit final performance.
  • The authors propose semi-supervised or weakly supervised learning to reduce annotation costs.
  • The study’s analyzed datasets are publicly available for research purposes.
Loading 2305.18279v2…