PaperlayerView exact revision on arXiv

Source-linked AI summary

Contextual Object Detection with Multimodal Large Language Models

Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, Chen Change Loy

arXiv:2305.18279v2cs.CVcs.AI

TL;DR

MLLMs lack object detection for human-AI interactive contexts, while existing detectors are constrained by predefined categories. ContextDET generates contextual object words before detecting them, and it outperforms BARON on open-vocabulary object detection.

  • Problem

    Existing detectors are constrained by predefined categories and their locate-then-classify design, leaving detection across human-language vocabulary in interactive contexts unaddressed.

    Source: §1 · p. 2

  • Method

    ContextDET uses a generate-then-detect model that conditions an LLM on visual and task-language tokens, then uses its tokens to predict boxes for contextual object words.

    Source: §1 · p. 3, §3.2 · p. 5

  • Results

    ContextDET outperforms BARON by 2.8%, 4.7%, and 4.2% on the novel, base, and all sets, respectively, in open-vocabulary object detection.

    Source: §5.3 · p. 11

  • Takeaways & Limitations

    ContextDET predicts object names and locations in images for human-AI interaction, illustrating MLLMs’ potential in contextual object detection.

    Source: §6 · p. 13

    Associating boxed objects with language captions incurs high annotation costs, so ContextDET uses less training data than previous MLLM papers, which may limit performance.

    Source: §6 · p. 13

Abstract

from arXiv · show

Recent Multimodal Large Language Models (MLLMs) are remarkable in vision-language tasks, such as image captioning and question answering, but lack the essential perception ability, i.e., object detection. In this work, we address this limitation by introducing a novel research problem of contextual object detection -- understanding visible objects within different human-AI interactive contexts. Three representative scenarios are investigated, including the language cloze test, visual captioning, and question answering. Moreover, we present ContextDET, a unified multimodal model that is capable of end-to-end differentiable modeling of visual-language contexts, so as to locate, identify, and associate visual objects with language inputs for human-AI interaction. Our ContextDET involves three key submodels: (i) a visual encoder for extracting visual representations, (ii) a pre-trained LLM for multimodal context decoding, and (iii) a visual decoder for predicting bounding boxes given contextual object words. The new generate-then-detect framework enables us to detect object words within human vocabulary. Extensive experiments show the advantages of ContextDET on our proposed CODE benchmark, open-vocabulary detection, and referring image segmentation. Github: https://github.com/yuhangzang/ContextDET.

1 Introduction

The introduction defines contextual object detection as understanding visible objects within human-AI interactive contexts, then presents ContextDET as a generate-then-detect framework for detecting human-vocabulary object words. It also introduces the CODE benchmark and reports ContextDET’s advantages across contextual detection, open-vocabulary detection, and referring image segmentation.

Problem and tasks

  • Contextual object detection covers human-vocabulary capacity, natural-language description, locating objects from language queries, and using language hints to complete object words.The introduction studies this setting through language cloze, visual captioning, and question answering.Source: §1 · pp. 1–2

Challenges

  • Standard closed-set detectors cannot generalize to unrestricted human-vocabulary categories, while open-vocabulary detectors remain constrained by predefined novel categories.The introduction notes that open-vocabulary methods therefore cannot detect some context-specific names, such as “hockey goalie,” “groom,” and “cowboy.”Source: §1 · p. 2
  • A locate-then-classify approach misses how visual objects and words change meaning across contexts, such as “person” becoming “goalie” or “labrador” replacing “dog.”The introduction argues that contextual object detection requires an approach suited to varied and changing contexts.Source: §1 · pp. 2–3

ContextDET framework

  • ContextDET combines a visual encoder, an LLM for multimodal context decoding, and a visual decoder that uses contextual tokens to predict matching scores and bounding boxes.The encoder produces local and full visual tokens; the decoder forms conditional object queries from LLM tokens and keys and values from full visual tokens.Source: §1 · p. 3

Contributions

  • The authors introduce CODE, a benchmark with 10,346 unique object words, and report ContextDET’s advantages on CODE, open-vocabulary detection, and referring image segmentation.They present the benchmark and framework as contributions intended to support empirical study of contextual object detection.Source: §1 · p. 3

2 Related Work

The paper distinguishes contextual object detection from prior vision-language tasks by combining multimodal context with object localization. ContextDET is presented as an end-to-end alternative to prompting frozen vision experts, with settings that extend beyond conventional grounding, captioning, and VQA.

Multimodal Large Language Models (MLLMs) / Prompting LLMs with Vision Experts

  • Unlike MLLMs limited to textual outputs, ContextDET produces bounding boxes by conditioning a visual decoder on latent MLLM features.This contrasts with prompting-based systems that use LLM text to direct external vision experts, whose frozen parameters may prevent knowledge and representation sharing.Source: §2 · p. 3

Object Detection with Contextual Understanding

  • Contextual object detection uses multimodal relationships between images and words, unlike prior context-aware detection focused on neighboring pixels or image regions.The paper also proposes cloze-test evaluation to assess contextual understanding.Source: §2 · p. 3

Object Detection on Novel Classes

  • ContextDET applies MLLMs to open-vocabulary detection without predefined base or novel classes, generating object names as contextually valid English words.Open-vocabulary detection allows additional image-text pairs (Zareian et al., 2021); prior approaches widely use CLIP (Radford et al., 2021; Zhong et al., 2022; Li et al., 2022; Zhang et al., 2022a; Gu et al., 2022; Zhou et al., 2022; Du et al., 2022; Zang et al., 2022; Rasheed et al., 2022; Kuo et al., 2022; Wu et al., 2023b,c,d).Source: §2 · pp. 3–4

Visual Grounding

  • ContextDET extends visual grounding: cloze tests require inferring masked object names and positions, while captioning has no query and QA uses interactive human-language descriptions.Visual grounding typically provides a language query describing a specific object and asks for its position; prior methods commonly use Transformer-based cross-modal structures or multimodal pre-training (Yang et al., 2022; Wang et al., 2022; Kamath et al., 2021).Source: §2 · p. 4

Image Captioning

  • Unlike image captioning systems that generate descriptive sentences, contextual captioning also predicts bounding-box locations for objects mentioned in the generated captions.Image captioning commonly encodes images with pretrained classification, detection, or vision-language models, then decodes features into sentences (Chen et al., 2017; Anderson et al., 2018; Mokady et al., 2021; Hochreiter and Schmidhuber, 1997; Vaswani et al., 2017).Source: §2 · p. 4

Visual Question Answering (VQA)

  • Unlike traditional VQA’s natural-language question-answer pairs, contextual QA asks about object names and locations and expects referring bounding boxes.Source: §2 · p. 4

3 Approach

ContextDET takes images interleaved with human text and produces freeform text with corresponding bounding boxes through an end-to-end visual encoder, multimodal LLM, and visual decoder. Its generate-then-detect design uses contextual language to guide visual localization.

3.1 Visual Encoder

  • A frozen pretrained vision backbone produces spatial features that become local tokens for the LLM and full visual tokens for the visual decoder.Local tokens come from pooled spatial bins and a linear projection; full tokens are encoded from the flattened feature grid by six Transformer layers.Source: §3.1 · p. 5

3.2 Multimodal Context Modeling with LLM

  • The LLM conditions on local visual tokens and task-specific language to generate multimodal context in an autoregressive sequence.Language inputs take the form of masked sentences for cloze tests, prompts for captioning, or questions for question answering; generation ends at [EOS].Source: §3.2 · pp. 5–6

3.3 Visual Decoder

  • ContextDET generates object words from language and visual context, then uses those words as conditional queries to locate matching objects beyond a fixed set of categories.It predicts which prefix or generated language tokens are object words and uses their latent embeddings to condition the visual decoder. Conditional matching includes only ground-truth boxes matching the queried object words.Source: §3.3 · p. 6
  • Six cross-attention layers use full visual tokens to refine object-word-conditioned queries, whose outputs predict box coordinates and binary matching scores.The object-word embedding is repeated across learnable queries and projected to the decoder dimension before cross-attention.Source: §3.3 · p. 6

3.4 Training Details

  • Training combines language-modeling, noun classification, matching classification, and box losses, with weights λcls = 1, λbox = 5, λlm = 1, and λnoun = 1.Multi-scale deformable attention and IoU-based label assignment are used to accelerate convergence; the vision encoder can also use pretrained weights from MLLMs such as BLIP-2 (Li et al., 2023).Source: §3.4 · pp. 6–7

4 CODE Benchmark

CODE is a contextual object-detection benchmark built from Flickr30k images and captions, with object names masked for cloze testing. Its evaluation matches predicted boxes to language-based object names and masked positions, and also supports captioning and question-answering prompts.

Dataset

  • CODE contains 10,346 unique object names across 29,781 training, 1,000 validation, and 999 test images, with 665,161, 22,061, and 21,641 boxes respectively.It adds caption object-name position annotations to Flickr30k and Flickr30k Entities data; object names are masked for the cloze test.Source: §4 · p. 7

Data Format

  • The benchmark follows COCO’s data format with additional evaluation fields and records tokenizer-derived start and end indices for masked object names.These indices identify the language tokens replaced by [MASK] in cloze-test inputs.Source: §4 · pp. 7–8

Word Clouds

  • Rare names such as “scooty” and “breadstick,” alongside frequent names, make contextual understanding important for predicting masked object words.The benchmark includes 10,346 unique masked object words.Source: §4 · p. 8

Evaluation Details

  • Cloze evaluation counts a predicted box as correct when it has high IoU, the same meaning, and the same masked-word index as the ground truth.Unlike predefined class-ID evaluation, masked indices distinguish same-named objects at different sentence positions; AP uses IoU thresholds from 0.5 to 0.95 in 0.05 steps, without per-name AP for long-tailed names.Source: §4 · pp. 8–9, Figure 5 · p. 8

AP@5 for Top-5 Predicted Names

  • AP@5 supplements top-1 AP by counting a prediction as correct when the ground-truth name appears among the top five predicted names.The supplement addresses cases involving difficult-to-distinguish synonyms or fine-grained categories.Source: §4 · p. 9

Evaluation for Context QA and Context Captioning

  • Cloze-test, captioning, and question-answering prompts evaluate contextual understanding of the same scenario and can be converted between settings using different text prompts.The examples present a masked sentence, a caption prompt, and a question about the same person standing with a bride and holding balloons.Source: §4 · p. 9

5 Experiments

Experiments evaluate contextual detection on CODE, open-vocabulary detection, referring image segmentation, and standard COCO detection; reported ContextDET results use OPT-2.7B and ResNet50.

5.2 Ablation Studies

  • On CODE, AP@1 rises from 10.2 with ResNet50 to 13.1 with Swin-B and 13.7 with OPT-6.7B; p = 9 yields the optimal results.The 10.2 and 13.1 values compare ResNet50 and Swin-B with OPT-2.7B; 13.7 uses OPT-6.7B. The study selects p = 9 as its default.Source: Table 3 · p. 10, §5.2 · pp. 10–11, Table 5 · p. 10
  • Removing local visual tokens drops top-1 accuracy from 48.7 to 30.9 (%), while only 6.4% of OPT-2.7B/ResNet50 parameters are trainable.Without local tokens, language-modeling loss barely decreases; the paper links this to difficulty generating text that describes visual content.Source: Table 4 · p. 10, §5.2 · p. 11

5.1 Contextual Object Detection

  • ContextDET outperforms captioner-plus-GLIP cascades across CODE settings, where 10,346 object names make detection challenging; LLM contextual embeddings help, while fine-tuning GLIP yields no significant gain.The paper attributes the improvement to contextual relationships in LLM hidden embeddings, which support contextual understanding across a large vocabulary.Source: §5.1 · p. 10

5.4 Referring Image Segmentation

  • On referring image segmentation, ContextDET gains 0.63% and 0.45% mIoU over PolyFormer on validation and test splits, respectively.The adaptation adds a pixel-level segmentation head using full visual tokens, trained with pixel-wise cross-entropy and Dice losses; evaluation covers RefCOCO, RefCOCO+, and RefCOCOg.Source: Table 7 · p. 11, §5.4 · pp. 11–12

5.3 Open-Vocabulary Object Detection

  • On OV-COCO, ContextDET exceeds BARON by 2.8%, 4.7%, and 4.2% on novel, base, and all classes, respectively.Training/testing times are 0.47/0.63 sec/iter for OV-DETR and 0.54/0.25 sec/iter for ContextDET; the paper attributes faster testing to conditioning only on classes present in the image.Source: Table 6 · p. 11, §5.3 · p. 11

5.5 Standard Object Detection

  • On standard COCO detection, ContextDET does not match specialized methods such as Deformable DETR but shows strength in detecting small objects.The paper attributes this strength to contextual information that disambiguates regions containing small objects.Source: Table 8 · p. 12, §5.5 · p. 12

5.6 Qualitative Results

  • Qualitative examples show contextual object-word localization, open-world names, and multi-round QA on web images, but errors include uncommon terms and occluded objects.The examples include names absent from CODE training annotations; the paper also contrasts ContextDET’s bounding-box predictions with MLLMs’ textual outputs alone.Source: §5.6 · pp. 12–13, Figure 6 · p. 12, Figure 7 · p. 13

6 Conclusion

ContextDET demonstrates the potential of MLLMs for contextual object detection, which predicts object names and locations for human-AI interaction. The conclusion highlights annotation costs as a limitation and points to semi-supervised learning and instruction-based prediction editing as future directions.

6 Conclusion

  • ContextDET demonstrates MLLMs’ potential for contextual object detection, predicting precise object names and locations in images for human-AI interaction.The conclusion situates this task within perception abilities beyond MLLMs’ established vision-language tasks such as question-answering.Source: §6 · p. 13
  • High annotation costs required fewer training data than in previous MLLM papers, potentially limiting ContextDET’s final performance.The annotations associate bounding-box object words with language captions.Source: §6 · p. 13
  • Future work could use semi-supervised or weakly supervised learning to reduce annotation costs and explore MLLMs’ interactive ability to edit detection outputs from language instructions.Proposed edits include shifting boxes, removing redundant overlapping boxes, and correcting a predicted class.Source: §6 · p. 13

Related papers

Loading 2305.18279v2…