PaperlayerView exact revision on arXiv

Source-linked AI summary

MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding

Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, Nicolas Carion

arXiv:2104.12763v2cs.CVcs.CLcs.LG

TL;DR

Fixed-vocabulary, black-box detectors limit multimodal reasoning over free-form visual concepts. MDETR replaces this pipeline with an end-to-end text-conditioned detector pretrained on aligned image-text data, achieving strong results across grounding, referring, segmentation, question answering, and few-shot detection tasks.

  • Problem

    Black-box detectors trained independently on fixed object and attribute vocabularies limit multimodal systems' access to free-form visual concepts.

  • Method

    MDETR jointly performs object detection and natural language understanding, using text with aligned boxes and soft token prediction to ground detected objects in referring text.

  • Results

    MDETR achieves best reported results on phrase grounding, referring expression comprehension, and referring expression segmentation, with competitive visual question answering and few-shot long-tailed detection performance.

  • Takeaways & Limitations

    Text-modulated pre-training supports downstream multimodal tasks and few-shot detection of rare categories, including settings with as little as 1 example per class.

  • Takeaways & Limitations

    The pre-training data contains no instances with zero boxes aligned to the text, preventing evaluation in a true zero-shot transfer setting.

Abstract

from arXiv · show

Multi-modal reasoning systems rely on a pre-trained object detector to extract regions of interest from the image. However, this crucial module is typically used as a black box, trained independently of the downstream task and on a fixed vocabulary of objects and attributes. This makes it challenging for such systems to capture the long tail of visual concepts expressed in free form text. In this paper we propose MDETR, an end-to-end modulated detector that detects objects in an image conditioned on a raw text query, like a caption or a question. We use a transformer-based architecture to reason jointly over text and image by fusing the two modalities at an early stage of the model. We pre-train the network on 1.3M text-image pairs, mined from pre-existing multi-modal datasets having explicit alignment between phrases in text and objects in the image. We then fine-tune on several downstream tasks such as phrase grounding, referring expression comprehension and segmentation, achieving state-of-the-art results on popular benchmarks. We also investigate the utility of our model as an object detector on a given label set when fine-tuned in a few-shot setting. We show that our pre-training approach provides a way to handle the long tail of object categories which have very few labelled instances. Our approach can be easily extended for visual question answering, achieving competitive performance on GQA and CLEVR. The code and models are available at https://github.com/ashkamath/mdetr.

1. Introduction

MDETR addresses the limits of black-box, fixed-vocabulary detectors with an end-to-end detector conditioned on free-form text. Its text-grounded predictions support strong performance across grounding, referring, segmentation, few-shot detection, and visual question answering tasks.

  • Motivation: Black-box detectors are independently trained, expose only detected regions, and restrict multimodal systems to fixed vocabularies of objects and attributes.This limits co-training with other modalities and makes novel combinations expressed in free-form text difficult to recognize.
  • Approach: MDETR performs end-to-end object detection jointly with natural language understanding using text and aligned boxes as supervision.The approach is based on DETR and detects nuanced concepts from free-form text, including unseen combinations of categories and attributes.
  • Approach: MDETR fuses image and text features through a shared embedding space and transformer encoder-decoder to predict boxes grounded in text.The architecture uses a convolutional visual backbone and a language model such as RoBERTa before joint transformer processing.
  • Results: MDETR achieves best reported results on Flickr30k phrase grounding, RefCOCO/+/g referring expression comprehension, and PhraseCut referring expression segmentation.It also achieves competitive performance on GQA and CLEVR visual question answering.
  • Results: The modulated detection approach transfers to phrase grounding, referring expression comprehension, visual question answering, referring expression segmentation, and few-shot long-tailed detection.These results connect modulated detection performance with downstream task performance across synthetic and real-image datasets.

2. Method

MDETR extends DETR by jointly encoding image and text, then training predictions to align detected objects with referring text tokens. Its losses combine DETR-style matching and box supervision with token prediction and contrastive alignment.

  • DETR background: DETR combines a convolutional backbone with a Transformer Encoder-Decoder whose object queries produce detected boxes and class labels.Hungarian matching supervises matched objects with cross-entropy, L1, and Generalized IoU losses, while unmatched queries predict no object.
  • Architecture: MDETR projects image and text features into a shared space, concatenates them, and processes the sequence with a joint cross encoder.A transformer decoder then uses object queries that cross-attend to the joint representation.
  • Alignment losses: MDETR adds soft token prediction and text-query contrastive alignment losses to encourage alignment between image objects and text.The contrastive loss directly encourages corresponding object and token representations to be closer than unrelated tokens.
  • Alignment losses: Soft token prediction replaces categorical object classes with a uniform distribution over text-token positions referring to each matched object.Unmatched queries predict no object, and multiple objects or tokens may correspond to one another.
  • Alignment losses: The contrastive alignment objective normalizes object-to-token and token-to-object losses over their positive alignments and compares representations through temperature-scaled similarities.The object-side and token-side terms use InfoNCE-inspired denominators over all tokens or objects.
  • Training objective: MDETR combines bipartite matching with soft token, L1, and Generalized IoU terms rather than predicting a class label for each object.The matching cost therefore incorporates text-grounded token supervision alongside box regression.

3. Experiments

The experiments evaluate MDETR on synthetic and natural-image benchmarks spanning modulated detection, grounding, referring expressions, segmentation, VQA, and few-shot long-tailed detection. Results show strong downstream performance, including state-of-the-art or competitive outcomes and gains in low-data detection.

  • Pre-training Modulated Detection: MDETR is evaluated on CLEVR-based datasets, with accuracies reported under standard, human-generated, compositional, and referring-expression settings.CLEVR-Humans is evaluated before and after fine-tuning; CoGenT tests transfer from condition A to B; CLEVR-Ref+ uses uniquely referred objects.
  • Pre-training Modulated Detection: 1.3M aligned image-text pairs are created by combining annotations from Flickr30k, MS COCO, Visual Genome, referring expressions, VG regions, Flickr entities, and GQA.A graph-coloring procedure combines compatible phrases while excluding downstream validation and test images.
  • Pre-training Modulated Detection: MDETR predicts token spans referring to matched objects, allowing text-conditioned detection to distinguish multiple objects described within one caption.Figure 3 illustrates disambiguation among three people using descriptions involving clothing and accessories.
  • Downstream Tasks: 8.5 points without additional pre-training and 12.1 points with pre-training are gained over prior best performance on Flickr30k phrase grounding under the ANY-BOX protocol.The pretrained comparison uses the same backbone as the best model on the test set.
  • Downstream Tasks: MDETR reports consistent gains over HULANet across referring-expression segmentation metrics, with EfficientNet further improving the results.The metrics include mean intersection-over-union and precision Pr@I.
  • Downstream Tasks: Large improvements over state of the art are reported across RefCOCO, RefCOCO+, and RefCOCOg referring expression comprehension datasets.MDETR directly predicts bounding boxes rather than ranking pre-extracted detector proposals.
  • Few-shot transfer for long-tailed detection: With as little as one example per class, MDETR outperforms a fully fine-tuned DETR on rare categories in few-shot LVIS detection.Rare-object performance drops from 20.9 AP with 10% data to 7.5 AP with 100% data under full fine-tuning, likely because of extreme class imbalance.

4. Related work

Related work places MDETR within transformer-based multimodal pre-training and text-conditioned detection. Prior systems use external detectors or specialized supervision, while MDETR emphasizes explicit phrase-box alignment and integrated reasoning.

  • Vision-language reasoning: CLEVR reasoning methods using functional-program annotations perform strongly on synthetic question answering but fail to generalize beyond synthetic data.Other approaches learn directly from images or pre-detected objects, with varying performance.
  • Multimodal pre-training: Multimodal pre-training methods commonly use single-stream or two-stream architectures to learn representations from aligned image-text data.These methods are generally pre-trained and then fine-tuned on downstream tasks.
  • Modulated architectures: Earlier modulated architectures reported improvements on counting and visual question answering, but the related-work discussion distinguishes these efforts from MDETR’s integrated detector formulation.The cited prior approaches are presented as part of the broader multimodal architecture literature.
  • Detector-based systems: External detectors provide noisy, oversampled, and ambiguous visual regions, while tag-based anchoring offers weaker supervision than explicit word-or-phrase alignment.MDETR uses explicit alignment between text and image objects as supervision.

5. Conclusion

The paper concludes that MDETR is a fully differentiable modulated detector with strong multimodal results and potential beyond its evaluated core tasks. It advocates integrated architectures that avoid black-box object detectors.

  • Conclusion: MDETR demonstrates strong performance across multimodal understanding datasets and potential applications in few-shot detection and visual question answering.The conclusion presents these results as evidence for broader downstream applicability.
  • Conclusion: The work aims to enable fully integrated multimodal architectures without relying on black-box object detectors.This conclusion directly addresses the limitations of independently trained detector components.

A. Model details and hyperparameters

MDETR is trained and adapted for text-grounded detection, referring expressions, segmentation, and question answering across natural and synthetic datasets.

  • Training: 40 epochs of pre-training are followed by downstream fine-tuning using AdamW, exponential moving average, and separate learning rates for the backbone and transformer.The reported weight decay is 1e−4 and EMA decay is 0.9998.
  • Phrase grounding: The soft-token classification loss ranks boxes by distributing probability over input-token positions referring to each predicted box.This ranking supports phrase grounding evaluation, including phrases such as “a cat” or “the fence.”
  • Referring expression comprehension: For referring expression comprehension, dependency parsing extracts the sentence root so alignment focuses on the phrase associated with the target box.Fine-tuning is performed for 5 epochs on RefCOCO, RefCOCO+, and RefCOCOg.
  • Dataset characteristics: RefCOCOg contains expressions more than twice as long on average as the other referring-expression datasets, with more descriptive language.The model aligns the box to the root phrase while using the full expression for disambiguation.
  • CLEVR: CLEVR provides synthetic scenes with 3–10 objects and questions requiring up to 20 reasoning steps, while object boxes are reconstructed from rendered scene graphs.MDETR uses a ResNet-18 backbone, DistilRoberta text encoder, and 25 object queries for CLEVR.
  • Question answering: For CLEVR question answering, additional numerical, binary, and attribute queries and specialized decoder heads are trained jointly with modulated detection.The CLEVR checkpoint is first pre-trained on modulated detection before QA fine-tuning.

B.3. Results and discussion

MDETR performs strongly on CLEVR-based reasoning and referring-expression comprehension, while compositional generalization remains limited by learned shape–color biases.

  • Question answering: MDETR closely matches NS-VQA on CLEVR and surpasses methods that do not use external program supervision.The comparison is reported for CLEVR question answering.
  • Question answering: Zero-shot performance on CLEVR-Humans improves substantially over the best competing model, but the gap narrows after fine-tuning.The authors credit the zero-shot improvement to the pre-trained language model.
  • Compositional generalization: Shape-query accuracy falls from 99.98% on testA to 34.68% on testB in CoGenT, indicating a significant compositional generalization gap.The evaluation reverses color–shape pairings between training condition A and zero-shot condition B.
  • Referring expression comprehension: 100% accuracy is reported on CLEVR-REF+ for unique-object expressions, greatly outperforming prior work.Accuracy requires the top-ranked box to achieve IoU of at least 0.5 with the target box.
  • Loss ablations: Removing contrastive alignment causes a drastic drop in modulated-detection AP, with filtering retained for some attributes but not color and texture.The ablation indicates that span prediction alone is insufficient for proper alignment.
  • Loss ablations: Replacing soft-token span prediction with a binary object/no-object label produces similar poor results in the reported ablation.The simplified formulation retains contrastive alignment but removes span distributions.

B.4.2 Question answering ablations

Question-answering ablations show that curriculum pre-training and separate heads for question types materially affect CLEVR performance.

  • Curriculum: Training directly on the full CLEVR set without CLEVR-Medium modulated-detection pre-training leads to poor question-answering accuracy.The model learns to detect only a subset of attributes under this training recipe.
  • Question-answering heads: Specialized heads for numerical, binary, and attribute questions have a big impact on final accuracy compared with a single QA head.The authors hypothesize that separation allows attention patterns to specialize by question type.
  • Evaluation: Table 9 reports class-agnostic AP and question-answering accuracy for the CLEVR ablations.The ablations compare training-recipe and QA-head choices on the validation set.

C. Dataset constructions

The training corpus combines heterogeneous phrase–box annotations while resolving redundancy, multi-object descriptions, and imperfect region references.

  • MS COCO: Referring-expression annotations treat each whole sentence as describing one object, without annotating constituent noun phrases separately.This construction applies to RefCOCO, RefCOCO+, and RefCOCOg annotations added to MS COCO.
  • Visual Genome: Visual Genome supplies diverse, highly descriptive region annotations, including sentences that may cover several concepts.Redundant sentences are detected by normalization before annotation reconciliation.
  • Visual Genome: Equivalent sentences with highly overlapping boxes are treated as redundant and reduced to one annotation.The overlap criterion is IoU > 0.7.
  • Visual Genome: Equivalent sentences with non-overlapping boxes are merged into annotations containing multiple boxes because the sentence may be underspecified.The merging process is iterated recursively until equivalent sentences remain unresolved.
  • Annotation refinement: When individual boxes for objects named within a region description are available, the region box is discarded in favor of those individual boxes.Remaining underspecified descriptions can still introduce noise because matching objects may lack annotations.
  • Qualitative behavior: Figure 13 shows mixed success on unseen attribute combinations: the model can isolate a pink elephant but confuses blue and normal elephants, then labels all three by elimination.The example illustrates both compositional errors and a successful multi-object disambiguation case.

D. Evaluating grounded detection

The section critiques Recall@k-based grounded-detection evaluation when phrases refer to multiple boxes or multiple references share a box. MDETR is additionally evaluated under merged-boxes after fine-tuning to match that protocol.

  • Evaluation challenges: Recall@k evaluates whether a correct box appears among the top k, using an IoU threshold commonly set to 0.5.The metric is suited to one-to-one phrase–box mappings but becomes ambiguous when phrases or boxes have multiple references.
  • Evaluation challenges: When one phrase refers to several objects, the ANY-Protocol accepts a prediction if it overlaps any ground-truth box.This preserves instance-level boxes but does not test whether all referred instances were found.
  • Evaluation challenges: Merged-box evaluation can produce a bloated box when referred instances are spatially separated.The alternative sacrifices fine-grained instance evaluation in favor of a single merged region.
  • Evaluation challenges: Evaluating co-referring phrases independently does not test whether the model understands that they refer to the same object.Examples include exact duplicates, synonyms, and references to a group or one of its sub-constituents.
  • Protocol adaptation: MDETR is evaluated under the merged-boxes protocol because literature-wide evaluation procedures differ.The model requires fine-tuning on appropriately merged Flickr30k Entities boxes, whereas the ANY-Protocol does not.

E. Error Analysis

The error analysis finds that annotation problems dominate the first 100 Flickr30k validation errors, while remaining failures involve instance selection, object concepts, OCR, and box precision.

  • Ground-truth errors: Ambiguous or inconsistent annotations include scene elements without canonical locations, missed instances, and distinct instances assigned one box.Other annotation errors involve boxes that are too large, too small, or unrelated to the referred object.
  • Model errors: Wrong-instance errors occur when the model selects the correct object type but ignores contextual information needed to identify the intended instance.This failure concerns contextual disambiguation rather than category recognition.
  • Model errors: Wrong-object errors often arise for long-tail concepts that the model does not appear to recognize.The model therefore selects an object that is not of the correct type.
  • Model errors: OCR errors occur when the referring phrase denotes written text, while imprecise predictions affect small and elongated objects.For elongated or small objects, a relatively small L1 error can yield low IoU with the ground truth.
  • Ground-truth errors: More than half of the analyzed errors stem from ground-truth annotation issues.The authors extrapolate this to approximately 10% label noise across the dataset.
  • Error breakdown: Figure 14 partitions the first 100 EfficientNetB5 errors into annotation mistakes, grounding errors, and localization errors.The figure uses yellow, blue, and green shading for these categories, respectively.

F. Experiments on VQA2

The VQA2 experiments differ from GQA because VQA2 lacks bounding-box annotations and therefore uses only question-answering supervision. On GQA, detection supervision is withheld during balanced-set fine-tuning.

  • Supervision: GQA experiments always had bounding-box information for the questions.This provides box supervision during the relevant experiments except where explicitly withheld.
  • Supervision: During balanced-set fine-tuning on GQA, detection losses are not supervised for 10 epochs.The passage identifies this as the exception to the usual access to bounding-box information.
  • Supervision: VQA2 experiments use no box annotations, so supervision comes solely from the question-answering loss.This contrasts with the GQA setup described in the same passage.
Loading 2104.12763v2…