Source-linked AI summary
GRiT: A Generative Region-to-text Transformer for Object Understanding
Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, Lijuan Wang
TL;DR
GRiT addresses the limitation of closed-set object recognition by formulating object understanding as region-text generation. It combines visual encoding, foreground object extraction, and text decoding to support detection and dense captioning, reporting strong results on both tasks.
Problem
Closed-set and open-vocabulary detectors remain tied to predefined object categories or class-name embeddings.
Method
GRiT uses a region-to-text architecture that localizes objects and autoregressively generates open-set descriptions, with beam search supporting multiple detection labels.
Results
GRiT reports 60.4 AP on COCO object detection and 15.5 mAP on Visual Genome dense captioning.
Takeaways & Limitations
One open-set framework can unify object detection and dense captioning across short labels and rich descriptive sentences.
Abstract
from arXiv · showhide
This paper presents a Generative RegIon-to-Text transformer, GRiT, for object understanding. The spirit of GRiT is to formulate object understanding as <region, text> pairs, where region locates objects and text describes objects. For example, the text in object detection denotes class names while that in dense captioning refers to descriptive sentences. Specifically, GRiT consists of a visual encoder to extract image features, a foreground object extractor to localize objects, and a text decoder to generate open-set object descriptions. With the same model architecture, GRiT can understand objects via not only simple nouns, but also rich descriptive sentences including object attributes or actions. Experimentally, we apply GRiT to object detection and dense captioning tasks. GRiT achieves 60.4 AP on COCO 2017 test-dev for object detection and 15.5 mAP on Visual Genome for dense captioning. Code is available at https://github.com/JialianW/GRiT
GRiT
GRiT is presented as a general object-understanding framework, with the paper acknowledging an author internship contribution.
- GRiT is introduced as a general object-understanding framework.
- The paper notes that the work was done during an author’s internship at Microsoft.
1. Introduction
GRiT addresses the limits of closed-set object recognition by generating open-set descriptions for localized objects. Its unified architecture supports both class-name detection and richer descriptive tasks, achieving strong reported results on COCO and Visual Genome.
- Existing and open-vocabulary detectors still classify objects using predefined categories or class-name embeddings.
- GRiT generates free-form object descriptions without a predefined category list, including attributes and actions.
- GRiT uses a visual encoder, foreground object extractor, and autoregressive text decoder to localize and describe objects.
- Task-specific begin tokens let one architecture generate short detection labels or long dense-caption descriptions during joint training.
- GRiT frames object understanding as a unified region-level task framework.
- 60.4 AP on COCO object detection and 15.5 mAP on Visual Genome dense captioning are reported for GRiT.
2. Related Work
Related work spans fixed-category detection, open-vocabulary recognition, and generative image-to-text methods. GRiT extends generation to regions while unifying object detection and dense captioning.
- Standard object detectors achieve strong localization and classification but recognize a fixed set of categories.
- Open-vocabulary detectors recognize categories absent from detection datasets but still rely on predefined class-name embeddings.
- The related-work discussion notes that open-vocabulary methods do not generate descriptive sentences.
- Generative image-to-text methods motivate GRiT’s free-form generation of region-level object descriptions.
- Unlike prior dense-captioning models, GRiT unifies object detection and dense captioning in one framework.
3. GRiT
GRiT combines visual encoding, foreground region extraction, and autoregressive text generation in an end-to-end region-to-text pipeline. Training and inference add task control, language modeling, beam search, and combined object-description scoring.
- Architecture: GRiT uses a visual encoder, foreground object extractor, and text decoder in an end-to-end architecture.
- Visual Encoder: The visual encoder uses a backbone and feature pyramid, while the foreground extractor predicts and refines object boxes.
- Foreground Object Extractor: The foreground extractor predicts objectness, removes overlapping boxes with NMS, and filters low-confidence boxes.
- Text Decoder: The text decoder crops fixed-size object features and autoregressively generates tokenized descriptions using visual context.
- Text Decoder: GRiT’s token-based decoder supports open-set descriptions because words can be represented as combinations of text tokens.
- Multi-task Training: Task-specific begin tokens control whether jointly trained models generate short or rich object descriptions.
- Training: Training combines foreground extraction loss with language-modeling loss applied to foreground objects.
- Inference: Beam search produces multiple class-name descriptions for detection, with beam size k = 3 found sufficient on COCO.
4. Experiments
GRiT is evaluated on object detection and dense captioning, with ablations examining training, scoring, decoding, feature inputs, and pre-training. It performs competitively on object detection and strongly on dense captioning, while joint training remains limited by annotation disagreement.
- Experimental Setup: GRiT is evaluated on COCO object detection and Visual Genome dense captioning, using ViT-based visual encoders and a foreground object extractor.COCO evaluates box AP and AR, while Visual Genome provides dense region descriptions.
- Joint Training: Separately trained models outperform the jointly trained model because COCO and Visual Genome disagree about foreground and box annotations.The disagreement produces false positives on COCO and may lower recall on Visual Genome.
- Joint Training: Unique task-specific begin tokens are necessary for jointly generating consistent detection-style and dense-captioning descriptions.Using only [BOS] mixes dense captions and COCO class names within the same image, especially for COCO-background regions.
- Ablation Studies: Beam search improves object detection, with beam size=3 offering a trade-off between accuracy and inference time.Beam search is particularly helpful for recall by allowing multiple class names for one box.
- Ablation Studies: Combining objectness and description scores improves ranking, while 49 object feature vectors perform similarly to 196.Description scoring adds 0.9 AP at beam size=1 and prevents shared-confidence false positives when beam size=3.
- Comparisons and Further Studies: GRiT supports incremental class expansion and benefits from larger models and MAE pre-training, while remaining comparable to closed-set detectors and surpassing dense-captioning models by 4 mAP.Adding 20 classes during training achieves similar results to training all classes throughout when added in the last one-third, and remains reasonable in the last one-ninth.
5. Conclusion
GRiT is a general, open-set framework that represents object understanding as region-text pairs and unifies region-level tasks. Its end-to-end architecture and experiments demonstrate effectiveness on object detection and dense captioning.
- GRiT formulates object understanding as region-text pairs, enabling multiple region/object-level tasks within one paradigm.
- The framework is end-to-end, spanning image feature extraction, foreground object detection, and object description generation.
- Experiments on object detection and dense captioning demonstrate GRiT’s effectiveness and generality.