GRiT: A Generative Region-to-text Transformer for Object Understanding

• Prior object detectors [1] [20] [27] [46] classify among closed-set nouns, and open-vocabulary detectors [12] [40] [44] still rely on defined class-name embeddings.These closed-set frameworks choose among a limited set of candidate labels.Source: §1 · pp. 1–2
• GRiT generates descriptions without a predefined category list, including attributes and actions, and can continuously learn added object classes without architectural adaptation.Its open-set generation spells words from text tokens, making the approach more challenging than closed-set recognition but suited to evolving data.Source: §1 · p. 2
• GRiT localizes image regions and autoregressively generates free-form descriptions, unifying object detection and dense captioning in one framework.Its visual encoder extracts image features, a foreground object extractor detects and crops object regions, and a text decoder generates descriptions from object features.Source: §1 · p. 2
• A special [task] token controls task-specific descriptions when GRiT is jointly trained on short- and long-description tasks, avoiding confusion between their outputs.The same architecture can train on object detection and dense captioning separately or together.Source: §1 · p. 2
• 60.4 AP on COCO object detection is comparable to closed-set standard detectors, while 15.5 mAP on Visual Genome dense captioning surpasses all dense captioning models.Source: §1 · p. 2

Related Work

• Classic R-CNN and transformer detectors achieve strong localization and classification but recognize only fixed object categories [1] [3] [27] [2] [41] [47].Source: §2 · p. 2
• Open-vocabulary detectors expand recognition beyond dataset categories using vision-language knowledge, but still require a predefined category set and do not generate descriptive sentences [12] [40] [44].RegionCLIP matches region features with CLIP text embeddings [26], ViLD [12] distills CLIP vision knowledge, and Detic [44] learns from large-scale image labels.Source: §2 · pp. 2–3
• Inspired by generative image-to-text methods [14] [33] [38], GRiT generates open-set, free-form region descriptions ranging from class names to richer sentences.Unlike category-list approaches, it does not require humans to define categories.Source: §2 · p. 3
• Unlike dense captioning models [15] [36] [37], GRiT unifies object detection and dense captioning and achieves state-of-the-art dense captioning without complex relation or context modeling.Source: §2 · p. 3

3.1. Architecture

• GRiT is end-to-end and combines a visual encoder, foreground-object extractor, and text decoder using the image-to-text transformer structure.The visual encoder uses ViT window self-attention with four global-attention blocks and builds a five-scale feature pyramid by up- and down-sampling ViT features.Source: §3.1 · p. 3
• The foreground-object extractor generates proposals, refines boxes with an RoI head, scores foreground versus background, and filters low-scoring or overlapping boxes.Its RoI head uses binary foreground/background classification rather than the multi-category classification of standard object detectors.Source: §3.1 · pp. 3–4
• The decoder turns cropped object features into autoregressive WordPiece tokens, enabling open-set descriptions informed by object appearance and global image context.It uses a six-layer transformer and a 30,522-token vocabulary; global self-attention supplies image context beyond the object crop.Source: §3.1 · p. 4
• Task-specific begin tokens tell the jointly trained model whether to generate short class names or richer descriptive sentences.Training selects the begin token associated with each annotation's task, and inference uses that token to indicate the desired description style.Source: §3.1 · p. 4

3.2. Training

• GRiT's loss sums object-extractor losses and language-modeling loss, with the latter applied only to predicted foreground objects and using cross-entropy label smoothing of 0.1.The object-extractor term includes box and classification losses for both the proposal generator and RoI head.Source: §3.2 · p. 4

3.3. Inference

• At inference, COCO detection uses beam size k=3 while dense captioning uses no beam search; confidence multiplies the square roots of objectness and mean token-description scores.Beam search expands the top k first-token choices into candidate descriptions; the decoder score averages scores across the description's tokens.Source: §3.3 · p. 4

4. Experiments

• The experiments use COCO for detection and Visual Genome for dense captioning, whose descriptions include attributes and actions as well as nouns.COCO detection uses box AP and AR across IoU thresholds 0.5–0.95; Visual Genome mAP averages localization thresholds .3–.7 and METEOR thresholds 0–.25. Evaluation removes generated COCO labels outside its 80 classes.Source: §4 · pp. 4–5

4.1. Implementation Details

• The implementation uses ViT-B, ViT-L, or ViT-H visual encoders and CenterNet proposals followed by a three-stage Cascade R-CNN RoI head.CenterNet generates 2,000 proposals during training and 256 during testing; the decoder receives the box predicted at the last stage.Source: §4.1 · p. 5

4.2. Joint Object Detection and Dense Captioning

• Separate training outperforms joint COCO–Visual Genome training, while task-specific begin tokens produce requested description styles more consistently than a shared [BOS] token.The authors attribute the joint-training drop to disagreements in box annotations: VG includes overlapped and COCO-background boxes, which can cause COCO false positives and lower VG recall.Source: §4.2 · pp. 5–6, Figure 4 · p. 6, Table 1 · p. 6

4.3. Ablation Studies

• GRiT’s open-set detector is comparable to a matched closed-set detector at a 0.8 AP gap; beam search improves recall, and description scoring helps reject false alarms.Using 49 rather than 196 object feature vectors yields similar performance. At beam size 1, description scoring improves AP by 0.9; without it, beam size 3 significantly worsens precision and AP.Source: §4.3 · p. 6, Table 2 · p. 7
• MAE pre-training outperforms GIT pre-training, especially at huge model size, while GRiT can learn new classes added during training without architecture changes.Adding the remaining classes in the last third of training gives similar results to training on all classes throughout; performance remains reasonable when they are added in the last ninth. GIT pre-training performs slightly better on dense captioning.Source: §4.3 · p. 7, Table 2 · p. 7, Table 3 · p. 7

4.4. Comparison to State-of-the-Arts on COCO

• GRiT performs comparably to state-of-the-art closed-set COCO detectors despite using an open-set framework.Listed detector results use their papers’ best settings, so training and testing configurations differ; some methods use extra detection data or larger input images.Source: §4.4 · p. 7, Table 4 · p. 8

4.5. Comparison to State-of-the-Arts on VG

• 4 mAP: GRiT outperforms the current best Visual Genome dense-captioning model without object-relation modeling.The paper notes that absolute dense-captioning metric values are low, hypothesizing that dense region annotations and a strict language metric contribute.Source: §4.5 · p. 7, Table 5 · p. 8

4.6. Zero-shot Object Understanding

• A zero-shot setup uses a COCO-caption-finetuned GIT model with its parameters fixed, enabling region-level descriptions but not reliably for every image.The feature pyramid and foreground extractor are finetuned on COCO detection data without class names; the authors hypothesize that the decoder struggles with cropped features because it was trained on whole-image features.Source: §4.6 · p. 8, Figure 5 · p. 8

5. Conclusion

• GRiT unifies region- and object-level tasks by representing object understanding as region-text pairs.The end-to-end framework extracts image features, detects foreground objects, and generates object descriptions.Source: §5 · p. 9
• Experiments on object detection and dense captioning demonstrate GRiT’s effectiveness and generality.Source: §5 · p. 9