Source-linked AI summary
Recognize Anything: A Strong Image Tagging Model
Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, Yandong Guo, Lei Zhang
TL;DR
RAM addresses limited category generalization and noisy, incomplete image-tagging data with a foundation model trained through a unified tagging-and-captioning pipeline and data-engine refinement. It demonstrates zero-shot recognition across common categories, outperforming fully supervised models and generalist approaches such as CLIP and BLIP, while retaining limitations on abstract and fine-grained tasks.
Problem
Existing tagging models such as Tag2Text are limited to fixed, predefined categories, while image-tagging development faces challenges involving the label system, dataset, data engine, and model design.
Method
RAM uses a unified label system and large-scale image-text data, combines image tagging with captioning, and applies a data engine to add missing tags and remove incorrect labels.
Results
RAM demonstrates zero-shot recognition of any common category with high accuracy, surpassing fully supervised models and existing generalist approaches such as CLIP and BLIP.
Takeaways & Limitations
RAM's semantic label queries support generalization to previously unseen categories and may empower recognition across diverse visual datasets and tasks.
Takeaways & Limitations
RAM struggles with abstract tasks such as object counting, lags behind task-specific models on fine-grained classification, and may reflect biases from open-source training datasets.
Abstract
from arXiv · showhide
We present the Recognize Anything Model (RAM): a strong foundation model for image tagging. RAM makes a substantial step for large models in computer vision, demonstrating the zero-shot ability to recognize any common category with high accuracy. RAM introduces a new paradigm for image tagging, leveraging large-scale image-text pairs for training instead of manual annotations. The development of RAM comprises four key steps. Firstly, annotation-free image tags are obtained at scale through automatic text semantic parsing. Subsequently, a preliminary model is trained for automatic annotation by unifying the caption and tagging tasks, supervised by the original texts and parsed tags, respectively. Thirdly, a data engine is employed to generate additional annotations and clean incorrect ones. Lastly, the model is retrained with the processed data and fine-tuned using a smaller but higher-quality dataset. We evaluate the tagging capabilities of RAM on numerous benchmarks and observe impressive zero-shot performance, significantly outperforming CLIP and BLIP. Remarkably, RAM even surpasses the fully supervised manners and exhibits competitive performance with the Google tagging API. We are releasing the RAM at \url{https://recognize-anything.github.io/} to foster the advancements of large models in computer vision.
1. Introduction
Image tagging remains limited by weak recognition scope or accuracy, scarce large-scale high-quality annotations, and inflexible model designs. RAM addresses these bottlenecks with a unified label system, annotation-free image-text data, automatic data cleaning, and semantic label queries for zero-shot recognition.
- Motivation: Image tagging supplies semantic labels for multiple objects, scenes, attributes, and actions, but existing approaches often have limited scope or poor accuracy.The paper positions tagging as a foundational task alongside localization.
- Motivation: Large-scale high-quality tagging data is difficult to collect because universal labels and efficient automatic annotation engines are lacking.The paper identifies data collection as one of two core bottlenecks.
- RAM approach: RAM establishes a unified 6,449-category label system by merging public dataset tags, commercial tags, and common text tags.Remaining open-vocabulary labels can be identified through open-set recognition.
- RAM approach: RAM obtains annotation-free image tags from large-scale image-text pairs through automatic text semantic parsing.This adapts image-text resources used to train powerful visual models for image tagging.
- RAM approach: A tagging data engine supplements missing tags, localizes tag regions, removes class outliers, and filters whole-image versus region prediction conflicts.These operations target noisy web-derived labels and produce cleaner annotations.
- RAM approach: RAM combines captioning and tagging with semantic label queries, enabling recognition of previously unseen categories beyond fixed predefined classes.The model uses semantically rich textual context in its label queries.
- Reported advantages: RAM demonstrates strong, general zero-shot tagging, while offering open-source, annotation-free training and flexible deployment for selected classes or combined localization pipelines.The strongest version requires three days of training on eight A100 GPUs.
2. Recognize Anything Model
RAM combines image tagging and captioning modules with semantically informed label queries to recognize fixed and previously unseen categories. Its lightweight recognition decoder supports efficient, customizable inference across label sets.
- Architecture: RAM uses an image encoder, image-tag recognition decoder, and text-generation encoder-decoder to support tagging and captioning.Cross-attention links image features with tags, while the recognition head predicts parsed tags during training and bridges images to tags during inference.
- Open-Vocabulary Recognition: RAM introduces open-vocabulary recognition, allowing categories beyond those seen during training instead of restricting predictions to fixed categories.Textual label queries encode individual tags with an off-the-shelf text encoder, supplying semantic context for unseen categories.
- Implementation: RAM uses a Swin Transformer image encoder, CLIP-based textual label queries with prompt ensembling, and CLIP image-feature distillation.The implementation also uses a 12-layer text-generation transformer and a 2-layer tag-recognition transformer.
- Training: RAM is pretrained at 224 resolution and fine-tuned at 384 resolution on smaller, higher-quality datasets, typically converging in fewer than five epochs.The authors report that this rapid convergence improves reproducibility with limited computational resources.
- Inference: Removing self-attention from the recognition decoder improves efficiency and prevents potential interference between label queries.During inference, users can customize both the categories and the number of label queries.
3. Data
RAM builds its tagging data around a unified label system and large-scale image-text datasets, then uses an automatic data engine to expand and clean annotations. The resulting pipeline addresses missing and incorrect labels in web-crawled data.
- Label System: The label system is designed to prioritize frequent tags, represent varied domains and contexts, and maintain a moderate tag quantity.Its tag conception includes objects, scenes, attributes, and actions.
- Label System: RAM’s label system combines public dataset tags, commercial tagging categories, and common text-derived tags into 6,449 fixed labels.Remaining open-vocabulary labels can be identified through open-set recognition.
- Dataset: Pretraining uses 4M and 14M image settings built from human-annotated and web-based image-caption datasets.The 4M setting includes COCO, Visual Genome, Conceptual Captions, and SBU Captions.
- Data Engine: Web-crawled image-text data contain missing and incorrect labels, motivating an automatic engine that generates additional tags and cleans erroneous ones.The engine supplements annotations and removes incorrect tags through generation and region-based cleaning procedures.
4. Experiment
RAM is evaluated across classification, detection, segmentation, and vision-language comparisons, with results showing broad zero-shot and open-set tagging capability. Ablations examine model components and data-engine improvements, while benchmark construction and metrics address annotation quality and comparability.
- Experimental setting: Evaluation uses mAP for classification comparisons and ablations, while Precision/Recall with manually adjusted thresholds supports comparisons when mAP is unavailable.
- Classification comparisons: RAM’s zero-shot generalization to OpenImages-common surpasses ML-Decoder’s fully supervised performance and extends to OpenImages-rare categories unseen during training.
- Classification comparisons: RAM-4M surpasses ML-Decoder on OpenImages-common despite using 4 million annotation-free image-text images versus 9 million annotated images.The training data yield 39.3 million common tags, compared with 33.9 million common tags from ML-Decoder’s training data.
- Detection and segmentation comparisons: RAM demonstrates open-set ability beyond supervised detection and segmentation models, which face broader-category challenges and greater computational overhead from localization.
- Vision-language comparisons: RAM outperforms CLIP and BLIP by over 20% accuracy across almost all datasets, but performs slightly worse on OpenImages-rare.The authors attribute the rare-category discrepancy to RAM’s smaller training dataset and limited emphasis on rare classes.
- Model ablations: Joint captioning and tagging improves tagging, while textual queries enable open-set recognition with little effect on seen categories.
- Model ablations: Expanding the label system minimally affects existing categories but improves coverage and open-set ability for unseen categories.
- Data engine ablations: Increasing tags from 12.0M to 41.7M and training images from 4M to 14M improves performance across all test sets.Cleaning tags slightly improves OPPO-common and OpenImages-common; COCO Caption fine-tuning substantially improves those two sets.
5. Conclusion
RAM is presented as a strong image-tagging foundation model with high-accuracy zero-shot recognition that surpasses fully supervised models and generalist approaches. The authors identify remaining limitations in scaling, abstract and fine-grained recognition, and dataset bias.
- RAM demonstrates zero-shot recognition of any category with high accuracy, surpassing fully supervised models and existing generalist approaches such as CLIP and BLIP.
- Further refinement could scale training beyond 14 million images, expand domain coverage, repeat the data-engine process, and increase backbone parameters.
- RAM efficiently recognizes common objects and scenes but struggles with abstract tasks such as object counting.
- Zero-shot RAM trails task-specific models on fine-grained classification, including car-model and flower- or bird-species identification.
- Because RAM is trained on open-source datasets, it could potentially reflect dataset biases.