Source-linked AI summary
The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World
Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, Yushi Chen, Tong Lu, Jifeng Dai, Yu Qiao
TL;DR
Open-world visual understanding lacks both abundant instance-level semantic data and models that use spatial information. The paper builds AS-1B through a semi-automatic data engine with human feedback and develops ASM, a unified location-aware vision-language model. ASM reports zero-shot region-recognition gains over CLIP and stronger performance than recent image- and region-level VLLMs across zero-shot and fine-tuned evaluation.
Problem
Open-world visual understanding faces scarce instance-text data and models that lack spatial-information modeling.
Source: §1 · p. 2Method
The paper builds AS-1B with a scalable semi-automatic data engine and human feedback, then develops ASM, a unified location-aware image-text model.
Source: §1 · p. 2Results
Takeaways & Limitations
The AS project brings region-level data and a unified model together for open-world panoptic visual recognition and understanding.
Source: §1 · pp. 2–3
Abstract
from arXiv · showhide
We present the All-Seeing (AS) project: a large-scale data and model for recognizing and understanding everything in the open world. Using a scalable data engine that incorporates human feedback and efficient models in the loop, we create a new dataset (AS-1B) with over 1 billion regions annotated with semantic tags, question-answering pairs, and detailed captions. It covers a wide range of 3.5 million common and rare concepts in the real world, and has 132.2 billion tokens that describe the concepts and their attributes. Leveraging this new dataset, we develop the All-Seeing model (ASM), a unified framework for panoptic visual recognition and understanding. The model is trained with open-ended language prompts and locations, which allows it to generalize to various vision and language tasks with remarkable zero-shot performance, including region-text retrieval, region recognition, captioning, and question-answering. We hope that this project can serve as a foundation for vision-language artificial general intelligence research. Models and the dataset shall be released at https://github.com/OpenGVLab/All-Seeing, and demo can be seen at https://huggingface.co/spaces/OpenGVLab/all-seeing.
1 Introduction
The paper targets gaps in open-world instance-level visual understanding: language models lack visual understanding, while existing visual datasets and models often lack instance-level semantics or spatial modeling. It pairs a semi-automatic, human-verified region dataset with a location-aware vision-language model.
1 Introduction
- The project addresses the limits of language-only visual perception and of open-world models that focus on whole images rather than individual instances.The paper identifies scarce open-world instance-text pairs and inadequate spatial-information modeling as two major challenges.Source: §1 · pp. 1–2
- A data-human-model loop uses model-generated annotations, human verification and feedback, and model fine-tuning to iteratively re-annotate regions.The dataset includes more than 3.5 million semantic tags, plus 3.3 billion visual question-answering pairs and 1.2 billion region captions for 1.2 billion regions.Source: §1 · p. 2
- ASM combines a location-aware image tokenizer conditioned on boxes, masks, or point sets with an LLM-based decoder and a shared decoding approach for alignment and generation.The shared approach reformulates alignment tasks as a special generation task.Source: §1 · p. 3
- The project’s stated contributions span instance-level recognition, image-text retrieval and generation, and a dataset-model data engine that iteratively improves scale, diversity, and model performance.The authors describe the system as supporting both discriminative and generative image-text tasks.Source: §1 · p. 3
- AS-1B is a large-scale dataset for open-world panoptic visual recognition and understanding, with 159 times more semantic tags and 33 times more regions than its counterparts, using an economical semi-automatic data engine.Its annotations include semantic tags, locations, question-answering pairs, and captions.Source: §1 · pp. 2–3
- 10.4 and 14.3 mAP over CLIP [67] on COCO [53] and LVIS [31], respectively, in zero-shot region recognition; ASM also outperforms recent image- and region-level VLLMs.The broader comparison covers both zero-shot and fine-tuned performance when ASM is trained with AS-1B and LaionCOCO.Source: §1 · p. 3
2 Related Work
Related work traces how visual datasets and models have expanded toward open-world understanding, while noting persistent gaps in region-level semantics, spatial reasoning, and task versatility.
2 Related Work
- Large language models trained on large text corpora show capabilities across language tasks, but the cited discussion notes growing concern about leading institutes sharing fewer technical details and roadmaps.The passage presents LLM world knowledge as groundwork for open-world understanding.Source: §2 · p. 3
- Prior visual datasets range from closed-world recognition and location benchmarks to web-scale image-text collections, but image-level descriptions do not provide instance-specific understanding.The related-work discussion also notes that SA-1B supplies open-world locations but lacks semantic details; the Table 1 caption reports greater scale and diversity for AS-1B than prior region-level datasets.Source: §2 · pp. 3, 5, Table 1 · p. 4
- Existing multimodal models span contrastive matching, generative training, and LLM-based systems, but the reviewed LLM-based models lack specific-region understanding or support only generative tasks.Location-aware methods are also discussed, with their generalization described as limited without large-scale instance-level visual-understanding data.Source: §2 · p. 5
- Earlier visual-language systems are described as trading off generation ability, use of pretrained vision and language foundations, and region-level understanding.The discussion characterizes contrastive models as lacking generation, some generative models as trained from scratch, and many LLM-based multimodal models as whole-image focused.Source: §2 · p. 5
3 The All-Seeing Dataset (AS-1B)
AS-1B provides 1.2 billion richly annotated regions across 11 million images, built with a semi-automatic data engine. Its localization pipeline combines proposals from multiple model types and merges overlapping regions while retaining their semantics.
3 The All-Seeing Dataset (AS-1B)
- AS-1B contains 1.2 billion regions in 11 million images, with categories, locations, attributes, captions, and question-answer pairs.Its instance-level location annotations and object descriptions distinguish it from earlier visual recognition and understanding datasets.Source: §3 · p. 5
3.1 Data Annotation Engine
- A semi-automatic data-human-model cycle uses foundation models to generate pseudo-data, then iteratively refines annotations with models fine-tuned on human feedback.The resulting annotations feed back into the pipeline to improve data quality.Source: §3.1 · p. 5, Figure 3 · p. 6
- The pseudo-data pipeline generates locations, semantic tags, region-tag matches, attributes, and detailed captions using ensembles of vision and language models.For locations, SAM proposals are combined with closed-set detections and grounding boxes to cover objects in view.Source: §3.1 · p. 6, §3.2 · p. 6
3.2 Open-World Localization
- Overlapping proposals are merged by IoU: high-overlap regions are removed while their tags are appended to the matched region, and low-overlap proposals are retained.This sequentially combines proposals from SAM, InternImage, EVA-02, and GLIP without relying on incomparable model score ranges.Source: §3.2 · p. 6
3.3 Open-World Semantic
The semantic pipeline uses language and vision-language models to generate open-world tags and detailed region descriptions. Specialized modules expand scene concepts, identify region-specific candidates, and compose attributes and answers into captions.
3.3 Open-World Semantic
- LLMs and VLLMs act as semantic generators for short tags or detailed annotations, drawing on world knowledge and reasoning to describe diverse objects.The annotations include attributes, question-answering pairs, and captions.Source: §3.3 · p. 7
3.3.1 Semantic Tags
- Four complementary taggers identify prominent objects, imagine plausible scene objects, split concrete objects into parts, and caption cropped regions for missing tags.The spotter also extracts text with OCR; the imaginator expands scene-specific candidates, while the splitter excludes nonphysical or indivisible concepts.Source: §3.3.1 · pp. 7–8
3.3.2 Detailed Descriptions
- Detailed descriptions are built by generating attribute or status questions, answering them from region content, and combining the answers into a caption.Vicuna generates three questions per tag, Husky answers using region content, and the writer paraphrases the answers into one sentence.Source: §3.3.2 · p. 8
3.4 Matching Location and Semantic
The matching pipeline assigns semantic tags to regions by measuring region-text similarity. It addresses cases where a bounding box contains a secondary object that could otherwise outrank the main object.
3.4 Matching Location and Semantic
- A region-text alignment model measures similarity between each region and its candidate semantic tags to select the tags most related to that region.The candidate list combines tags from language-model annotators with tags from closed-set and grounding detectors.Source: §3.4 · p. 9
- CLIP confidence is modulated by the CLIPSeg mask area for each candidate, helping select the main object when a box also contains another object.The issue arose because CLIP alone could classify a box framing a person as a backpack when the person carried one.Source: §3.4 · p. 9
3.5 Human Verification
Human verification is used to address noise in automated annotations, including errors caused by cropping and flawed question-answer pairs. The process samples semantic tags, corrects or rejects questionable answers, and subjects verified annotations to expert review.
3.5 Human Verification
- Cropping can discard context and cause automated labels to confuse objects with similar shapes or colors, motivating human verification.A lampshade may be described as a cup.Source: §3.5 · p. 9
- Concept-wise sampling targets long-tail tags by sampling six regions from least-frequent concepts and 90 from the most frequent concepts.Annotators filter incorrect tags from the top five candidates for each sampled region.Source: §3.5 · p. 9
- Question-answer pairs are checked for wrong answers, unanswerable questions, or incorrect semantic tags, with human correction or rejection answers assigned as appropriate.Pairs marked correct are retained; wrong answers are re-annotated by humans in a second stage.Source: §3.5 · p. 9
- Fifty annotators verify model-generated annotations, and ten experts review them in groups of 100; packages below 95% accuracy are sent back for re-verification.Source: §3.5 · p. 9
3.6 Data Engine Iteration
The data engine iteratively uses model-generated annotations, human review, and model fine-tuning to improve annotations. Its loop repeats refinement until data quality meets requirements.
3.6 Data Engine Iteration
- The loop generates initial annotations, trains ASM, verifies and corrects sampled annotations with humans, then fine-tunes ASM on the verified data.The fine-tuned model re-ranks semantic tags and generates region captions and answers before the cycle repeats.Source: §3.6 · p. 10
4 The All-Seeing Model (ASM)
ASM combines location-aware visual tokens with an LLM-based decoder to handle image- and region-level vision-language tasks. Task prompts guide generation or discrimination within a unified model.
4.1 Overal Architecture
- A location-aware tokenizer and LLM decoder support contrastive and generative tasks at both image and region levels.The stated task range includes retrieval, classification, question answering, reasoning, captioning, and grounding with a class-agnostic detector.Source: §4.1 · p. 10
- ASM trains with next-token prediction for generation and contrastive region-text alignment for discriminative tasks.Each region is treated as an image when calculating the CLIP-style discriminative loss.Source: §4.1 · pp. 10–11
4.2 Location-Aware Image Tokenizer
- The tokenizer conditions queries on boxes, masks, or points, extracts region features, and converts them into soft-prompt visual tokens for decoding.Without location input, the box is assumed to cover the whole image, giving a consistent approach to region and image tokenization.Source: §4.2 · p. 11
4.3 LLM-Based Decoder
- Generative prompts combine task tokens, location-aware visual tokens, and user instructions, after which the LLM generates response tokens autoregressively.The task prompt contains five tokens.Source: §4.3 · p. 12
- For discriminative tasks, a trainable ⟨align⟩ token represents the input sequence, and its embedding supplies the region-text matching similarity.Text inputs use task prompts and caption or class-name tokens, while vision tokens are omitted to avoid information leakage.Source: §4.3 · p. 12
- The LLM-based decoder is intended to leverage LLM world knowledge and reasoning, while generating both image and text alignment embeddings through an LLM.The authors describe the latter as bridging the pre-training task for language models and language-image pre-training.Source: §4.3 · p. 12
5 Data Analysis
AS-1B combines large-scale region annotations with diverse open-world concepts, while its analysis examines annotation quality and the human effort required to refine the data.
5.1 Data Scale
- 1.2 billion region-text pairs from 11 million images cover 3.5 million semantic tags, and detailed captions contain about 42.2 billion tokens.The full detailed-description statistics report 132.2 billion tokens.Source: §5.1 · pp. 12–13, Table 4 · p. 14
- AS-1B has about 33 times more regions than BigDetection and 1,941 times more VQA pairs and 222 times more region captions than Visual Genome.Its semantic-tag count is about 159 times ImageNet-22k's and 268 times V3Det's category count.Source: §5.1 · p. 13, §5.2 · p. 14
5.2 Data Diversity
- AS-1B covers fine-grained categories, proper nouns, object parts, and attributes, with frequent broad tags and rare fine-grained or attribute-specific tags.LLMs supply 55% and BLIP 24% of top-1 semantic candidates; closed-set detectors supply 19%.Source: §5.2 · pp. 13–14
- VQA-guided region captions are longer and more informative; captions generated without semantic-tag and question guidance can be inaccurate or hallucinatory.The detailed-description statistics include 132.2 billion tokens.Source: §5.2 · p. 14, Table 4 · p. 14
5.3 Data Quality
- Verified annotations yield 54.8% top-1 accuracy and 47.0% candidate accuracy, while VQA pairs have 47.1% accuracy.LLMs and BLIP are less accurate on small regions, whereas closed-set detectors perform better there; the pattern reverses for larger regions.Source: §5.3 · p. 14
- Verifying every million regions takes about 2,750 working hours, but 50 annotators can verify them in approximately 15 days; annotating all regions would take about 42 years.The authors therefore describe model annotation corrected with limited human review as feasible and efficient at billion-region scale.Source: §5.3 · pp. 14–15
6 Experiments
ASM is evaluated on image- and region-level captioning and zero-shot region recognition, alongside experiments on data scaling, cleaning, and human verification. Results report gains over comparison models and improved performance from data engineering.
6.1 Implementation Details
- Evaluation covers zero-shot region recognition, image- and region-level captioning, and training with AS-1B region labels plus LaionCOCO image captions.A second-stage fine-tuning setting adds image- and region-level data, including human-verified region annotations.Source: §6 · p. 15, §6.1 · p. 15, §6.2 · p. 15
6.2 Text Generation
- ASM exceeds Kosmos-2 by 1.4 points on zero-shot RefCOCOg region captioning, and ASM-FT sets a new record there.For image-level captioning, zero-shot Flickr30K CIDEr is 77.9 and rises to 87.7 after second-stage fine-tuning.Source: §6.2 · pp. 16–17, Table 6 · p. 18, Table 7 · p. 18
6.3 Zero-shot Region Recognition
- 10.4 mAP on COCO and 14.3 mAP on LVIS are ASM's zero-shot region-recognition improvements over CLIP.On LVIS's 1,203 categories, the reported improvement is 14.3 mAP over CLIP; ASM also performs captioning and region recognition with the same weights.Source: §6.3 · p. 18, Table 8 · p. 19
6.4 Data Engineering
- Fine-tuning with 40k human-verified semantic tags improves R-CLIP by +2.4 COCO mAP at 224 resolution and +3.3 at 896.The authors report that human verification can correct model bias and hard cases and supports later data-engine iteration.Source: Table 11 · p. 19, §6.4 · p. 20, Figure 8 · p. 20
- More semantic-tag training data, from 1M to 5M images, improves R-CLIP zero-shot region recognition.Data cleaning compresses 2.14 billion regions to 1.2 billion and improves mAP by 6.0% on COCO and 7.5% on LVIS.Source: §6.4 · p. 19, Table 10 · p. 19
6.5 Human Evaluation
- In a five-participant study across four datasets, users preferred ASM captions over MiniGPT4 and LLaVA, citing informativeness and factual accuracy.ASM was more frequently favored for image-level captions, while the compared VLLMs often produced region captions with over-associations, hallucinations, and factual errors.Source: §6.5 · p. 20, Table 12 · p. 21
7 Conclusion
The conclusion presents the All-Seeing Project as a data-and-model system for open-world panoptic visual recognition and understanding.
7 Conclusion
- ASM achieves state-of-the-art results on image- and region-level tasks, while an AS-1B-only region-aware CLIP model surpasses original CLIP in region recognition.The passage describes the margins as significant but gives no values.Source: §7 · p. 21
- A semi-automatic data engine with automatic annotation and human verification produces AS-1B’s more than 1 billion region-level annotations at controllable cost.The engine combines annotation automation with human verification.Source: §7 · p. 21
- ASM is a region-aware multimodal LLM whose unified decoder models region-text alignment and image-conditioned text generation.Source: §7 · p. 21
- The authors hope the data engine, AS-1B, and ASM will inspire further work toward AI systems with deeper understanding of the world.Source: §7 · p. 21
Credit Attribution of Equal Contribution Authors
The contribution statement assigns implementation, data-engine, annotation, model, evaluation, and manuscript work across the listed contributors.
Credit Attribution of Equal Contribution Authors
- Wang handled ASM implementation and experiments, annotation-pipeline development, LLM/VLLM module efficiency, R-CLIP code optimization, and related method and experiment writing.Source: Credit Attribution of Equal… · p. 21
- Shi managed data-engine construction and debugging, semantic-tag pipelines, cleaning and analysis, R-CLIP work, open-world semantic matching, human verification, and manuscript contributions.Source: Credit Attribution of Equal… · p. 21
- Li led open-world localization, contributed R-CLIP implementation and refinement experiments, established ASM human evaluation, and drafted portions of the manuscript.Source: Credit Attribution of Equal… · p. 21
- Wenhai Wang managed technical task decomposition and optimization, while Huang and Xing contributed human verification, location annotation, semantic-generation or review work, and manuscript drafting.Wenhai Wang also drafted and revised the manuscript; Huang guided manual annotation, and Xing optimized location annotation.Source: Credit Attribution of Equal… · p. 21
- Xizhou Zhu and Hao Li are acknowledged for the project’s preliminary idea and verification.Source: Credit Attribution of Equal… · p. 21