PaperlayerView exact revision on arXiv

Source-linked AI summary

General Object Foundation Model for Images and Videos at Scale

Junfeng Wu, Yi Jiang, Qihao Liu, Zehuan Yuan, Xiang Bai, Song Bai

arXiv:2312.09158v1cs.CV

TL;DR

Visual foundation models often lack unified object-level capabilities, despite object information supporting a broad range of vision tasks. GLEE combines image, text, and visual prompts in a framework trained on diverse data to address multiple object-centric tasks. The paper reports state-of-the-art results across numerous tasks and strong zero-shot transfer to new data and tasks.

  • Problem

    Visual foundation models often focus on image-level features or specific subdomains, while object localization and identification are foundational for many vision tasks.

    Source: §1 · p. 1

  • Method

    GLEE unifies image, text, and visual-prompt inputs to learn general object representations from diverse data and supervision.

    Source: §1 · pp. 1–2

  • Results

    GLEE reports state-of-the-art performance on numerous object-level tasks and strong zero-shot transfer to new data and tasks.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    GLEE can provide general visual object-level information for object-centric multimodal large language models.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

We present GLEE in this work, an object-level foundation model for locating and identifying objects in images and videos. Through a unified framework, GLEE accomplishes detection, segmentation, tracking, grounding, and identification of arbitrary objects in the open world scenario for various object perception tasks. Adopting a cohesive learning strategy, GLEE acquires knowledge from diverse data sources with varying supervision levels to formulate general object representations, excelling in zero-shot transfer to new data and tasks. Specifically, we employ an image encoder, text encoder, and visual prompter to handle multi-modal inputs, enabling to simultaneously solve various object-centric downstream tasks while maintaining state-of-the-art performance. Demonstrated through extensive training on over five million images from diverse benchmarks, GLEE exhibits remarkable versatility and improved generalization performance, efficiently tackling downstream tasks without the need for task-specific adaptation. By integrating large volumes of automatically labeled data, we further enhance its zero-shot generalization capabilities. Additionally, GLEE is capable of being integrated into Large Language Models, serving as a foundational model to provide universal object-level information for multi-modal tasks. We hope that the versatility and universality of our method will mark a significant step in the development of efficient visual foundation models for AGI systems. The model and code will be released at https://glee-vision.github.io .

1. Introduction

Visual foundation models often focus on image-level features or specific subdomains, while object localization and identification support many higher-level vision tasks. GLEE unifies object-centric perception across images and videos, learning from diverse supervision to transfer across tasks without task-specific adaptation.

1. Introduction

  • Visual foundation models often serve specific subdomains or learn image-level features, while object localization and identification underpin many complex vision tasks.The passage names segmentation, scene understanding, tracking, event detection, and activity recognition as tasks supported by object-level capabilities.Source: §1 · p. 1
  • GLEE combines an image encoder, text encoder, visual prompter, and detector to handle diverse object-centric tasks through a unified framework.The tasks include detection, instance segmentation, grounding, tracking, and interactive segmentation.Source: §1 · pp. 1–2
  • A unified input-output paradigm supports zero-shot generalization to new data and tasks and low-cost scaling with automatically labeled data.The introduction reports scaling the dataset to 10 million images with automatically annotated data, further enhancing generalization.Source: §1 · p. 2
  • Joint training uses over five million images from diverse benchmarks and varying levels of supervision, including boxes, masks, and object descriptions.Video data add temporal information, while open-world data contribute class-agnostic object annotations.Source: §1 · p. 2
  • GLEE reports state-of-the-art performance across object-level image tasks and superior zero-shot transfer on large-vocabulary open-world video tracking.The reported image tasks include detection, referring expression comprehension, and open-world detection without task-specific designs or fine-tuning.Source: §1 · p. 2

2. Related Work

Prior visual foundation models often lack either object-level semantics or broad task coverage, and some unified approaches remain limited in generalization. GLEE is positioned as a unified object-level model that aims to provide general object representations for downstream tasks.

2.1. Visual Foundation Model

  • Many visual foundation models learn image-level features, while SAM provides prompted segmentation without the semantic context needed for broader object-level tasks.GLEE aims to address this by directly solving downstream tasks without additional parameters or fine-tuning.Source: §2.1 · pp. 2–3

2.2. Unified and General Model

  • Unified models cover multiple tasks, but some focus on image-level understanding, generate boxes and masks slowly, or lack zero-shot generalization from closed-set training.GLEE is distinguished by addressing object-level tasks in a unified manner and providing universal object representations.Source: §2.2 · p. 3

2.3. Vision-Language Understanding

  • Language-based open-vocabulary detectors can be constrained by language-model capabilities and biases, motivating object representations that support both detection and identification.The stated objective is to use existing datasets to build a general object-level foundation model for downstream tasks.Source: §2.3 · pp. 3–4

3. Method

GLEE uses image, text, and visual-prompt inputs with an object decoder to predict objects and unify image and video perception tasks. Its joint training combines task-dependent losses with diverse data and supervision.

3.1. Formulation

  • GLEE encodes images, arbitrary textual descriptions, and visual prompts, then integrates them in an object decoder to extract objects.The decoder uses text-object embedding similarity for category alignment and predicts masks from mask embeddings and a pixel embedding map.Source: §3.1 · p. 4
  • Early fusion makes image features prompt-aware, while visual prompts can be points, boxes, or scribbles.The model encodes a prompt crop before the transformer and also samples prompt-related pixel embeddings for interaction with object queries.Source: §3.1 · pp. 4–5, §3.2 · p. 5

3.2. Task Unification

  • The unified framework supports detection, segmentation, grounding, tracking, and open-world or large-vocabulary image and video tasks.Category lists and object expressions are encoded as text prompts for detection, instance segmentation, and referring tasks.Source: §3.2 · p. 5
  • For video tracking, decoder object embeddings can be matched directly, but severe occlusion requires video training to strengthen temporal consistency.The approach contrasts embeddings across two video frames and tracks detections by bipartite matching.Source: §3.2 · p. 5

3.3. Training Unification

  • Joint end-to-end training selects different losses for different datasets, including semantic, box, mask, confidence, contrastive tracking, and distillation losses.The text encoder is trained toward the CLIP text-embedding space with an L1 loss.Source: §3.3 · pp. 5–6
  • GLEE scales training to 10 million data points by adding 2 million SA1B examples and 5 million GRIT examples.SA1B contributes mask annotations, while GRIT contributes referring-expression and bounding-box pairs.Source: §3.3 · p. 6

4. Experiments

Across image and video benchmarks, GLEE transfers to new tasks without task-specific fine-tuning, with strong zero-shot results and additional gains as training data scales. Its experiments also test classic video segmentation and integration into a language model.

4.1. Experimental Setup

  • Training proceeds from object detection pretraining to joint training with segmentation, video, referring-expression, and open-world datasets; adding SA1B and GRIT produces GLEE-scale.The second stage uses datasets with varying annotation granularity, and the scaled model shows stronger zero-shot performance on downstream tasks.Source: §4.1 · p. 6

4.2. Comparison with Generalist Models

  • 8.9 points: GLEE outperforms ODISE on open-world instance segmentation when identifying plausible instances class-agnostically.It also outperforms generalist models on COCO and LVIS detection and segmentation, remains competitive with specialist approaches, and achieves REC and RES results comparable to PolyFormer.Source: §4.2 · pp. 7–8

4.3. Zero-shot Evaluation Across Tasks

  • 36.0%: GLEE surpasses OVTrack in TAO, nearly triples the best baseline in BURST, and outperforms OV2Seg by 43.6% in LV-VIS zero-shot tracking.The evaluated model had not seen images from these datasets or trained on video-level data.Source: Table 2 · p. 7, §4.3 · p. 8
  • GLEE reaches state-of-the-art results across model sizes on YTVIS2019, but on occlusion-heavy OVIS it reaches state of the art only after a few hours of fine-tuning.The paper identifies lengthy videos and extensive object occlusion as settings where temporal capabilities are particularly important.Source: §4.3 · p. 8, Table 4 · p. 8
  • GLEE exceeds language-based detectors on OmniLabel P-categ but trails on AP-descr; adding GRIT data raises AP-descr to a comparable level, and GLEE beats GLIP on ODinW average mAP.ODinW results cover 13 datasets in a zero-shot setting.Source: §4.3 · pp. 8–9, Table 3 · p. 8, Table 5 · p. 9

4.5. Ablation

  • Larger GLEE-Pro training sets improve zero-shot performance across TAO, BURST, OVIS, and YTVIS.The tested data fractions were 10%, 20%, 50%, and 100%; the unified training approach accepts manually or automatically annotated data.Source: Figure 5 · p. 9, §4.5 · p. 9

4.4. Serve as Foundation Model

  • Replacing SAM with frozen pretrained GLEE-Plus in LISA yields comparable results after the same number of training steps.The modified model feeds GLEE object queries into LLAVA and uses SEG tokens with GLEE features to generate masks.Source: §4.4 · p. 9, Figure 4 · p. 9

5. Conclusion

GLEE is an object-level foundation model for a broad range of image and video tasks. The paper reports state-of-the-art performance and zero-shot transfer, and presents GLEE as a source of object-level information for object-centric multimodal LLMs.

5. Conclusion

  • GLEE is designed for a wide range of object-level image and video tasks.The paper describes it as directly applicable to these tasks.Source: §5 · p. 9
  • GLEE achieves state-of-the-art performance on numerous object-level tasks and excels at zero-shot transfer to new data and tasks.The paper attributes this to its versatility and generalization abilities.Source: §5 · p. 9
  • GLEE provides general visual object-level information, which the paper says is currently missing in modern LLMs, as a foundation for object-centric mLLMs.Source: §5 · p. 9

6. Datasets and Implementation Details

GLEE is trained jointly on image and video data with varied annotation granularity, supplemented by automatically annotated data. The implementation uses a multistage training schedule and an object decoder.

Data Preparation

  • Training data span image and video datasets with bounding-box, mask, description, temporal, and class-agnostic object annotations.Table 6 lists the tasks and datasets used in training.Source: §6 · p. 16, Table 6 · p. 16
  • The scale-up training uses 2,000,000 SA1B images and 5,000,000 GRIT samples, with mask-IoU-based NMS and area scoring applied to SA1B annotations to remove part-level objects.Source: §6 · p. 16

Model and Training Details

  • The model uses a 6-layer deformable transformer encoder, a 9-layer decoder, 300 object queries, query denoising, and hybrid matching.Training proceeds through stages, including a 500,000-iteration stage 1 and a 500,000-iteration stage 2.Source: §6 · p. 16
  • Scale-up training adds 500,000 iterations; optimization uses AdamW with a 1 × 10−4 base learning rate and 0.05 weight decay.Learning rates for the image backbone and text encoder are multiplied by 0.1.Source: §6 · p. 16

7. Transfer to Video Tasks

GLEE is evaluated on semi-supervised video object segmentation and referring video object segmentation. It is reported to outperform prior referring-video approaches and to compare favorably with non-memory-based methods on YouTube-VOS.

VOS

  • For semi-supervised VOS, GLEE receives the target object's first-frame mask and is evaluated on YouTube-VOS 2018 and MOSE.The reported fine-tuning jointly uses YouTube-VOS, YTVIS2019, YTVIS2021, OVIS, and UVO-video for 40,000 iterations.Source: §7 · pp. 16–17
  • GLEE achieves the best results among non-memory-based methods on YouTube-VOS and competitive results against memory-based methods on the more challenging MOSE dataset.Table 8 presents the video object segmentation comparison.Source: §7 · p. 17, Table 8 · p. 17

RVOS

  • For referring video segmentation, GLEE selects the object query with the highest confidence for a language expression and adds query similarity to the previous frame's selected query to the current confidence score.The stage-2-trained GLEE is evaluated directly on Ref-YouTube-VOS.Source: §7 · p. 17
  • GLEE outperforms all previous referring-video object segmentation approaches and unified methods on Ref-YouTube-VOS.Table 9 presents the performance comparison for this task.Source: §7 · p. 17, Table 9 · p. 17

8. Object Detection in the Wild

The section evaluates GLEE's transfer to ODinW detection through few-shot and full-data fine-tuning. The supplied passages describe the evaluation protocol and comparison table, but do not give performance outcomes.

Few-shot transfer evaluation

  • ODinW transfer is tested from 1-shot through 10-shot settings and with all available training data, using full-model tuning.Models use the same hyperparameters, and each setting is trained three times with different random seeds.Source: §8 · p. 17

Evaluation protocol

  • The few-shot evaluation averages test scores and standard deviations across three random-seed runs, with the optimal model selected on the validation split.Source: §8 · p. 17

Comparison scope

  • Table 10 compares per-dataset performance with DyHead, GLIP-T, and GLIP-L; PascalVOC uses mAP (IoU=0.50:0.95).The caption specifies that “Full” denotes full-model tuning.Source: Table 10 · p. 18

9. Interactive Segmentation and Tracking

GLEE uses visual prompts to segment specified objects interactively in images and videos. For video segmentation, object features and masks from an earlier frame guide segmentation in subsequent frames.

9. Interactive Segmentation and Tracking

  • Points, boxes, or scribbles paired with an image prompt GLEE to segment the specified object.The figure shows these visual prompts used for image-level interactive segmentation.Source: §9 · p. 18, Figure 6 · p. 19
  • For video segmentation, the previous frame’s mask and prompt feature enter early fusion and self-attention to guide segmentation in the current frame.The previous frame’s object features serve as referring features; the figure describes using the first-frame masked feature to guide subsequent frames.Source: §9 · p. 18, Figure 6 · p. 19
  • Figure 6 illustrates interactive segmentation with different prompts on images and videos.It shows image prompts and video segmentation guided by the first frame’s masked feature.Source: §9 · p. 18, Figure 6 · p. 19

Related papers

Loading 2312.09158v1…