Source-linked AI summary
PerceptionGPT: Effectively Fusing Visual Perception into LLM
Renjie Pi, Lewei Yao, Jiahui Gao, Jipeng Zhang, Tong Zhang
TL;DR
Visual large language models still face challenges with complex perception tasks, while discrete visual tokens can make training difficult and outputs lengthy. PerceptionGPT represents perception signals with LLM token embeddings and lightweight visual encoders and decoders. It reports SOTA results on REC and RES with 300M tunable parameters, around 4% of the previous P-VLM approach.
Problem
Complex perception tasks such as detection and segmentation remain challenging for VLLMs, and discrete-token representations can make training difficult, introduce quantization error, and lengthen outputs.
Source: §1 · p. 1Method
PerceptionGPT uses the LLM token embedding, marked by a special <vis> token, to represent visual signals with lightweight encoders and decoders.
Source: §1 · p. 2Results
SOTA on REC and RES uses 300M tunable parameters, around 4% of the previous P-VLM approach; mask decoding takes 0.3% and 3.7% of Numerical and Vocabulary formulation inference time, respectively.
Source: §1 · p. 2, §4.4 · p. 8Takeaways & Limitations
The authors describe depth and pose estimation as perception tasks that can be easily integrated into the framework.
Source: §1 · p. 2Two-stage approaches that require external vision experts are less applicable to tasks without such expert models.
Source: §2.2 · p. 3
Abstract
from arXiv · showhide
The integration of visual inputs with large language models (LLMs) has led to remarkable advancements in multi-modal capabilities, giving rise to visual large language models (VLLMs). However, effectively harnessing VLLMs for intricate visual perception tasks remains a challenge. In this paper, we present a novel end-to-end framework named PerceptionGPT, which efficiently and effectively equips the VLLMs with visual perception abilities by leveraging the representation power of LLMs' token embedding. Our proposed method treats the token embedding of the LLM as the carrier of spatial information, then leverage lightweight visual task encoders and decoders to perform visual perception tasks (e.g., detection, segmentation). Our approach significantly alleviates the training difficulty suffered by previous approaches that formulate the visual outputs as discrete tokens, and enables achieving superior performance with fewer trainable parameters, less training data and shorted training time. Moreover, as only one token embedding is required to decode the visual outputs, the resulting sequence length during inference is significantly reduced. Consequently, our approach enables accurate and flexible representations, seamless integration of visual perception tasks, and efficient handling of a multiple of visual outputs. We validate the effectiveness and efficiency of our approach through extensive experiments. The results demonstrate significant improvements over previous methods with much fewer trainable parameters and GPU hours, which facilitates future research in enabling LLMs with visual perception abilities.
1. Introduction
Although visual large language models have advanced multimodal capabilities, complex perception tasks such as detection and segmentation remain challenging. PerceptionGPT uses LLM token embeddings to represent visual outputs, reporting strong REC and RES performance with a small trainable model.
1. Introduction
- VLLMs combine visual inputs with LLMs, but complex perception tasks such as object detection and segmentation remain challenging.Two-stage methods can rely on vision experts, while end-to-end P-VLMs integrate perception capabilities into the LLM.Source: §1 · p. 1
- Discrete visual tokens make continuous signals difficult to train, introduce quantization error, and lengthen sequences for complex outputs.The paper cites the training cost of prior approaches and the difficulty of handling long sequences for masks.Source: §1 · pp. 1–2
- PerceptionGPT uses the LLM token embedding as a carrier for visual signals, with a special <vis> token and lightweight encoders and decoders.The embedding can encode context-dependent outputs such as segmentation masks, which a lightweight decoder reconstructs.Source: §1 · p. 2
- PerceptionGPT combines language-modeling loss with task-specific visual objectives, while one <vis> token reduces the sequence needed for each perception signal.The paper says the approach predicts exact values to address quantization errors and can be trained by tuning a small fraction of parameters.Source: §1 · p. 2
- SOTA on REC and RES uses 300M tunable parameters, around 4% of the previous P-VLM approach.The paper also reports extensive evaluation on referring expression comprehension, referring expression segmentation, image captioning, and visual question answering.Source: §1 · p. 2
2. Related works
Related work covers the growth of vision-language models, systems that delegate perception to external experts, and end-to-end models that represent perception with discrete tokens. PerceptionGPT instead uses a continuous embedding representation for visual outputs.
2.1. Vision Large Language Models
- Vision-language models build on multimodal interaction, and PerceptionGPT supports image-level and region-level understanding, localization, and segmentation.The framework illustration describes lightweight perception encoders and decoders trained from scratch, without pretrained visual experts.Source: §2.1 · p. 2, Table 1 · p. 3, Figure 2 · p. 3
2.2. Two-stage Vision Language Assistant
- Two-stage assistants use LLMs to plan or call vision experts, but their applicability is limited when a task lacks a suitable expert model.Examples include ViperGPT calling visual functions and LISA using SAM for segmentation.Source: §2.2 · p. 3
2.3. Perception-enhanced Vision Language Model
- End-to-end P-VLMs encode perception with discrete tokens, which can make training difficult, introduce quantization error, and lengthen contexts for many boxes or masks.PerceptionGPT instead represents each box or mask with a continuous embedding and uses lightweight perception encoders and decoders.Source: §2.3 · pp. 3–5, Figure 2 · p. 3
3. Method
PerceptionGPT equips a P-VLM with visual perception by using the LLM token embedding as the carrier for perception signals. Lightweight task encoders and decoders connect those signals to the LLM, alongside language and task-specific training objectives.
Model Composition
- PerceptionGPT combines an image-text-aligned ViT, an LLM, and lightweight perception encoders and decoders, feeding image tokens and text embeddings to the LLM.The LLM autoregressively predicts the next token.Source: §3 · p. 5
Empowering Visual Perception Capability
- A special <vis> token marks a perception signal; its embedding is passed to decoders to restore the signal, while encoders map input signals into LLM embedding space.The described decoders include two-way transformer blocks for masks and three-layer MLPs for boxes.Source: §3 · p. 5
Training Objective
- Training combines autoregressive language loss with task-specific visual losses; language loss supervises responses and <vis> timing, while box and mask losses supervise their respective predictions.The described box loss combines L1 and GIoU, and the mask loss combines BCE and Dice; task-specific objectives also make training easier and more efficient.Source: §3 · p. 5
REC evaluation
- On REC, PerceptionGPT reports outstanding performance against generalist and specialist models using only parameter-efficient LoRA training.The supplied table caption does not provide individual scores.Source: Table 2 · p. 6
Multi-layer Visual Feature Fusion
- Adaptive weights fuse features from all ViT layers, addressing the reliance on only the final layer in previous VLM approaches.The learned weights are applied to each layer's image features to form the input image feature for the LLM.Source: §3 · pp. 5–6
4. Experiments
Across detection, segmentation, and conventional vision-language tasks, PerceptionGPT achieves strong results, while ablations examine training difficulty, feature fusion, and decoding speed. The experiments also report qualitative capabilities and the training and evaluation setup.
4.1. Training and Evaluation
- Training uses RefCOCO, RefCOCO+, RefCOCOg, Visual Genome, and Flickr30k; SAM generates masks where Visual Genome and Flickr30k lack segmentation annotations.Captioning also uses COCO and caption data curated by LLaVA; experiments use Vicuna with LoRA rank 32 and run for 70 hours on eight A40 GPUs.Source: §4.1 · p. 6
4.2. Qualitative Results
- Generated examples show spotting captioning with boxes and masks, reasoning-based detection and segmentation, and image- and region-level captioning.The authors report reasoning-based detection and segmentation even though that ability was not specifically considered in the training data.Source: §4.2 · p. 6
4.3. Quantitive Results
- PerceptionGPT achieves SOTA on REC and RES, and is comparable to specialized and generalist models on conventional VQA and captioning tasks.The RES comparison reports gains over end-to-end methods on all dataset splits and over LISA on most splits.Source: §4.3 · pp. 6–7, Table 3 · p. 7, Table 5 · p. 8
4.4. Ablation Study
- On bounding-box Acc@0.5, PerceptionGPT performs well with LoRA rank 8, while numerical representations do not support parameter-efficient LoRA training and discrete representations decline without Visual Genome.The authors suggest continuous embeddings avoid modeling dependencies between discrete coordinate tokens and support vision-specific losses.Source: §4.4 · p. 7, Table 6 · p. 8
- Top ViT layers help captioning, bottom layers help fine-grained perception, and adaptive fusion adjusts layer weights and boosts performance across tasks.The learned weights across ViT layers are shown in Figure 4.Source: §4.4 · pp. 7–8, Figure 4 · p. 8
- For 7B P-VLM mask decoding, PerceptionGPT takes 0.3% of Numerical and 3.7% of Vocabulary inference time.The comparison represents a mask with 32 points to ensure acceptable mask quality.Source: §4.4 · p. 8, Table 7 · p. 8
5. Conclusion
PerceptionGPT trains perception-enhanced vision-language models using LLM token embeddings. It achieves promising results with few trainable parameters, compact representations, and faster decoding.
5. Conclusion
- PerceptionGPT uses the representational power of LLM token embeddings to train perception-enhanced vision-language models efficiently, with few tuned parameters, compact representations, and accelerated decoding.The authors present the framework as offering new insights for future research on P-VLMs.Source: §5 · p. 8