PaperlayerView exact revision on arXiv

Source-linked AI summary

Perception Encoder: The best visual embeddings are not at the output of the network

Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, Junke Wang, Marco Monteiro, Hu Xu, Shiyu Dong, Nikhila Ravi, Daniel Li, Piotr Dollár, Christoph Feichtenhofer

arXiv:2504.13181v2cs.CV

TL;DR

Existing vision encoders typically use task-specific pretraining objectives, while no single simple and scalable technique had been shown to provide state-of-the-art features across downstream tasks. PE uses robust contrastive image and video pretraining, then aligns strong intermediate features for language and spatial applications. The resulting PE family achieves best-in-class performance across recognition, multimodal language, and spatial tasks, while its useful features diminish toward the network output for many tasks.

  • Problem

    No single, simple, and easily scalable pretraining technique had been shown to learn state-of-the-art features for all downstream tasks.

  • Method

    PE combines robust contrastive image pretraining with video finetuning, then uses language and spatial alignment to extract useful intermediate features.

  • Results

    The PE family achieves best-in-class results across zero-shot recognition, multimodal language, and spatial tasks.

  • Takeaways & Limitations

    Contrastive vision-language pretraining can scale across a wide range of downstream vision tasks when intermediate features are aligned to the network end.

  • Takeaways & Limitations

    Strong performance often diminishes rapidly toward the network output, with final-layer performance abysmal for certain tasks such as LLM-based grounding.

Abstract

from arXiv · show

We introduce Perception Encoder (PE), a state-of-the-art vision encoder for image and video understanding trained via simple vision-language learning. Traditionally, vision encoders have relied on a variety of pretraining objectives, each tailored to specific downstream tasks such as classification, captioning, or localization. Surprisingly, after scaling our carefully tuned image pretraining recipe and refining with our robust video data engine, we find that contrastive vision-language training alone can produce strong, general embeddings for all of these downstream tasks. There is only one caveat: these embeddings are hidden within the intermediate layers of the network. To draw them out, we introduce two alignment methods: language alignment for multimodal language modeling, and spatial alignment for dense prediction. Together, our PE family of models achieves best-in-class results on a wide variety of tasks, including (1) zero-shot image and video classification and retrieval, simultaneously obtaining 86.6 average zero-shot ImageNet robustness and 76.9 zero-shot Kinetics-400 video classification; (2) document, image, and video Q&A, enabling 94.6 DocVQA, 80.9 InfographicVQA, and 82.7 PerceptionTest with an 8B LLM; and (3) spatial tasks such as detection, tracking, and depth estimation, setting a new COCO state-of-the-art of 66.0 box mAP. To foster further research, we release our models, code, and novel dataset of synthetically and human-annotated videos: https://github.com/facebookresearch/perception_models

1 Introduction

Perception Encoder shows that simple contrastive vision-language pretraining can produce general features across vision tasks, although those features emerge at different intermediate layers and require alignment tuning for downstream use. The resulting PE family combines robust image and video training with language and spatial alignment to support recognition, multimodal language, and spatial tasks.

  • A single, simple, scalable pretraining technique had not previously been shown to learn state-of-the-art features across all downstream tasks.
  • Global vision-language contrastive learning alone produced internal features aligned to OCR, VQA, grounding, detection, depth estimation, and tracking.
  • Specific useful features occur at different encoder layers, so alignment tuning transfers them to the network output for multimodal language and spatial tasks.The paper introduces language alignment for PElang and spatial alignment for PEspatial.
  • PEcore uses large-scale contrastive image and video pretraining to achieve strong zero-shot classification and retrieval performance.The recipe develops robust image pretraining, then uses a frame-based encoder and synthetically aligned video captions for video finetuning.
  • PEcoreG scales the robust image and video strategy to 2B parameters, outperforming SigLIP2 on most zero-shot image tasks and InternVideo2 on most zero-shot video tasks.
  • The PE family extends one contrastive pretraining approach across downstream vision tasks and is released with models, code, and the PE Video Dataset.

2 Perception Encoder: Core

PE combines robust image contrastive pretraining with aligned video finetuning to produce a unified encoder whose intermediate features support broad visual tasks. The resulting models achieve strong zero-shot image and video performance, generalize across difficult benchmarks, and are released with the PE Video Dataset.

  • Robust image pretraining: Contrastive learning benefits from long schedules because each batch can provide new information through changing sample relationships.The recipe emphasizes regularization, stability, and training efficiency to exploit this property.
  • Robust image pretraining: Training longer with the revised recipe yields nearly linear gains across most datasets, indicating further performance remains available at L scale.The scaling study covers 120K, 240K, and 360K steps and reports 24B samples already seen.
  • Video data engine: +5.6% to +11.7% video classification accuracy and +7.7 to +15.3 video-retrieval top-1 recall arise as synthetic video data scales, without plateauing.Image benchmarks and retrieval also improve, showing gains beyond video-only evaluation.
  • Video data engine: PE Video Dataset contains 1M diverse motion-centered videos, with 120K human-refined annotations offered as verified or detailed automated captions.The released dataset includes first-person and third-person views across a wide range of scenes.
  • Core results: PEcoreG outperforms existing contrastive models across zero-shot image tasks and exceeds or matches specialized encoders on common frozen-feature evaluations.Reported gains include +9.6% over the next-best model on iNaturalist and state-of-the-art performance across common zero-shot categories.

3 General Features in a Contrastive Disguise

PEcore’s contrastive pretraining produces general features across language and spatial tasks, but these features peak in intermediate layers rather than at the network output. The robust recipe scales to larger models, motivating alignment methods that expose the hidden embeddings.

  • Layerwise Feature Analysis: Layerwise frozen-feature analysis sweeps model layers across classification, language modeling, and spatial tasks using task-specific decoders.Performance is normalized across models, and the optimal layer is not assumed to be the final one.
  • Layerwise Feature Analysis: PEcore’s intermediate layers perform near or on par with AIMv2 on language tasks and DINOv2 on spatial tasks despite contrastive training.This includes depth estimation, where contrastive encoders are not typically considered state of the art.
  • An Alignment Problem: PEcore’s strong intermediate-layer performance often diminishes rapidly toward the network’s end, with especially poor final-layer grounding performance.The pattern is less pronounced when downstream tasks are closer to the pretraining objective.
  • An Alignment Problem: The authors interpret this as an alignment problem: contrastive training learns general embeddings but does not naturally output them, requiring subsequent downstream alignment.The hidden features occur at different layers for different tasks.
  • The Downstream Effects of Robust Pretraining: Robust pretraining improves the best COCO detection layer by almost 10 mAP over vanilla CLIP, while some recipe changes shift the optimal layer.Progressive resolution and attention pooling move the best-layer index; batch-size and optimizer changes do not improve detection performance.
  • Scaling Behavior: The robust recipe’s best-layer performance scales from smaller models to G scale at 2B parameters, whereas vanilla CLIP plateaus at L scale.Final-layer performance still stagnates for both recipes, leaving the general features several layers deep.
  • An Alignment Problem: Alignment methods are therefore used to move the general intermediate features to the network output for language modeling and spatial tasks.This preserves the simple contrastive pretraining approach while making the features usable by downstream encoders.

4 Perception Encoder: Language Alignment

PElang lifts strong intermediate PEcoreG features to the network output through language alignment, producing a general encoder for diverse MLLM tasks. The method adapts PEcoreG to an LLM and selects a regularized configuration that improves average downstream performance.

  • PElang is designed as a single language-aligned encoder spanning language models, input resolutions, and diverse MLLM tasks.
  • PElang evaluation covers OCR, chart and document Q&A, visual Q&A, captioning, video understanding, and grounding.
  • The alignment adapts PEcore to a pretrained decoder-only Llama 3 using a vision projector and autoregressive next-token prediction.Training begins with a frozen-parameter projector warmup on 1M image-text samples, followed by full finetuning on 70M multimodal samples.
  • Layer 47 works best for language alignment, matching the optimal frozen VQ&A layer, while LayerScale and DropPath stabilize training.
  • 82.2 average score is achieved by the final PElang after scaling training to 70M samples, adding 2.1 points over the 20M-sample ablation setting.
  • Language alignment moves the performing layer to the last layer across categories, including improved grounding despite no grounding data in the training mix.

OCR Q&A Visual Q&A

PElang is benchmarked against vision encoders spanning different pretraining losses, sizes, and resolutions using multiple language-modeling setups. It performs strongly across MLLM benchmarks, transfers to QwenLM, and supports competitive system-level comparisons.

  • Figure 13 analyzes how language alignment changes PE’s internal features by adapting each layer to a new LLM.
  • PElang is compared with MetaCLIP, SigLIP2, CLIP, AIMv2, DINOv2, and InternViT2.5 across varied pretraining losses, encoder sizes, and resolutions.
  • The MLLM benchmark covers OCR, chart and document Q&A, visual Q&A, captioning, video understanding, and grounding at native encoder resolution.
  • PElang shows strong performance across all reported benchmarks, including comparisons against models three times its size.
  • Language alignment transfers well from Llama 3.2 3B to QwenLM 2.5 7B, where PElang outperforms the compared vision encoders.
  • The PLM-8B system uses PElangG with 36 image tiles and 32 video frames in comparison with open-access MLLMs.

5 Perception Encoder: Spatial Alignment

Spatial alignment addresses PEcoreG’s layer-dependent spatial representations by preserving semantic features while restoring local correspondence. Self-distillation and SAM 2.1 mask-logit distillation together produce PEspatial models that perform strongly on dense prediction and detection.

  • Spatial Alignment Motivation: PEcoreG’s optimal spatial layer differs by task: higher-level detection and depth favor around layer 40, while tracking favors around layer 30.
  • Feature Analysis: Tracking peaks at layer 32 because attention remains local through that layer before global tokens appear abruptly at layer 33.
  • Alignment Objectives: Spatial alignment must preserve semantic information near layer 40 while emphasizing locality for tasks with shallow decoders.
  • Semantic Retention: The method self-distills frozen layer 41 features with cosine-similarity training, using DropPath and LayerScale regularization while keeping the teacher fixed.
  • Locality Alignment: SAM 2.1 is queried on a 32 × 32 grid, and its 1024 mask logits are concatenated into an H × W × 1024 feature map for locality alignment.
  • Locality Alignment: Pairwise token cosine similarities are aligned with MSE loss; mask-logit features are interpolated to train at PEcore’s 448px resolution.
  • Alignment Effects: Combining self-distillation and SAM 2.1 mask-logit alignment yields PEspatialG, whose effects are evaluated against the original PEcoreG and each alignment separately.
  • Alignment Effects: Layer 41 alignment helps detection, depth, and segmentation, whereas SAM mask logits primarily improve zero-shot tracking by providing precise locality.

6 Related Work

Related work situates PE within vision-language contrastive learning, efficient CLIP training, intermediate-layer feature extraction, and alignment tuning. The paper distinguishes its goal of building a general vision encoder from work focused primarily on constructing MLLM systems.

  • Vision-semantic representations support zero-shot classification, image-text retrieval, open-vocabulary detection and segmentation, and MLLM development.
  • Contrastive vision-language pretraining evolved from early multimodal objectives into large-scale CLIP and ALIGN models.
  • Efficient CLIP training research has explored large batch sizes, LAMB optimization, and rotary positional embeddings.
  • Prior work shows intermediate layers can outperform the last layer for downstream tasks when encoders are trained on proxy or self-supervised objectives.
  • PE studies alignment tuning for language through MLLM adaptation and for spatial understanding through self-distillation plus a locality teacher.
  • Unlike typical MLLM midtraining, PElang prioritizes making the vision encoder general across language models, resolutions, and tasks.
  • PEspatial combines intermediate-feature alignment with semantic and spatial distillation ideas related to REPA and RADIO.

7 Conclusion

Perception Encoder is presented as a family of foundation models spanning image, video, language, and spatial perception tasks. The authors conclude that simple contrastive vision-language pretraining can address a broad range of downstream vision tasks.

  • Perception Encoder comprises PEcore, PElang, and PEspatial.
  • PEcore outperforms WebLI- and JFT-3B-trained models in zero-shot image recognition while also excelling in zero-shot video recognition.
  • PElang enables a multimodal language model that is at the forefront of the field in performance.
  • PEspatial matches the long-standing state of the art in object detection with a significantly simpler decoder.
  • The authors conclude that Perception Encoder scales simple contrastive vision-language pretraining across diverse downstream vision tasks.

A Video Data Engine

The PE Video dataset contains one million curated videos, including human-refined captions and a held-out benchmark. Its captions are produced through an LLM-based summarization pipeline that combines metadata, video captions, and frame captions.

  • The summarization prompt combines metadata, video captions, and frame captions into concise alt-text-style captions.
  • 1M videos form the PE Video dataset, with 120K human-refined captions and 15K videos selected as a benchmark.
  • The dataset is collected and curated from a licensed data source.

A.2.1 Video Data Filtering Pipeline

The video filtering pipeline selects motion-rich, high-quality, and diverse videos while rejecting static or artificial content. It combines motion, visual, language-model, quality, and clustering signals.

  • The pipeline targets object motion, camera motion, object interactions, human actions, action sequences, and object manipulation, while rejecting static, artificial, or highly edited videos.
  • Motion features include frames per second, frame counts, I-frame counts, motion-vector magnitude, and motion-vector variance.
  • Three uniformly sampled frames are encoded with DINOv2 and SigLIP to extract visual features.
  • An LLM answers 26 questions about scene motion, objects, actions, and related video characteristics to produce language-model features.
  • A random forest predicts a 0–5 video-quality score, where higher scores indicate multiple temporal events requiring several frames for accurate captioning.
  • K-means clustering and within-cluster ranking reduce duplicated videos in the final dataset.

B.1.1 Architecture and Training Setups

PEcore uses Transformer-based vision and text encoders with attention pooling and positional embeddings, then follows staged image, video, and distillation training. Evaluation uses standardized zero-shot protocols and frozen intermediate-layer features for multimodal adaptation.

  • Architecture and Training Setups: PEcore comprises Transformer-based vision and text encoders, using attention pooling to construct image and video embeddings.
  • Architecture and Training Setups: PEcore training has three stages: image pretraining, image and video finetuning, and model distillation for smaller models.
  • Architecture and Training Setups: The largest G-scale model is followed by image pretraining, image distillation, and video finetuning for smaller models.
  • Architecture and Training Setups: Zero-shot evaluation covers image and video classification and retrieval using CLIPBench, extended with MSR-VTT and Kinetics datasets.
  • Architecture and Training Setups: Evaluation uses center-crop and squash transformations, reporting the maximum result across both for every model and dataset.
  • Architecture and Training Setups: For multimodal adaptation, baselines are searched across intermediate layers and compared by average performance across language, captioning, and video-understanding tasks.

B.3.1 Training Details

Spatial alignment combines self-alignment to an intermediate PEcoreG layer with locality alignment from SAM mask logits, using frozen teachers and their summed loss.

  • Loss Functions: The core alignment loss matches student last-layer features to frozen PEcoreG layer 41 features using cosine similarity.Layer 41 was chosen heuristically; nearby layers around 40 also work.
  • Loss Functions: The locality loss matches student and teacher pairwise token-similarity maps with mean squared error.The teacher map is formed from SAM mask logits, with an optional temperature applied to its cosine-similarity term.
  • Loss Functions: The full spatial-alignment objective is the sum of the core and locality losses.Lspatial = Lcore + Lloc.
  • Training Setup: Both teachers process the same augmented image as the student and remain frozen during training.SAM 2.1 outputs are generated at 1024 resolution, then upsampled or downsampled as needed.
  • Training Setup: The spatial-alignment training settings are documented separately for the student, PEcoreG teacher, and SAM 2.1 teacher.These settings are summarized in Tables 21–23.

B.3.2 Visualization Method

The visualization method filters feature representations before PCA so spatially contiguous information is visible, while the section also records evaluation and training caveats for dense prediction and robustness experiments.

  • Visualization Method: PCA reduces each 1536-dimensional token feature to three dimensions, but naive PCA can be misleading when global-token information dominates.The resulting dimensions are mapped through LCh color space into RGB images.
  • Visualization Method: A low-pass filter reveals spatial information that raw-feature PCA misses, including visually coherent background tiles.Raw features emphasize global tokens, whereas filtered features expose spatially contiguous parts of the feature space.
  • Dense Prediction Evaluation: Frozen tracking evaluation propagates masks by matching patches from the last seven frames to the current frame on DAVIS.This protocol requires neither finetuning nor probing.
  • Dense Prediction Evaluation: Semantic segmentation uses linear and convolutional probes on single-layer intermediate features resized to 518 × 518.The probing layers are finetuned while the pretrained features are evaluated as described.
  • Dense Prediction Evaluation: Depth estimation uses a DPT head on frozen single-layer features, with 20 training epochs to reduce comparison differences from overfitting.The authors report noisy, overfitting-prone results on the small NYUv2 dataset with large models.
  • Dense Prediction Evaluation: End-to-end COCO experiments tune training settings because the default 100-epoch ViTDet protocol can overfit billion-parameter encoders.The adjustments include epochs, learning rate, drop path, and learning-rate decay.
  • Robustness Results: Robustness ablations improved nearly all metrics, with progressive resolution slightly hurting the average and mask regularization slightly hurting ImageNet Adversarial.The raw results are provided in Table 26.
  • Robustness Results: Several tested changes did not improve performance, including average pooling, a larger text tower, hue or contrast jitter, and fixed-resolution token dropping.The discussion reports only changes known to work in the main recipe.

C.2 PEcore: Video Data Scaling

Scaling synthetic video data generated by the video data engine improves image and video classification and retrieval, while distillation and video finetuning strengthen smaller models.

  • Video Data Scaling: Increasing synthetic video data consistently improves image and video classification and retrieval performance.The detailed results are reported in Table 27.
  • Video Data Scaling: Image benchmark gains plateau earlier on ImageNet val and v2 than on ObjectNet, ImageNet Adversarial, and MS-COCO retrieval.MS-COCO retrieval continues gaining as synthetic video data increases.
  • Distillation: Teacher-distribution temperature is an ablation for distillation using a pretrained vanilla CLIP ViT-B/14 baseline and a 50K-step schedule.The student matches teacher image-to-text and text-to-image similarity distributions with KL-divergence loss.
  • Distillation: A smaller fixed teacher temperature sharpens the teacher distribution and significantly improves distillation effectiveness.The student temperature remains learnable.
  • Smaller Models: For PEcoreL, image distillation followed by video finetuning and distillation improves both image and video benchmarks.The step-by-step strategy is summarized in Table 29.

C.4 PElang: Additional Results

PElang maintains strong performance with dynamic tiling and achieves broad zero-shot grounding results across RefCOCO splits, with comparisons shaped by language-model pretraining exposure.

  • Dynamic Resolution: The dynamic-tiling evaluation covers eight tile ratios formed from one thumbnail and up to four 448 × 448 image tiles.The listed ratios include combinations from 1 × 1 through 4 × 1.
  • Dynamic Resolution: PElang largely outperforms baseline vision encoders across all categories of MLLM tasks with dynamic tiling of up to four tiles plus a thumbnail.The alignment tuning itself used native-resolution inputs.
  • Zero-Shot Grounding: PElang L or G achieves the best performance across RefCOCO, RefCOCO+, and RefCOCOg splits with Llama 3.1-instruct 8B, except with Qwen2.5 7B.The RefCOCO datasets are unseen in the stated SFT data.
  • Zero-Shot Grounding: The Qwen2.5 exception is attributed to InternViT 2.5 being midtrained with Qwen2 and having seen RefCOCO-family data during pretraining or midtraining.This comparison is reported as a zero-shot evaluation of spatial understanding.
  • Qualitative Visualization: Figure 20 compares PEcoreG last-layer features with features aligned to layer 41, SAM 2.1 mask logits, or both as PEspatialG.The four feature-space visualizations follow the source image.
Loading 2504.13181v2…