PaperlayerView exact revision on arXiv

Source-linked AI summary

PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding

Jang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi, Triantafyllos Afouras, Tushar Nagarajan, Muhammad Maaz, Yale Song, Tengyu Ma, Shuming Hu, Suyog Jain, Miguel Martin, Huiyu Wang, Hanoona Rasheed, Peize Sun, Po-Yao Huang, Daniel Bolya, Nikhila Ravi, Shashank Jain, Tammy Stark, Shane Moon, Babak Damavandi, Vivian Lee, Andrew Westbury, Salman Khan, Philipp Krähenbühl, Piotr Dollár, Lorenzo Torresani, Kristen Grauman, Christoph Feichtenhofer

arXiv:2504.13180v3cs.CVcs.AIcs.LG

TL;DR

Closed models and proprietary distillation obscure the evidence needed to understand VLM training and measure scientific progress. This paper builds an open, reproducible PLM, studies synthetic-data scaling, and adds human-labeled video data and PLM-VideoBench; it achieves comparable performance with open-weight state-of-the-art models across 40 image and video benchmarks without proprietary distillation.

  • Problem

    Proprietary VLM designs, training methods, and data obscure how to measure scientific progress and understand training VLMs from scratch.

  • Method

    The paper develops an open, reproducible PLM, analyzes non-distilled training and synthetic-data scaling, and releases human-labeled video data with PLM-VideoBench.

  • Results

    Across 40 image and video benchmarks, PLM achieves comparable performance with existing state-of-the-art open-weight models without proprietary distillation.

  • Takeaways & Limitations

    The released data, benchmark, models, code, and training recipes provide a foundation for reproducible and transparent visual-perception research.

  • Takeaways & Limitations

    PLM is less competitive on long-video benchmarks because it does not extensively experiment with long-video modeling components.

Abstract

from arXiv · show

Vision-language models are integral to computer vision research, yet many high-performing models remain closed-source, obscuring their data, design and training recipe. The research community has responded by using distillation from black-box models to label training data, achieving strong benchmark results, at the cost of measurable scientific progress. However, without knowing the details of the teacher model and its data sources, scientific progress remains difficult to measure. In this paper, we study building a Perception Language Model (PLM) in a fully open and reproducible framework for transparent research in image and video understanding. We analyze standard training pipelines without distillation from proprietary models and explore large-scale synthetic data to identify critical data gaps, particularly in detailed video understanding. To bridge these gaps, we release 2.8M human-labeled instances of fine-grained video question-answer pairs and spatio-temporally grounded video captions. Additionally, we introduce PLM-VideoBench, a suite for evaluating challenging video understanding tasks focusing on the ability to reason about "what", "where", "when", and "how" of a video. We make our work fully reproducible by providing data, training recipes, code & models. https://github.com/facebookresearch/perception_models

1 Introduction

Closed-source VLMs and proprietary distillation obscure how model design, training, and data contribute to benchmark results. PLM addresses this with an open, reproducible pipeline, large human-labeled video data, and evaluation focused on detailed understanding.

  • Closed-source VLMs hide their designs, training methods, and data, while distillation from black-box models produces strong benchmark results.
  • Distillation makes scientific progress difficult to measure because teacher-model training and evaluation-set overlap are unknown.
  • Open questions include the importance of each training stage, synthetic data, and data gaps addressed artificially by proprietary distillation.
  • PLM is a fully open and reproducible model that analyzes standard training without proprietary distillation and uses synthetic-data scaling laws to identify detailed video-understanding gaps.
  • 2.8M human-labeled instances target fine-grained video QA and spatiotemporally grounded captions, while PLM achieves comparable performance with open-weight models across 40 image and video benchmarks.

2 Related Work

Related work spans image, video, and joint vision-language models, but open research is constrained by proprietary training data and model-generated annotations. Existing benchmarks also underrepresent fine-grained activity understanding and spatiotemporally grounded reasoning.

  • Vision-language models address image understanding, video understanding, and joint image-video understanding using advances such as dynamic resolution and multimodal positional embeddings.
  • Many VLMs rely on unreleased proprietary data or annotations generated by proprietary models, limiting measurement of ground-up training progress.
  • Molmo provides open-data image VLMs, but its relatively small-scale training data limits performance compared with this paper’s experiments.
  • Existing benchmarks cover broad perception, reasoning, captioning, grounding, hallucination, and video QA, but neglect fine-grained activity and spatiotemporally grounded reasoning.

3 PLM: Overview

PLM combines a vision encoder, Llama 3 decoder, synthetic data, and human annotations in a three-stage training pipeline. The design scales synthetic supervision while reserving human-labeled data for complex video understanding.

  • Model: PLM connects a pretrained Perception Encoder to a Llama 3 decoder through a 2-layer MLP projector.
  • Data: Synthetic data supports general capabilities, whereas human-annotated data adds supervision for more complex image and video tasks.
  • Data: Data engines generate approximately 66.1M synthetic samples, combined with existing and newly collected annotations targeting fine-grained and spatiotemporal video reasoning.
  • Training stages: PLM training has three stages: projector warm-up, large-scale synthetic-data midtraining, and supervised fine-tuning with human-annotated data.
  • Training stages: The projector warm-up freezes the vision encoder and language model while training the projector on 1M synthetic-captioned images to improve later stability.

4 Synthetic Data Generation and Scaling

The paper studies open-source synthetic-data generation and scaling to determine where synthetic supervision helps and where data gaps remain. Scaling follows power laws on base tasks but is weak for challenging spatiotemporal and fine-grained video understanding.

  • Synthetic data generation: The open-source data engine generates captions and question-answer pairs for natural images, documents, diagrams, OCR, and videos.
  • Synthetic data generation: 66.1M synthetic samples are curated from diverse image and video sources, including text, charts, figures, documents, and egocentric and exocentric videos.
  • Scaling laws: Scaling experiments vary synthetic-data volume, decoder and encoder size, resolution, and video frames, then fit compute-error power laws across benchmark categories.
  • Scaling laws: −0.15, −0.20, and −0.11 are the power-law exponents for Video QA, OCR QA, and Natural Image QA, respectively.
  • Synthetic-data limitation: −0.03 is the scaling trend for challenging video tasks, compared with −0.15 for general video QA, showing synthetic scaling is effective mainly for established base tasks.
  • Synthetic-data limitation: Human-annotated PLM-FGQA and PLM-STC data are introduced to address missing fine-grained activity, temporal grounding, and long-video capabilities.

5 Human-annotated High Quality Data

The paper addresses gaps in detailed video understanding by adding large-scale human annotations for fine-grained activities and spatio-temporal reasoning, then evaluates these capabilities with PLM-VideoBench.

  • Motivation: Synthetic data scales poorly on challenging video tasks requiring fine-grained activity understanding, temporal grounding, and long-video reasoning.These tasks demand detailed spatial, temporal, and semantic understanding beyond generic video recognition.
  • PLM–FGQA: PLM–FGQA contains human-verified question-answer pairs about actions, movement direction, repetition counts, hand pose, object locations, and spatial relationships.Questions are generated from salient video segments and answers are verified or replaced by annotators.
  • PLM–FGQA: 2.4M question-answer pairs span more than 780k video clips, making PLM–FGQA nearly 8 times larger than the largest existing human-annotated video QA dataset.The clips cover diverse domains and both egocentric and third-person viewpoints.
  • PLM–STC: PLM–STC pairs detailed activity descriptions with timestamps and subject masklets to represent when and where events occur.The dataset uses refined spatio-temporal segmentation and localized descriptions of subject activity changes.
  • PLM–STC: 194.2K spatio-temporal captions are converted into RCap, RTLoc, and RDCap tasks for region-level captioning, temporal localization, and dense captioning.RDCap requires a sequence of localized event descriptions covering the subject’s video duration, including periods when it is not visible.
  • PLM–VideoBench: PLM-VideoBench evaluates what activities occur, how they are performed, and where and when they occur across FGQA, SGQA, RCap, RTLoc, and RDCap.The benchmark targets capabilities that existing video benchmarks do not adequately evaluate.

6 Experiments

The experiments evaluate PLM against proprietary and open-access baselines across image, video, and PLM-VideoBench tasks, then analyze data components through ablations.

  • Evaluation Setup: PLM is evaluated against proprietary and open-access models across broad image, video, and specialized video-understanding benchmarks.The evaluation covers 20 image benchmarks, 25 video benchmarks, and PLM-VideoBench.
  • Image Benchmark Results: PLM shows strong performance across image benchmarks using open-access data and a white-box data engine.Image grounding results additionally outperform specialist models and VLM baselines at all model scales.
  • Video Benchmark Results: PLM outperforms both open-access and proprietary models on widely adopted video benchmarks despite using data free from proprietary model artifacts.Reported categories include video captioning, video QA, fine-grained video QA, and hallucination detection.
  • PLM-VideoBench Results: On PLM-VideoBench, PLM is 6.5 points above baselines on FGQA, while remaining 2.1 points below InternVL2.5 on SGQA.FGQA scores are 67.7 versus 61.2; SGQA remains below the open-access best and the strongest proprietary model.
  • Ablation Studies: Human-annotated PLM-STC and PLM-FGQA data improve distinct benchmark categories, and using them together produces the best overall ablation performance.PLM-STC improves spatial and temporal tasks by 6.8 points, while PLM-FGQA improves PLM-FGQA by 13.1 points and PLM-SGQA by 7.3 points.

7 Conclusion

The paper presents PLM as a fully reproducible, open model and releases data and PLM-VideoBench to support transparent visual-perception research.

  • Conclusion: PLM is trained without private black-box distillation using existing open-access datasets and synthetic samples from a data engine.The paper identifies detailed video-understanding gaps that synthetic data cannot fill alone.
  • Conclusion: The work releases 2.8M human labels for fine-grained video QA and spatio-temporally grounded captioning, alongside PLM-VideoBench.The benchmark evaluates fine-grained and spatio-temporal video understanding capabilities.

A PLM Training Details

PLM combines open datasets, synthetic annotations, and human-labeled data across staged training to support image, video, and text capabilities.

  • Training Data: The training data spans 47.8M synthetic image samples, 18.4M synthetic video samples, and 2.9M human-labeled video samples.The datasets support image, multi-image, video QA, fine-grained QA, region temporal localization, and region captioning tasks.
  • Data Mix: The final data mix combines publicly accessible synthetic and manually annotated image, video, and text-only datasets.Its modalities include QA, captioning, OCR, visual grounding, temporal localization, grounded captioning, and dense captioning.
  • Ablation Setup: The ablation averages validation performance across selected benchmarks and matches the final run’s hyperparameters except for SFT data-mix size.Selected benchmarks cover video captioning, short video QA, fine-grained QA, hallucination, spatial-temporal tasks, and image perception.

B Synthetic Scaling Experiments

Synthetic scaling experiments examine encoder size, input size, and compute across video, OCR, chart, document, and natural-image QA tasks.

  • Encoder Scaling: Increasing the vision encoder from 300M to 2B improves the video-QA scaling trend, with the power-law fit changing from −0.15 to −0.19.The scaling lines intersect around 8B, motivating PE-L for 1B and 3B models and PE-G for 8B models.
  • Input Scaling: Higher input size scales better for Video QA and OCR QA, while Natural QA shows a similar trend.The comparison uses 16 versus 8 tiles or frames, with dynamic tiling for images and uniform sampling for videos.
  • Synthetic Data Scaling: Synthetic data improves every OCR, chart, and document-QA benchmark by more than 10 points over human-only baselines at each model scale.These tasks show no sign of saturation, while remaining tasks exhibit a power-law relationship between metric error and FLOPs.
  • Evaluation Tasks: The benchmark suite spans image captioning, perception and reasoning, charts and documents, hallucination, video captioning, short and long video QA, fine-grained QA, and PLM-VideoBench tasks.PLM-VideoBench includes FGQA, SGQA, region captioning, dense region captioning, and region temporal localization.
  • Evaluation Metrics: PLM-VideoBench uses task-specific metrics including multi-binary accuracy, LLM-judge accuracy or similarity, SODA, and Mean Recall@1 across IoU thresholds.FGQA requires all three independent binary questions about a video to be correct for an instance to count as accurate.

C.4 Evaluation Protocols

PLM-VideoBench uses standardized frame sampling, judge models, prompts, and greedy decoding to evaluate video understanding consistently. The protocol also adapts prompting for proprietary models to improve formatting and constrain unsupported answers.

  • 32 frames are sampled uniformly from each video unless otherwise specified, and all models use greedy sampling with temperature 0.LLM-judge evaluations use Llama 3.3-70B-Instruct with established evaluation prompts where available.
  • SG-QA judging compares predicted and correct answers by meaningful semantic match, accepting synonyms and paraphrases on a 0–5 scale.The requested output is a yes/no prediction plus an integer score.
  • PLM-RCap judging scores predicted captions against ground truth from 0 to 10, penalizing missing or incorrect information.High scores require accurate conveyance of all key ground-truth information.
  • Inference prompts vary between benchmark instances while preserving the same underlying task instruction and information.For example, RCap prompts may use different wording for describing events in a highlighted region.
  • GPT-4o and Gemini receive additional instructions to enforce output formatting, reduce hallucinations, and discourage refusals.These prompts explicitly require answers to use the supplied frames rather than external knowledge or commonsense.

D Additional PLM-VideoBench Results

Additional results evaluate PLM across model scales and benchmark families, with the reported approach consistently outperforming baselines across 1B, 3B, and 8B models. The section also documents human evaluation and comparisons spanning image grounding, captioning, and long-video understanding.

  • Across 1B, 3B, and 8B scales, PLM consistently outperforms baselines, including proprietary models of unknown scale.These results supplement the 8B results reported in the main paper.
  • Human performance is reported on PLM-VideoBench using standard task prompts, with independently collected dense captions for the difficult RDCap task.Annotators answer each test instance, while RDCap uses the data annotation pipeline instead of the standard task instruction.
  • Image evaluations compare PLM with LLaMA-3V and other proprietary and open-access baselines across image benchmarks, including captioning and grounding.The synthetic data engine uses LLaMA-3V-90B to generate image captions.
  • PLM performs competitively on RefCOCO, RefCOCO+, and RefCOCOg grounding across model scales and outperforms specialist image-grounding models.The comparison is summarized in Table 14.
  • Long-video results compare PLM with open-access and proprietary baselines across three long-video question-answering benchmarks.Reported values combine literature results marked with an asterisk and results reproduced using official code.
  • PLM-FGQA is presented as a fine-grained video-QA dataset addressing scarce data for nuanced questions about how actions are performed.The dataset contains 2.4M question-answer pairs collected using a dedicated data engine.

G.1 Annotation process: Data Engine

The FGQA data engine constructs fine-grained video question-answer data from multiple video sources through segment selection, action labeling, question-answer generation, human verification, and quality control. It emphasizes visually groundable actions and questions answerable from video frames alone.

  • Data engine overview: The FGQA data engine combines temporal segment generation, question generation, answer generation, human annotation, and quality control.These modules form the pipeline used to collect the PLM-FGQA dataset.
  • Video sources: The pipeline uses publicly available videos with existing segment annotations, precisely localized physical-activity clips, and untrimmed instructional videos from HT100M.Existing annotations are adopted directly where available, while physical activities use 2–3-second segments centered on narrations.
  • Automatic segmentation: Untrimmed instructional videos are processed through temporal proposal generation, relevance filtering and ranking, and automatic segment captioning.Visual feature clustering and shot-boundary detection refine proposals before filtering and caption generation.
  • Segment filtering: Segment relevance combines talking-head detection, hand-object interaction, ASR groundability, and a learned relevance classifier.The classifier is a two-layer MLP trained on manually labeled segments, and module scores are thresholded using a manually annotated validation set.
  • Action labeling: Action labels are generated by feeding image captions, video captions, and object detections to an LLM, then fusing detailed captions with ASR content.The resulting labeled segments provide the foundation for subsequent question-answer generation.
  • Question design: Question generation is explicitly constrained toward answers grounded solely in video frames rather than external knowledge or judgments about execution quality.The goal is to collect questions not answerable from language priors alone.
  • Spatio-temporal annotations: PLM-STC adds paired object mask tublets and detailed temporal descriptions on SA-V videos while excluding test videos to avoid cross contamination.The dataset targets spatial-temporal reasoning for training and evaluation.

J Synthetic Data Engine

The synthetic data engine generates captions and question-answer pairs for image and video understanding using specialized perception models, OCR, metadata, and LLMs. Its pipelines cover captioning, OCR, natural-image QA, chart and diagram understanding, and video QA.

  • Engine scope: The engine targets image captioning, visual question answering, OCR, chart and diagram understanding, and video understanding with separate image and video pipelines.Different metadata levels are incorporated when generating captions and question-answer pairs.
  • Image captions: High-quality images are captioned with Llama 3.1V 90B for SA1B, Object365, and OpenImages.The detailed caption pipeline is illustrated in Figure 17.
  • OCR question answering: Pre-extracted OCR is passed to Llama 3.3 70B to generate five question-answer pairs for PDFAcc and UCSF.This pipeline is illustrated in Figure 18.
  • Image captions plus QA: When OCR is insufficient, an image caption and auxiliary metadata are combined with OCR and passed to Llama 3.3 70B for question-answer generation.This pipeline supports ArxivQA, DocVQA, InfoVQA, and Ai2d.
  • Video captions plus QA: Video captioning plus QA combines key-frame captions, an overall video caption sampled at 1 fps, and metadata before LLM generation of detailed captions and multiple-choice QA pairs.The pipeline is used for YT-1B, Ego4d, DiDeMo, Charades, and related video datasets.
  • Generated examples: The generated examples include natural-image descriptions, sports and chart question-answer pairs, and video captions and QA involving events, players, and touchdowns.Figures 21–28 showcase PLM-8B OCR, image QA, captioning, video region captioning, grounding, and video QA.
  • Video examples: Video examples use titles, frame-level descriptions, concise event summaries, and detailed image or video prompts to support captioning and question answering.The examples include football highlights and prompts requesting detailed image descriptions.

L Limitations and Future Work

PLM remains limited by long-video modeling, base-LLM characteristics, and a data mix focused on visual perception. The authors also acknowledge residual safety risks despite mitigation efforts, while positioning open releases as support for reproducible research.

  • Modeling limitations: PLM is less competitive on long-video benchmarks because it does not extensively experiment with long-video modeling components.The authors note compatibility with newer components such as token compression and dynamic temporal resolution for future work.
  • Modeling limitations: Performance is sensitive to the base LLM, with especially low results on MMMU, MME, and Video-MME.The strongest baselines often use more verbose LLMs with likely much larger language components.
  • Data limitations: The training data focuses on visual perception and excludes multi-step reasoning, robotics, and world-knowledge data.
  • Broader impact: Despite dataset selection and refusal training, PLM may still contain hidden biases or generate inappropriate or harmful content.The authors also remove samples containing any mention of human faces from the datasets.
  • Future impact: Openly releasing models and data allows others to reproduce the work and supports broader community involvement.
Loading 2504.13180v3…