PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding

Motivation and approach

• Closed-source VLMs obscure their designs, training methods, and data, while black-box distillation [1] [3] can produce strong benchmark results by labeling training data [3] [6].This obscures whether benchmark gains reflect model or training advances or proprietary teachers’ exposure to evaluation-related data.Source: §1 · p. 1

Contributions

• PLM is a fully open and reproducible model with a vision encoder and a small-scale (<8B-parameter) LLM decoder for transparent image and video understanding research.The study analyzes standard training pipelines using available data without proprietary model distillation.Source: §1 · p. 2

Motivation and approach

• The study investigates large-scale synthetic data and establishes scaling laws to identify critical video-understanding gaps, especially in spatio-temporal reasoning and fine-grained understanding.It also examines which gaps may be artificially addressed by distillation from proprietary models.Source: §1 · p. 2

Contributions

• Across 40 image and video benchmarks, PLM achieves comparable performance to open-weight state-of-the-art models such as InternVL2.5 [10] and greatly outperforms fully open models such as Molmo [11].These results are obtained without distilling from proprietary models.Source: §1 · p. 2
• The authors release 2.8M human-labeled instances of fine-grained video question-answer pairs and spatio-temporally grounded captions, nearly an order of magnitude larger than the largest existing datasets of each type [8] [9].The release supports reproducible and transparent training and evaluation of vision-language models.Source: §1 · p. 2, Figure 1 · p. 2

Vision-Language Models

• VLMs address image understanding [1] [13] [14] [15] [16] [17] [18] [19], video understanding [20] [21] [22] [23] [24] [25] [26] [27], and joint image-video understanding [10] [28] [29] [30].Representative modeling advances include dynamic high resolution inputs, adaptive token compression, and multimodal positional embeddings.Source: §2 · p. 2

Open source, open data VLMs

• Many VLMs rely on unreleased proprietary data [34] [35] or data generated by proprietary models such as GPT4o [3], hindering ground-up training research.Molmo [11] provides open-data image VLMs, but its relatively small-scale training data limits performance.Source: §2 · p. 2

VLM Benchmarks

• PLM–VideoBench targets neglected video-centric capabilities through fine-grained activity understanding and spatio-temporally grounded reasoning [91] [92] [96] [97] [98] [99] [100] [101] [102] [103].Recent benchmarks address long-video and fine-grained temporal reasoning, but these areas remain relatively neglected.Source: §2 · pp. 2–3

Model

• PLM connects a pre-trained Perception Encoder [104] to a Llama 3 decoder [13] through a 2-layer MLP projector, supporting 1B, 3B, and 8B model sizes.It uses PE L/14 for the 1B and 3B models and PE G/14 for the 8B model.Source: §3 · p. 3
• PLM supports high-resolution inputs with dynamic image tiling up to 36 448^2 tiles and video processing of 32 448^2 frames, using 2 × 2 average pooling to compress visual tokens.The same spatial pooling is applied to each video frame.Source: §3 · p. 3

Data

• The training data combines diverse synthetic samples for general capabilities with human-annotated data for complex, high-quality image and video tasks.Synthetic sources cover OCR, chart, document, diagram, captioning, and visual question-answering capabilities, while human annotations include existing and newly collected data for fine-grained video understanding and spatio-temporal reasoning.Source: §3 · p. 3
• Data engines scale synthetic generation across image and video modalities to approximately 66.1M samples, while human-annotated data supplies richer supervision for fine-grained and spatio-temporally grounded reasoning.The engines cover natural images, charts, documents, figures, egocentric videos, and exocentric videos; synthetic data may be noisy but is available at scale.Source: §3 · p. 3

Training stages

• PLM is trained in three stages: projector warm-up on 1M synthetic images, synthetic midtraining on about 64.7M images and videos, and supervised fine-tuning with human-annotated data.The first stage freezes the vision encoder and language model for projector stability; the second uses up to 16 image tiles and video frames; the final stage increases capacity to 36 tiles and 32 frames for challenging video tasks.Source: §3 · p. 3

4 Synthetic Data Generation and Scaling

• The open-source data engine targets base image and video understanding capabilities without relying on proprietary models, scaling generation to around 66.1M samples.The resulting corpus spans natural images, in-the-wild text, charts, figures, documents, egocentric videos, and exocentric videos.Source: §4 · p. 4, §4.1 · p. 4

4.1 Data Engine

• For images, Llama 3 [13] generates factual detailed captions, while OCR, captions, and metadata guide informative question-answer generation for natural and document-rich images.The image engine covers natural images, documents, diagrams, and OCR-recognizable text while aiming to minimize hallucinations.Source: §4.1 · p. 4
• For videos, scene detection yields approximately 30-second clips whose frame captions, PLM-generated video captions, metadata, and LLM refinement produce detailed captions and question-answer pairs.The process uses keyframes and metadata such as action labels and time tags to create cohesive video-level descriptions.Source: §4.1 · p. 4

4.2 Scaling Laws with Synthetic Data

• Synthetic-data training follows power laws with exponents −0.15 for Video QA, −0.20 for OCR QA, and −0.11 for Natural Image QA.The analysis varies synthetic-data scale, decoder and encoder sizes, resolution, and video frames, evaluating 20 downstream vision-language benchmarks across three task categories.Source: §4.2 · pp. 4–5, Figure 2 · p. 4
• Challenging video tasks scale poorly (−0.03) versus general video QA (−0.15), showing synthetic-data scaling remains effective mainly for established base tasks.The challenging set covers fine-grained activity understanding, temporal grounding, and long-video reasoning; the paper addresses this limitation with human-annotated PLM–FGQA and PLM–STC data.Source: §4.2 · p. 5

5 Human-annotated High Quality Data

• Synthetic data offers limited gains on challenging video benchmarks, motivating human-annotated data covering fine-grained activity understanding, temporal grounding, and long-video understanding.Training on tens of millions of synthetic annotations hardly improves performance on new challenging benchmarks.Source: §5 · p. 5, Figure 3 · p. 5
• PLM–FGQA contains 2.4M human-verified question-answer pairs across more than 780k diverse video clips, covering fine-grained actions, temporal perception, spatial relationships, and activity execution details.Its multi-stage pipeline extracts salient segments, generates questions and answers, and has humans verify or replace incorrect answers.Source: §5 · p. 5
• PLM–FGQA is approximately 8 times larger than the largest existing human-annotated video QA dataset and adds scarce annotations about movement direction, object states, locations, and spatial relations.The dataset spans domains including cooking, DIY, carpentry, automotive repair, and bike repair, with both egocentric and third-person viewpoints.Source: §5 · pp. 5–6, Figure 4 · p. 6
• PLM–STC provides 194.2K spatio-temporal captions grounded to activity timestamps and high-frame-rate subject masklets, yielding 522.7K RCap, RTLoc, and RDCap training samples.Annotators refine SAM 2-generated mask tublets and write temporally localized descriptions, including automatic “out of frame” captions when subjects leave view.Source: §5 · p. 6

5.1 PLM–VideoBench

• PLM–VideoBench evaluates holistic video understanding through fine-grained activity and execution questions situated in spatio-temporal contexts: what, how, where, and when.It includes FGQA, SGQA, RCap, RTLoc, and RDCap tasks spanning multiple-choice, open-ended, captioning, temporal localization, and dense captioning formats.Source: §5.1 · p. 7, Figure 6 · p. 7
• The benchmark also measures nuanced activity recognition with 4,371 FGQA pairs, smart-glasses assistance with 665 SGQA pairs, region captioning with 10,060 RCap instances, and dense subject-centric event description with 2,620 RDCap samples.FGQA uses multi-binary accuracy, SGQA and RCap use LLM-judge accuracy with Llama3.3 70B, and RDCap uses the LLM-judged SODA score.Source: §5.1 · pp. 7–8
• RTLoc requires precise start and end timestamps for a specified event and subject, uses unambiguously localized captions, and reports average recall@1 over IoU thresholds 0.3, 0.5, 0.7, and 0.9.Its filtered test set contains 7,910 instances and defines the inverse of RCap by receiving the caption and predicting its time interval.Source: §5.1 · p. 7

6.1 Setup

• The evaluation compares PLMs with proprietary GPT-4o, Gemini-Pro 1.5 [34], and Gemini-Flash 2.0 [35], plus open-access Molmo-O [11], LLaVA-OneVision [28], Qwen2.5-VL [106], and InternVL2.5 [10].Proprietary models are evaluated through API calls, while open-access models use their official inference code.Source: §6.1 · p. 8

6.2 Image Benchmark Results

• PLM performs strongly across 20 image benchmarks using solely open-access data, and it outperforms specialist and VLM baselines at every tested scale on RefCOCO/+/g grounding.The image evaluation spans charts, diagrams, documents, captioning, perception and reasoning, and hallucination tasks.Source: §6.2 · pp. 8–9, Table 3 · p. 8

6.3 Video Benchmark Results

• PLM achieves strong results across video captioning, video QA, fine-grained QA, and hallucination benchmarks, outperforming both open-access and proprietary models.The reported competitive results include EgoSchema (68.8 %), MotionBench (61.4 %), TOMATO (33.2 %), TempCompass (72.7 %), TemporalBench (28.3 &), and Charades-STA (58.6 %).Source: §6.3 · p. 9, Table 4 · p. 9

6.4 PLM-VideoBench Results

• On PLM-VideoBench, PLM leads proprietary and open-access baselines on FGQA by 6.5 points (67.7 vs 61.2), remains competitive overall, and supports grounded spatio-temporal reasoning.On SGQA, PLM is 2.1 points below InternVL2.5, while open-access baselines fail to perform grounded reasoning on RDCap, DCap, and RTLoc.Source: §6.4 · pp. 9–10, Table 5 · p. 9

6.5 Ablation Studies

• Stage 2 synthetic training improves performance broadly, while PLM-STC and PLM-FGQA add complementary gains across grounding, captioning, fine-grained understanding, hallucination, and spatial-temporal tasks.PLM-STC yields +27.4 points on its category, +2.4 on video captioning, and +6.8 on spatial and temporal tasks; PLM-FGQA yields +13.1 on its category and +7.3 on PLM-SGQA.Source: §6.5 · p. 10, Table 6 · p. 10
• Using all human-annotated data produces the best overall ablation performance, and improves HardQA [97] [100] [89] [101] [99] [92] beyond synthetic-data limitations.Human-annotated data also improves fine-grained video tasks (+1.3 points), video hallucination (+3.0 points), and spatial and temporal tasks (+2.2 points).Source: §6.5 · p. 10, Figure 7 · p. 10

7 Conclusion

• PLM is a fully reproducible vision-language model designed to tackle visual perception without distillation from private black-box models.The model uses data from existing open-access datasets and synthetic samples generated by a data engine.Source: §7 · p. 10
• Synthetic data could not fill identified gaps in detailed video understanding capabilities.Source: §7 · p. 10
• 2.8M human-labeled instances address these gaps through fine-grained video question answering and spatio-temporally grounded captioning.Source: §7 · p. 10
• PLM-VideoBench evaluates detailed video-understanding capabilities targeted by the released data.The benchmark was created alongside the human-labeled question-answering and captioning data.Source: §7 · p. 10
• The released open dataset, benchmark, and models are intended to foster transparent research in visual perception.Source: §7 · p. 10
• The 23.8M datamix comprises TextQA 500K, Image QA 2.8M, and Video QA 500K.Further details are provided in Table 9.Source: §7 · p. 10

A PLM Training Details

• 47.8M synthetic image, 18.4M synthetic video, and 2.9M human-labeled video samples support standard and fine-grained visual understanding tasks.Supported tasks include Image, Multi-image, and Video QA, plus Fine-grained QA, Region Temporal Localization, Region Captioning, and Region Detailed Captioning.Source: Figure 8 · p. 12

A.1 PLM Training Setting

• Across all three stages, PLM uses AdamW with 0.05 weight decay, FSDP, FlashAttention2, and a PyTorch implementation.Stage-specific settings otherwise vary in batch size, learning rate, resolution, token budget, and the amount of dynamic image or video content.Source: §A.1 · pp. 12–13
• Stage 1 is a warm-up using 1M SA-1B samples with generated detailed captions, frozen vision encoder and LLM parameters, and next-token prediction.Training uses learning rate 1 × 10^-4, global batch size 512, 448 × 448 resolution, and scale-specific Perception Encoder variants.Source: §A.1 · p. 12
• Stage 2 trains on 72.5M samples, combining 66M synthetic image/video annotations with 6.5M human-annotated samples while jointly optimizing the vision encoder, projector, and LLM.Inputs use 448 × 448 tiles or frames, dynamic tiling, 1-fps video sampling, and up to 16 tiles or frames.Source: §A.1 · p. 12
• Stage 3 uses 19.1M high-quality multimodal samples and full-parameter training, with up to 36 image tiles or 32 video frames and an 11,264-token sequence length.The learning rate is 1 × 10^-5 for 8B models and 4 × 10^-5 for 1B and 3B models.Source: §A.1 · pp. 12–13

A.2 PLM Training Datamix

• The training datamix combines public synthetic and manually annotated image, video, and text-only datasets while excluding annotations generated by proprietary models.Stage 2 emphasizes synthetic captioning data, whereas Stage 3 contains roughly one third data focused mostly on human-annotated samples across diverse tasks.Source: §A.2 · p. 13, Table 9 · p. 14

A.3 Ablation Experiment Details

• The ablation evaluates average scores across five benchmark categories and matches the final SFT run’s hyperparameters, changing only the SFT datamix size.Selected benchmarks cover video captioning, short-video and fine-grained QA, hallucination, spatial and temporal tasks, and image perception.Source: §A.3 · p. 14

B Synthetic Scaling Experiments

• The larger 300M-to-2B vision encoder improves video-QA scaling, changing the power-law fit from −0.15 to −0.19 and benefiting PLMs at 8B scale or larger.The two scaling lines intersect around 8B; the default uses PE-L for 1B and 3B language models and PE-G for 8B.Source: B Synthetic Scaling Experiments · pp. 14–15, Figure 9 · p. 15
• 2.0 points lower metric error results from training with 16 rather than 8 video frames at 10^13 FLOPs, motivating 16 as the final stage-2 setting.Higher input size also improves OCR QA scaling, but not Natural QA; models use dynamic tiling, uniform video sampling, and 448 × 448 tiles or frames.Source: B Synthetic Scaling Experiments · p. 15, Figure 10 · p. 15
• Synthetic data improves every OCR, Chart, and Document QA benchmark by more than 10 points over human-only baselines, without observed saturation.The authors hypothesize these tasks resemble pixel-to-text translation, while remaining tasks exhibit clean power-law relationships between error and compute.Source: B Synthetic Scaling Experiments · p. 15, Figure 11 · p. 16

C VLM Benchmark Details

• The benchmark appendix defines image and video benchmark compositions, evaluation metrics, inference parameters, prompts, and available evaluation-code pointers for the evaluations in §6.It covers image benchmarks in §C.1, video benchmarks in §C.2, and PLM-VideoBench in §C.3.Source: C VLM Benchmark Details · p. 16

C.1 Image Benchmarks

• Image evaluation spans captioning, general and hard perception, charts and documents, and hallucination robustness, using CIDEr, accuracy, or HallusionBench’s aAcc metric.The covered benchmarks include COCO [49], nocaps [50], Flickr, MMMU [37], VQAv2, MMBench, and HallusionBench [67].Source: §C.1 · pp. 16–17

C.2 Video Benchmarks

• Video evaluation covers short and long video captioning and QA, fine-grained reasoning, and hallucination robustness using benchmark-specific CIDEr, accuracy, LLM-judge, AutoDQ F1, VDC, and MBAcc metrics.Benchmarks include YouCook2 [83], VATEX [84], EgoSchema, LVBench [92], TemporalBench [99], and VideoHallucer [88].Source: §C.2 · p. 17

C.3 PLM-VideoBench

• PLM-VideoBench targets fine-grained and spatio-temporal video reasoning through FGQA, SmartGlasses-QA, region captioning, dense region captioning, and temporal localization.Its suite includes FGQA, SGQA, PLM-RCap, PLM-RDCap, and PLM-RTLoc.Source: §C.3 · pp. 17–18
• PLM-VideoBench uses stricter multi-binary accuracy for FGQA and Mean Recall@1 across IoU thresholds 0.3, 0.5, 0.7, and 0.9 for PLM-RTLoc.Captioning uses LLM-judge similarity, with dense captioning feeding those similarities into an adapted SODA metric.Source: §C.3 · pp. 17–18

• Video evaluations uniformly sample 32 frames, use Llama3.3-70B-Instruct [13] for LLM judging, follow established framework prompts, and decode outputs greedily at temperature 0.Source: §C.4 · p. 18
• SG-QA judging compares predicted and correct answers by meaningful semantic match, accepts synonyms and paraphrases, and returns a yes/no prediction with a 0–5 score.Source: §C.4 · p. 18
• PLM-RCap judging scores captions from 0 to 10 based on contextual correctness, penalizing missing or incorrect information and requiring all key ground-truth content for high scores.Source: §C.4 · p. 18
• PLM-VideoBench provides task-specific inference prompts, with permissible wording variation when prompts preserve the same underlying instruction and information.Source: §C.4 · p. 18, Table 10 · p. 19
• Proprietary models receive extra anti-hallucination, anti-refusal, formatting, and in-context-example instructions, which are released with the code for reproducibility.Source: §C.4 · p. 18

D Additional PLM-VideoBench Results

• Across 1B, 3B, and 8B scales, PLM consistently outperforms baselines, including proprietary models of unknown size, in PLM-VideoBench results.Table 11 supplements the main paper’s 8B results with evaluations across all model scales.Source: D Additional PLM-VideoBench Results · p. 19, Table 11 · p. 19

E Baseline Implementation Details

• Proprietary GPT and Gemini baselines are evaluated through APIs using specified checkpoints, frame sampling, image quality, and audio-excluded video inputs.GPT-4o uses 32 uniformly sampled frames, while Gemini processes VQA videos internally at 1 fps.Source: E Baseline Implementation Details · pp. 19–20
• Human performance is reported on PLM-VideoBench using annotator answers to each task’s test instances and standard prompt, with independently collected dense captions for difficult RDCap.The RDCap procedure reuses the data-annotation pipeline rather than the standard task instruction.Source: E Baseline Implementation Details · p. 20

F.1 Comparison with LLaMA-3V

• PLM is compared with LLaMA-3V-90B on image benchmarks, which also identifies LLaMA-3V-90B [13] as the image-caption generator in the synthetic-data engine.The comparison is presented in Table 12.Source: Table 12 · p. 20

F.2 Image Captioning

• Image-captioning benchmarks compare PLM with proprietary models and open-access baselines at comparable scale.These results are presented in Table 13.Source: Table 13 · p. 20

F.3 Image Grounding

• On RefCOCO, RefCOCO+, and RefCOCOg, PLM performs competitively across model scales and outperforms specialist image-grounding models.The comparisons are reported in Table 14.Source: Table 14 · p. 21

F.4 Long Video Understanding

• Long-video results compare PLM with open-access and proprietary baselines across three long-video question-answering benchmarks, combining reproduced results with literature-reported values.Table 15 marks literature-reported cells with an asterisk and reproduces the remaining results using official code.Source: Table 15 · p. 21

G PLM–FGQA: Fine-grained QA

• PLM–FGQA is a 2.4M-question-answer video dataset focused on nuanced “how” actions, created with a data engine because fine-grained video Q&A resources are scarce.The dataset uses specially designed questions and carefully annotated answers; Table 16 compares it with existing video-QA datasets.Source: G PLM–FGQA: Fine-grained QA · p. 22, Table 16 · p. 22

G.1 Annotation process: Data Engine

• The data engine combines temporal segment generation, question and answer generation, human annotation, and quality control to produce PLM–FGQA data.Figure 12 summarizes these modules and their progression into the FG-QA component of PLM–VideoBench.Source: §G.1 · p. 22, Figure 12 · p. 22

G.1.1 Temporal Segment Generation

• Temporal segments are adopted from annotated datasets, extracted as 2–3-second EgoExo4D clips centered on narration timings, or automatically proposed from untrimmed HT100M videos.For HT100M videos, proposals are refined by aligning boundaries with nearby shot boundaries, and segments are captioned by combining perception-model outputs with ASR through Llama 3.3 70B [13].Source: §G.1.1 · pp. 22–24

G.1.2 Automatic Question Generation

• Questions target visually observable fine-grained details such as tools, object locations and states, movement direction, hand pose, spatial relations, and temporal changes.Generation combines action-conditioned LLM prompts, an instance-specific VLM question generator trained on approximately 600k synthetic instances, and follow-up questions based on existing Q&A pairs; questions are designed to require video perception rather than external knowledge.Source: §G.1.2 · pp. 24–25

G.1.3 Automatic Answer Generation

• Answers are generated by a privileged-information PLM using metadata, existing Q&As, dense frame features, OWLv2 object embeddings, and CLIP ViT-L14 embeddings.A Perceiver-IO [227] tokenizer incorporates off-the-shelf embeddings, and the answer generator is fine-tuned on 1M manually annotated QA pairs before producing answers for unlabeled questions.Source: §G.1.3 · pp. 25–26

G.1.4 Human Annotation

• Human annotation either verifies model-generated answers or writes answers from scratch, with manually annotated answers comprising 82.2% of the PLM–FGQA training split and 100% of evaluation data.Rejected questions are discarded, while factually incorrect or insufficient answers are corrected; certification, golden examples, audits, and quality-control checks covered 13% of training data and 100% of PLM–VideoBench samples.Source: §G.1.4 · pp. 26–27

G.2 FGQA PLM–VideoBench Construction

• FG-QA VideoBench converts held-out PLM–FGQA data into balanced MCQs using semantically close hard negatives, text-only-answer filtering, human verification of distractors, and question-type/domain balancing.The benchmark reports multi-binary accuracy [99], which compares the correct answer with each distractor and requires consistent selection across all comparisons.Source: §G.2 · pp. 27–28

H PLM–STC Details

• PLM–STC provides object mask tubelets paired with detailed temporal descriptions for spatial-temporal reasoning, using diverse SA-V [124] videos while excluding test videos to prevent contamination.The dataset includes statistics on samples, splits, object types, and segment duration, and is presented as larger and higher quality than existing datasets.Source: H PLM–STC Details · p. 28

H.1 Annotation Process

• The PLM–STC annotation pipeline has three stages: object selection and tracking, temporal segmentation and captioning, and verification and quality control.The overall process is summarized in Figure 14.Source: §H.1 · p. 28, Figure 14 · p. 29

H.1.1 Object Selection and Tracking

• Annotators select dynamically changing objects and refine SAM 2-generated mask tublets for high-quality spatial-temporal segmentation.Interesting regions contain significant evolving actions by people, animals, or objects, rather than static or insignificant activity.Source: §H.1.1 · p. 29

H.1.2 Temporal Segmentation and Captioning

• Annotators segment each action temporally and caption major visible movements and directionality without assumptions or unnecessary details.Captions are tied to time segments and restricted to directly observable actions.Source: §H.1.2 · p. 29

H.1.3 Verification and Quality Control

• A final annotation stage verifies masks and captions by removing redundancy, correcting inaccuracies, and adding missing major actions.Mask refinement repeats the earlier pipeline while annotators only improve mask quality; captions preserve existing atomic actions while improving event coverage.Source: §H.1.3 · p. 29, Table 20 · p. 30

H.2 PLM–STC Benchmark

• PLM–STC uses balanced, non-overlapping validation and test splits to evaluate Dense Video Region Captioning, Video Region Captioning, and Region Temporal Localization.RDCap covers what and when with dense temporal coverage, RCap describes activities in specified time regions, and RTLoc predicts event start and end frames while excluding out-of-frame or ambiguous events.Source: §H.2 · p. 30

I.1 Data collection and annotation

• PLM-SGQA source videos are hands-free egocentric recordings across 14 scenarios, with participants asking spontaneous visual questions and annotating relevant temporal segments and answers.The scenarios include shopping, cooking, and neighborhood walking, while questions emphasize stable visual content rather than regularly changing information.Source: §I.1 · pp. 30–31

I.2 SGQA Benchmark

• The SGQA evaluation set contains 655 perception-based Q&As filtered for human activity and visual understanding, with clips trimmed to annotated boundaries ending when the wearer asks the question.Evaluations must therefore specify that the question refers to the end of the video clip.Source: §I.2 · p. 31, Figure 15 · p. 31, Figure 16 · p. 31

J Synthetic Data Engine

• The synthetic-data engine targets image captioning, visual question answering, OCR, chart and diagram understanding, and video understanding using pipelines with varying metadata levels.Separate image and video pipelines support these base VLM capabilities.Source: J Synthetic Data Engine · p. 31
• Image captioning uses Llama 3.1V 90B to caption SA1B, Object365, and OpenImages.The pipeline is demonstrated with a detailed caption example in Figure 17.Source: J Synthetic Data Engine · p. 31, Figure 17 · p. 32

Pipeline

• OCR-based generation feeds pre-extracted OCR to Llama 3.3 70B, which produces five question-answer pairs for PDFAcc and UCSF.Figure 18 shows the OCR-to-question-answer pipeline.Source: J Synthetic Data Engine · p. 31, Figure 18 · p. 32
• When OCR is insufficient, captioning with Llama 3.1V 90B is combined with auxiliary metadata and Llama 3.3 70B to generate captions and question-answer pairs.This pipeline supports ArxivQA [59], DocVQA [53], InfoVQA, and Ai2d, as illustrated in Figure 19.Source: J Synthetic Data Engine · p. 31, Figure 19 · p. 33
• Video generation combines key-frame captions, a 1-fps video caption, and metadata before Llama 3.3 70B or Llama 3 405B produces detailed captions and multiple-choice question-answer pairs.The pipeline is used for YT-1B, Ego4d, DiDeMo, Charades, and Kinetics710 [121]; DiDeMo, Charades, and Kinetics710 use only frame captions with Llama 3.1 8B.Source: J Synthetic Data Engine · p. 31, Figure 20 · p. 34
• The pipeline examples show detailed outputs for image captions, OCR question answering, image captioning with multiple-choice questions, and video captions with multiple-choice questions.Examples include a winter train-station scene, standings-table questions, an astronomy histogram question, and football-video descriptions.Source: J Synthetic Data Engine · p. 32, Pipeline · pp. 33–34

K Qualitative Results

• The qualitative examples include PLM-8B OCR, OCR in the wild, natural-image question answering, image captioning, and video region captioning, grounding, and fine-grained question answering.Source: Figure 21 · p. 35, Figure 22 · p. 35, Figure 23 · p. 35

L Limitations and Future Work

• PLM performs strongly against open-data and proprietary models, but long-video results remain less competitive because long-video modeling components were not extensively explored.Future work can incorporate token compression and dynamic temporal resolution, which PLM supports.Source: L Limitations and Future Work · p. 39
• PLM performance is sensitive to the base LLM, with especially low results on MMMU [37], MME [41], and Video-MME [75].These benchmarks often favor more verbose models with likely larger language components.Source: L Limitations and Future Work · p. 39
• PLM performs relatively poorly on SGQA, while chatbot-focused systems such as GPT-4o excel at tasks extending beyond core perception.SGQA combines perception and knowledge questions for smart glasses.Source: L Limitations and Future Work · p. 39
• PLM’s data mix focuses on visual perception and excludes multi-step reasoning, robotics, and world-knowledge data, despite contributing strong benchmarks and reproducible VLM research.The authors characterize PLM as setting a new standard for fully reproducible VLMs.Source: L Limitations and Future Work · p. 39

M Broader Impact

• The work advances open, reproducible research by releasing models, data, and benchmarks without proprietary distillation, while acknowledging risks of hidden bias and harmful content.Mitigations include refusal training for bias, harassment, and adult-content questions, plus removing samples mentioning human faces.Source: M Broader Impact · p. 39
• The openly released fine-grained video dataset can broaden image- and-video research, improve reproducibility, and support safer, more accountable investigation of biases and limitations.The release includes video question answering and spatio-temporal grounding annotations and invites broader community involvement.Source: M Broader Impact · p. 39
• By openly sharing models and data while addressing ethical concerns, the work aims to contribute positively to vision-language research.This broader-impact goal complements the dataset, model, and benchmark releases.Source: M Broader Impact · p. 39