PaperlayerView exact revision on arXiv

Source-linked AI summary

PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models

Yueyi Sun, Yuhao Wang, Jason Li, Ye Tian, Tao Zhang, Jacky Mai, Yihan Wang, Haochen Wang, Jinbin Bai, Ling Yang, Yunhai Tong

arXiv:2606.19534v1cs.CVcs.AIcs.CL

TL;DR

Captioning multiple image regions efficiently remains difficult for autoregressive and diffusion-based multimodal models. PerceptionDLM enables joint region description generation, achieving competitive caption quality with substantially faster multi-region inference.

  • Problem

    Existing multimodal models lack efficient, well-explored methods for fine-grained concurrent perception of multiple image regions.

  • Method

    PerceptionDLM combines a diffusion multimodal baseline with region-aware mask embeddings and structured attention masking for joint multi-region caption generation.

  • Results

    PerceptionDLM achieves 62.4% average accuracy on ParaDLC-Bench and up to 3.5× throughput speedup over autoregressive models in dense perception scenarios.

  • Takeaways & Limitations

    Diffusion-based multimodal models can provide competitive localized caption quality with improved efficiency for multi-region perception.

  • Takeaways & Limitations

    Inference speed remains bounded by diffusion models’ multi-step denoising, while dense similar regions can still cause attribute entanglement and hallucinations.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have achieved remarkable progress in visual understanding tasks. However, most existing MLLMs rely on autoregressive generation, which limits their efficiency for perception tasks that require captioning multiple regions. In this work, we propose PerceptionDLM, a multimodal diffusion language model optimized for efficient parallel region perception. Built upon PerceptionDLM-Base, a strong foundational baseline that achieves state-of-the-art performance among open-source diffusion MLLMs, our architecture fully leverages the parallel decoding nature of DLMs. Specifically, we introduce efficient prompting and structured attention masking to enable simultaneous perception of multiple masked regions, allowing the model to generate region descriptions in parallel at both the sequence and token levels. This design significantly improves inference efficiency compared with existing approaches that process regions sequentially. To systematically evaluate the parallelism property of visual perception capability for DLMs, we construct a new Parallel Detailed Localized Captioning Benchmark (ParaDLC-Bench) by scaling the DLC-Bench to include multiple region masks per image, enabling joint evaluation of both caption quality and inference efficiency. Experiments demonstrate that PerceptionDLM maintains competitive performance in region captioning while achieving substantial speed improvements for multi-region perception tasks. Our results highlight the potential of multimodal diffusion language models for efficient, parallel visual perception. To the best of our knowledge, we are the first to achieve parallel region caption and perception by leveraging the advantages of diffusion language models. Code, models, and datasets are released.

1 Introduction

PerceptionDLM addresses the scalability bottleneck of sequential autoregressive multi-region perception by enabling joint region captioning with diffusion-based parallel generation. It introduces a parallel perception mechanism and ParaDLC-Bench to evaluate caption quality and inference efficiency together.

  • Motivation: Autoregressive MLLMs generate each region description sequentially and token-by-token, causing inference cost and latency to grow rapidly with the number of queried regions.
  • Method: PerceptionDLM is a diffusion-based framework that generates multiple region descriptions jointly through region-aware mask embeddings and structured attention masking.PerceptionDLM-Base integrates a pretrained vision encoder with a diffusion language backbone via visual instruction tuning.
  • Evaluation: ParaDLC-Bench extends DLC-Bench from single-region to concurrent multi-region evaluation and measures both caption quality and inference efficiency.
  • Results: 15 of 16 benchmarks: PerceptionDLM-Base outperforms LLaDA-V, establishing a strong baseline among open discrete diffusion VLMs.
  • Results: 62.4% average accuracy: PerceptionDLM on ParaDLC-Bench nearly doubles LLaDA-V's 35.2% while maintaining competitive region-caption quality and improving inference speed over sequential pipelines.

2 PerceptionDLM-Base: Stronger DVLM Baseline

PerceptionDLM-Base establishes a strong, scalable diffusion-based multimodal perception baseline by combining visual instruction tuning with a vision encoder, connector, and DLM decoder. It is trained through a four-stage pipeline that progressively improves alignment, knowledge, instruction following, reasoning, and robustness.

  • PerceptionDLM-Base extends large-language diffusion models to visual instruction tuning as a strong, scalable baseline for diffusion-based multimodal perception.
  • Its multimodal architecture combines a pretrained vision encoder, lightweight vision-language connector, and LLaDA-8B DLM decoder.SigLIP-2 extracts visual features, while a two-layer GELU MLP projects them into the language model’s text embedding space.
  • Training corrupts only target response tokens with the diffusion process, preserving visual embeddings and instruction tokens as conditioning context.The model is trained with a discrete diffusion loss over masked response-token indices.
  • Four-stage training: The four-stage training pipeline progresses from connector-focused vision-language alignment, through large-scale middle-stage training, to instruction tuning and high-quality refinement.Instruction tuning uses 22M LLaVA-OneVision-1.5-Instruct-Data samples, while final refinement uses Honey-Data-15M with dual-level chain-of-thought annotations.
  • A dynamic-resolution strategy partitions images into 512×512 tiles and optionally appends an original-image thumbnail to preserve fine-grained spatial details.

3 Multimodal Diffusion Language Model for Parallel Perception

PerceptionDLM formulates multi-region captioning as a parallel diffusion-language-model generation problem, using region prompting and structured attention masking to preserve region distinctions and efficiency. The section also introduces ParaDLC-Bench and ParaCaption-5.7M to evaluate and train parallel multi-mask perception.

  • Problem Definition: Existing region-captioning methods process each region independently, causing inference latency to grow linearly with the number of regions.PerceptionDLM instead jointly models all region descriptions within a single diffusion process.
  • Architecture: PerceptionDLM builds on a region-centric visual feature extraction baseline and adds region prompting plus structured attention masking.Learnable region embeddings encode target identity, while masking enforces region-wise independence and preserves global context.
  • Architecture: The architecture simultaneously describes multiple masked regions, transforming region-level perception from a sequential process into structured parallel generation.This design leverages the intrinsic parallel decoding capability of diffusion language models for improved perception efficiency.
  • ParaDLC-Bench: ParaDLC-Bench extends DLC-Bench to multi-mask images using reference-free LLM judging and questions that test target attributes and cross-region hallucination.The benchmark uses GPT-5.2 instead of DLC-Bench’s Llama-3.1-8B and applies expert cross-verification and judge-model robustness checks.
  • ParaCaption-5.7M: ParaCaption-5.7M provides high-quality single-image, multi-mask caption data constructed because Describe Anything lacks concurrent multi-mask samples.Its construction draws on selected SA-1B and COCONut data, with filtering, region-caption generation, and category extraction steps.

4 Experiment

PerceptionDLM-Base establishes a strong baseline across multimodal understanding benchmarks, while PerceptionDLM delivers leading region-captioning accuracy and parallel decoding efficiency. Its advantages are most pronounced for multi-region perception, where it generates multiple captions simultaneously.

  • General multimodal understanding: PerceptionDLM-Base outperforms LLaDA-V on 15 of 16 evaluated multimodal benchmarks and maintains a comprehensive advantage over SDAR-VL-8B and Dream-VL-7B.It is particularly strong in general VQA, fine-grained visual perception, and hallucination robustness.
  • General multimodal understanding: PerceptionDLM-Base achieves superior or comparable scores on most benchmarks against similarly sized autoregressive VLMs, systematically outperforming Qwen2.5-VL-7B and InternVL3-8B in fine-grained perception.These results indicate a distinct advantage in region-sensitive understanding.
  • Region captioning: 62.4% average accuracy on ParaDLC-Bench nearly doubles SDAR-VL’s 31.3% and LLaDA-V’s 35.2%, while DLC-Bench accuracy reaches 51.9% versus 24.6% for baselines.PerceptionDLM leads existing diffusion-based VLMs on both multi-region and single-region captioning benchmarks.
  • Inference efficiency: PerceptionDLM achieves a Tokens Per Forward rate of 2.9, whereas autoregressive models and standard diffusion VLM baselines are restricted to 1.The higher TPF enables simultaneous generation of multiple region captions; the advantage is less exploitable on single-mask DLC-Bench instances.
  • Inference efficiency: PerceptionDLM achieves near-linear TPS growth with stable per-image latency of ∼2.9s, while GAR-8B has nearly constant TPS and latency that degrades approximately linearly with region count.The profiling evaluates computational behavior as the number of regions and parallelism increase.

5 Conclusion

PerceptionDLM is a diffusion-based multimodal model for parallel region perception that generates multiple region captions in a single denoising step. It preserves competitive caption quality while improving multi-region efficiency, and introduces ParaDLC-Bench for jointly evaluating caption accuracy and inference speed.

  • 5 Conclusion: PerceptionDLM is a diffusion-based multimodal model designed for parallel region perception.It is built on a stronger diffusion VLM baseline.
  • 5 Conclusion: Multiple region captions are generated in a single denoising step instead of being decoded one by one.This design preserves competitive caption quality while substantially improving efficiency on multi-region perception tasks.
  • 5 Conclusion: ParaDLC-Bench evaluates caption accuracy and inference speed for parallel localized captioning.The benchmark supports assessment of both caption quality and efficiency in multi-region perception.

6 Contributions · Appendix · A Overview

The paper is authored by researchers affiliated with Peking University, ByteDance, WHU, CASIA, and NUS. Its appendices provide implementation details, benchmark documentation, related work, qualitative analyses, and limitations.

  • 6 Contributions: The authors are Yueyi Sun, Yuhao Wang, Jason Li, Ye Tian, Tao Zhang, Jacky Mai, Yihan Wang, Haochen Wang, Jinbin Bai, Ling Yang, and Yunhai Tong.
  • 6 Contributions: The authors are affiliated with Peking University, ByteDance, WHU, CASIA, and NUS; Sun and Wang conducted the work during their ByteDance internships, and Tong is the corresponding author.
  • Appendix: Appendix B reports additional implementation details, experiments, and ablations of PerceptionDLM’s core components.
  • Appendix: Appendix C documents ParaDLC-Bench’s annotation guidelines, quality-control protocols, and overall dataset statistics.
  • Appendix: Appendix D extends the related-work discussion to diffusion language models and region-level multimodal perception and captioning.
  • Appendix: Appendix E presents extensive visualizations and qualitative results for fine-grained image understanding and parallel region captioning.
  • Appendix: Appendix F discusses limitations of the approach and analyzes typical failure cases.

B Additional Experiments · C Details of ParaDLC-Bench · C.1 Image and Instance Selection

Additional experiments show that PerceptionDLM’s performance depends on preserving visual representations and its three parallel-captioning design modules, while scaling data improves accuracy. ParaDLC-Bench uses diverse images and deliberately challenging multi-instance combinations to test interactions among nearby or confusable targets.

  • B Additional Experiments: The partially frozen ViT strategy significantly outperforms full-parameter training across most multimodal benchmarks and is adopted to preserve broad visual understanding.The paper also reports that fully updating the vision encoder degrades fine-grained ParaDLC-Bench performance.
  • B Additional Experiments: 62.4% average accuracy is achieved with SA-1B(SAM) annotations, compared with 57.7% using COCOnut data and 53.7% for the DAM baseline.The results show consistent gains from increasing both training-data scale and diversity.
  • B Additional Experiments: Removing region prompting reduces average accuracy to 1.1%, while standard full attention causes a 6.2% decrease and removing RoI-aligned Feature Replay causes a 2.4% drop.These ablations identify region prompting, structured attention, and localized feature replay as essential components for parallel regional perception.
  • B Additional Experiments: Reusing one visual prompt across masks lowers Qwen3.5-27B average accuracy from 73.5% to 68.6%, but the model retains parallel perception under this exceeded-prompt condition.For dense scenarios, the paper recommends splitting masks across inference passes; prompt capacity can also be increased during training.
  • B Additional Experiments: Single-mask training yields performance similar to the parallel-training baseline, indicating that parallelization does not cause the single-region gap relative to autoregressive models.This is reported from the DLC-Bench ablation.
  • C.1 Image and Instance Selection: ParaDLC-Bench draws base images from Objects365 V2 and DaTaSeg Objects365, comprising 54 images with 178 mask instances and 46 images with 121 mask instances, respectively.Both source datasets provide high-quality instance-segmentation masks.
  • C.1 Image and Instance Selection: Human annotators select multi-instance combinations that are spatially adjacent, semantically confusing, or prone to feature entanglement, followed by strict deduplication.Unlike DLC-Bench’s single challenging target, ParaDLC-Bench emphasizes interactions among multiple targets.

C.2 Data Annotation Pipeline

ParaDLC-Bench’s annotation pipeline combines GPT-5.2 extraction, expert refinement, and automated conversion into questions for multi-target scenarios. It extracts target attributes and deceptive distractors, verifies them against masked regions, and produces 2,345 multiple-choice questions.

  • Pipeline Overview: The pipeline comprises LLM extraction, human curation, and automated format conversion, using GPT-5.2 for complex logical reasoning in multi-target scenarios.The authors provide the construction prompts in Figures 8 to 14.
  • Attribute Extraction: GPT-5.2 receives cropped and segmented target images, extracting visible parts and attributes as object-part-property-value tuples.Attributes cover color, shape, texture, material, and size.
  • Attribute Extraction: Negative attribute extraction creates deceptive distractors, especially features belonging to other concurrently evaluated targets, to test cross-target confusion.The distractors also include standard background objects.
  • Human Curation & Refinement: Human experts cross-check GPT-5.2 outputs, add missing salient attributes, remove ambiguity, and ensure attributes match masked regions while distractors remain deceptive.This refinement compensates for factual errors in the initial generated lists.
  • Question Generation: 2,345 high-quality multiple-choice questions were constructed by converting manually verified attribute lists into mutually exclusive questions with GPT-5.2.Question generation occurs after manual verification of the attribute lists.

C.3 Scoring Mechanism

ParaDLC-Bench evaluates descriptions of multiple masked regions using either concurrent or sequential generation, followed by reference-free LLM judging. Its scoring requires correct base-target identification and averages per-mask scores to weight regions equally.

  • Evaluation procedure: Models generate descriptions for all specified masked regions concurrently in one pass when parallel generation is supported, or sequentially otherwise.Evaluation inherits the reference-free judge paradigm from DLC-Bench and uses an LLM Judger, GPT-5.2.
  • Scoring rule: Positive or negative points are awarded only when the base target is correctly identified, preventing gains from excessively long or ambiguous descriptions.This constraint is designed to prevent models from cheating for higher scores.
  • Scoring rule: The final score averages individual scores across all masks rather than global positive and negative scores, giving each region equal evaluation weight.This differs from DLC-Bench’s global averaging scheme.

C.4 Data Statistics · D Related Work

ParaDLC-Bench is designed for dense, fine-grained multi-region perception, with multiple masks per image and many tiny targets. Related work spans diffusion language modeling, multimodal extensions, autoregressive captioning, and localized region understanding, while this work targets parallel multi-object captioning at sequence and token levels.

  • C.4 Data Statistics: ParaDLC-Bench uses multidimensional statistical analysis to characterize its data composition and task difficulty.The analysis is conducted on the finalized dataset.
  • C.4 Data Statistics: Every benchmark image contains at least 2 masked regions, most contain 2–4, and difficult samples contain up to 8 masks.This dense multi-target setting tests local perception and resistance to interference in complex contexts.
  • C.4 Data Statistics: The average mask area ratio is 0.07, with many micro-detail targets occupying extremely small portions of each image.The distribution emphasizes fine-grained local description and small-scale understanding.
  • D Related Work: Masked diffusion models have emerged as promising alternatives to autoregressive language modeling, with LLaDA scaling masked diffusion to 8B parameters.The passage states that LLaDA demonstrates diffusion models can approach strong autoregressive LLM performance.
  • D Related Work: Multimodal DLMs include LLaDA-V for visual instruction tuning and MMaDA for combined generation and understanding, alongside autoregressive adaptations using pretrained AR knowledge.This work focuses on multimodal understanding and builds a stronger state-of-the-art diffusion-VLM baseline.
  • D Related Work: Image-captioning research covers dense, detailed, grounded, and region-sensitive settings, but prior solutions rely on autoregressive image-to-text generation.The passage situates captioning within multimodal pre-training and text-vision alignment.
  • D Related Work: This work explores parallel multiple-object caption generation with DLMs at both sequence and token levels, claiming to be the first to achieve this function.Sequence-level parallelism corresponds to objects, while token-level parallelism operates within their descriptions.
  • D Related Work: Region-level understanding requires modeling region-specific attributes, fine-grained visual details, and spatial relationships within designated areas.The passage contrasts this localized focus with conventional image-level MLLM understanding and identifies regional perception as a central research focus.

E Visualization and Qualitative Results

PerceptionDLM qualitatively demonstrates parallel perception by generating detailed descriptions for multiple masked regions in one forward pass. It preserves fine-grained visual attributes and better isolates region-specific context than the compared baselines on complex masks.

  • Parallel Perception Capabilities: PerceptionDLM simultaneously generates detailed, accurate descriptions for multiple masked regions in a single forward inference pass.This capability applies to both tightly packed objects and spatially distributed subjects.
  • Parallel Perception Capabilities: The model captures fine-grained attributes including color, texture, and shape.
  • Comparison with Baseline Models: On adjacent, visually complex masks, PerceptionDLM isolates region-specific context and aligns descriptions with the intended target regions, unlike baselines that often hallucinate or confuse features.The comparison includes LLaDA-V, GAR-8B, and Gemini-3.1-Pro.

F Limitations and Future Works

PerceptionDLM improves multi-region perception efficiency but remains limited by diffusion denoising costs and several visual failure modes. Future work targets fewer denoising steps and reduced attribute entanglement, while qualitative analysis identifies additional challenges in tiny, hallucinated, and fine-grained text regions.

  • Future Works: Inference speed remains bounded by diffusion models’ multi-step denoising process, motivating step distillation methods such as pseudo-trajectory distillation.Step distillation could compress generation into fewer steps.
  • Future Works: Attribute entanglement remains challenging in extremely dense or semantically similar regions, where neighboring colors, textures, or materials can leak into descriptions.Potential remedies include explicit region contrastive losses or finer-grained attention masking.
  • Failure Case Analysis: Tiny or heavily occluded regions can produce vague or partially incorrect descriptions because they provide limited visual evidence.An example is an overly specific but inaccurate description for a small, low-resolution region.
  • Failure Case Analysis: The model may hallucinate typical-but-absent attributes or borrow properties from co-occurring objects, especially when pixels are tiny or blurry.One failure invents white laces, eyelets, and tread for an indiscernible shoe region.
  • Failure Case Analysis: Fine-grained text recognition remains difficult, with the model misreading a license plate as “201-VH” instead of “267-JAN”.This reflects continued difficulty reading small embedded text through OCR.
Loading 2606.19534v1…