Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs

1 INTRODUCTION

• Textual box coordinates can vary in format, lack semantic alignment with visual targets, and split into discontinuous tokens that may hinder prediction accuracy.These issues can make parsing and structured output harder and can lead to repetition or hallucination between coordinates and visual targets.Source: §1 · p. 2
• PaDT interleaves Visual Reference Tokens derived from query-image patches with text, enabling MLLMs to generate textual and diverse visual targets in a unified way.Each token corresponds to a specific image patch; processing tokens independently at each forward pass preserves positional information and avoids a fixed global codebook.Source: §1 · pp. 2–3
• PaDT’s 3B model surpasses the previous state-of-the-art by 19.0 mAP on COCO detection and achieves 93.6 average accuracy on the referring expression comprehension (REC) task, outperforming 78B InternVL3.Source: §1 · p. 2
• PaDT pairs its VRTs with a fine-tuning strategy and robust per-token cross-entropy loss tailored to stabilize training and mitigate overfitting.The paper identifies this strategy and loss as part of the approach to strong performance.Source: §1 · p. 2
• A lightweight PaDT Decoder uniformly converts generated VRTs into structured visual outputs, including segmentation masks and bounding boxes.The decoder is presented as a shared component for diverse fine-grained outputs.Source: §1 · p. 3

Multimodal Large Language Models

• CLIP and ALIGN established large-scale contrastive pretraining for joint vision-text representations, followed by Q-former alignment and instruction-tuned MLLMs for open-ended visual reasoning.Later work extended MLLMs to higher-resolution, multi-image, and video inputs, among other capabilities.Source: §2 · p. 3

MLLMs for Visual Perception & Understanding

• General-purpose MLLMs remain limited in fine-grained perception; adaptive tiling and reinforcement-learning approaches address parts of the challenge, while unified architectures remain open.The reviewed literature also leaves the potential of learned queries as visual anchors underexplored.Source: §2 · pp. 3–4

Unified Visual Tokenization

• Prior visual tokenization methods use region, patch, or pixel representations, but deeper semantic integration between vision and language remains limited.The paper positions its dynamic multimodal token space as a way to closely correspond language tokens with visual patches in autoregressive modeling.Source: §2 · p. 4

3.1 REVISITING MULTIMODAL LARGE LANGUAGE MODELS

• Text-based visual targets can vary in format, disrupt numerical continuity, and lack semantic information needed for image understanding.Source: §3.1 · p. 5

3.2 VISUAL REFERENCE TOKEN

• PaDT combines a dynamic visual-token codebook, an augmented output head, and a lightweight decoder to produce bounding boxes, masks, and grounding outputs.Source: §3.2 · p. 5

3.2.1 UNIFIED MULTI-MODAL FORMAT WITH VRTS

• At each forward pass, PaDT expands the text codebook with current-image patch prototypes, embeds VRTs as inputs, and predicts them as outputs, using several object VRTs for decoding.Source: §3.2.1 · pp. 5–6

3.2.2 LIGHT-WEIGHT PADT DECODER

• The decoder groups predicted VRT features by object, adds box, mask, and score tokens, then projects them into boxes, segmentation masks, and confidence scores.Source: §3.2.2 · p. 6, Figure 4 · p. 6

3.2.3 TRAINING STRATEGY

• Training randomly samples foreground VRTs and masks unselected logits in per-token cross-entropy, alongside box, mask, and score losses, to encourage diverse visual references.Source: §3.2.3 · pp. 6–7

4 EXPERIMENT

• Evaluation covers REC and RES on RefCOCO/+/g, open-vocabulary detection on COCO2017, and RIC; joint training across benchmarks improves performance and supports prompt-based task switching.The RIC benchmark is constructed by re-annotating COCO with vision-language model supervision.Source: §4 · p. 7

4.1 VISUAL PERCEPTION & UNDERSTANDING TASKS

• PaDT Pro (3B) reaches a 93.6 overall average on REC, and both PaDT and PaDT Pro outperform the much larger 78B InternVL3 model.PaDT Pro (7B) reaches 94.5 overall average.Source: Table 1 · p. 7, §4.1 · pp. 8–9
• Both PaDT and PaDT Pro achieve the best RES performance among compared methods, including approaches using SAM segmentation models.The lightweight PaDT decoder translates visual reference tokens into segmentation masks.Source: Table 2 · p. 8, §4.1 · p. 9
• PaDT Pro reaches 38.2 mAP at 3B and 39.0 mAP at 7B on COCO2017 open-vocabulary detection, nearly doubling prior best methods.Qwen2.5-VL (3B) reaches 13.7 mAP, while InternVL3 (8B) reaches 17.5 mAP.Source: Table 3 · p. 8, §4.1 · p. 9
• On RIC, PaDT Pro (3B) reaches 1.45 CIDEr and 82.3% GreedyPrecision, while its 7B version achieves 45.8% GreddyRecall.PaDT Pro (7B) reports 1.39 CIDEr; the results indicate caption quality alongside detection-oriented performance.Source: Table 4 · p. 8, §4.1 · p. 9

4.2 ABLATION EXPERIMENTS

• SAM2-L point prompts do not improve PaDT, whereas box and mask prompts help, and combining prompts yields greater improvement at added inference cost.Figure 5 provides visualizations of the mask-refinement findings.Source: Table 6 · p. 9, Figure 5 · p. 9, §4.2 · p. 10
• Ablations find that VRTs with robust cross-entropy loss and random VRT selection improve REC and RES, while both the projection module and robust loss are needed.Using all foreground VRTs during training may hurt performance, possibly from bias toward high-density regions.Source: §4.2 · pp. 9–10, Table 5 · p. 9
• Objects365 pretraining yields stronger zero-shot performance than the Qwen2.5-VL base model, and COCO fine-tuning outperforms direct task-specific training.The analysis examines generalization and data-scaling properties.Source: Table 7 · p. 10, §4.2 · p. 10

5 CONCLUSION

• PaDT equips MLLMs to generate textual and visual outputs through Visual Reference Tokens, with a lightweight decoder and training strategy supporting perception and understanding tasks.Experiments span detection, segmentation, grounding, and captioning, with reported state-of-the-art performance.Source: §5 · p. 10
• The authors report using publicly available datasets and state that the research involved no human subjects, personally identifiable data, or sensitive information.They also report considering fairness, privacy, and security risks.Source: ETHICS STATEMENT · p. 10
• The authors state that they will release code and reproduction instructions to support reproducibility.They report that the main text describes methods and experimental setup, with additional results and dataset details in the appendix.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1.1 DATASET CONSTRUCTION

• RIC addresses the limited object-level grounding in conventional image captions by re-annotating COCO2017 with fine-grained annotations.The dataset is intended to make it possible to assess whether captions capture entities' spatial locations.Source: §A.1.1 · p. 16
• For each image, Qwen2.5-VL-72B is prompted with box-level annotations to produce captions that reference corresponding box IDs.The prompt requires listed instance IDs and instructs the model to append them after mentioning objects.Source: §A.1.1 · p. 16
• The illustrated Qwen2.5-VL-72B example links mentioned objects—including a person, sink, and table—to their instance IDs.The example is a fluent caption describing a kitchen scene.Source: §A.1.1 · p. 16

A.2 TEMPLATE EXAMPLE FOR EACH VISION TASK

• The templates are designed to be concise and efficient while allowing textual tokens and VRT predictions to be interleaved during training.They cover open-vocabulary detection, referring expression comprehension, and referring image captioning.Source: §A.2 · p. 16
• The open-vocabulary detection template places VRT sequences after each named object category in the assistant response.The prompt asks the model to detect a target list in the image.Source: §A.2 · p. 16
• For referring expression comprehension and segmentation, the assistant response associates the described object with a sequence of VRTs.The user prompt supplies a natural-language description of the target object.Source: §A.2 · p. 16
• The referring image captioning template attaches a VRT sequence to each described object in the caption.The accompanying dataset-sample figure describes images with 3–5 captions asking MLLMs to refer to objects via boxes or VRTs.Source: §A.2 · p. 17, Figure 6 · p. 17

• Open-vocabulary detection prompts request target bounding boxes in JSON, with a “None” response when no targets are detected.Source: §A.3 · p. 18
• REC and RES prompts ask the model to identify the described object and return the final answer in JSON.Source: §A.3 · p. 18
• Referring image captioning prompts ask for object descriptions with bounding boxes embedded in the sentence.Source: §A.3 · p. 18

A.4 THE FORMULA OF THE TASK-SPECIFIC LOSSES ON THE PADT DECODER OUTPUT

• The decoder-output losses are defined for predicted boxes, masks, and confidence scores against their respective ground truths.Source: §A.4 · p. 18

A.5.1 TOKEN ACTIVATION MAP ANALYSIS

• Token activation maps show stronger associations between VRTs and target image patches than digit-by-digit coordinate predictions.Source: §A.5.1 · p. 18, Figure 7 · p. 19

A.5.2 ABLATION STUDY OF OTHER USED LOSSES

• Using all visual-task losses yields 93.2% REC accuracy and 76.1 mask cIoU; removing the dynamic embedding module or any individual loss degrades REC and segmentation.Source: §A.5.2 · p. 18, Table 8 · p. 19

A.5.3 ABLATION STUDY OF THE NUMBER OF SELECTED VRTS PER TARGET

• Increasing selected patches per target from 1 to 5 steadily improves box accuracy and mask cIoU, while 8 patches bring diminishing or negative returns.Source: §A.5.3 · p. 19, Table 9 · p. 20
• Training with all foreground patches performs worst, as redundant, low-resolution patches make decoding trivial or overlapping regions harder and impair boundary and mask learning.Source: §A.5.3 · p. 19, Figure 8 · p. 20

A.6.1 OPEN VOCABULARY DETECTION ON COCO2017 DATASET

• On COCO2017, PaDT qualitatively shows higher recall, robustness among similar objects, and fewer invalid outputs than MLLMs that serialize bounding-box coordinates.Source: §A.6.1 · p. 21, Figure 9 · p. 22
• For REC/RES, five patch-linked VRTs feed box and mask decoding; for OVD, labels and VRTs interleave to represent multiple targets.Source: §A.6.1 · p. 21

A.6.2 REFERRING IMAGE CAPTIONING ON RIC DATASET

• On RIC, PaDT shows better box accuracy and object recall than InternVL3 and Qwen2.5-VL, with examples interleaving VRTs alongside subjects.Source: §A.6.2 · p. 21, Figure 11 · p. 24, Figure 12 · p. 24