PaperlayerView exact revision on arXiv

Source-linked AI summary

DVD: Dynamic Vector Decoding for Efficient MLLM-based Perception

Jinghua Hou, Zhe Liu, Hengshuang Zhao

arXiv:2610.12266v1cs.CVcs.AI

TL;DR

Existing MLLM perception methods face token overhead from text coordinates and range and precision constraints from fixed-range quantization, particularly in 3D. DVD maps heterogeneous 2D and 3D representations to compact discrete tokens, and benchmark results report superior performance with lower token count and inference latency.

  • Problem

    MLLM perception methods rely on token-intensive text coordinates or fixed-range quantization, which constrains range and precision for 3D scenes.

    Source: §1 · p. 2

  • Method

    DVD converts 2D boxes, 2D masks, and 3D boxes into 1D sequences, maps them to compact high-dimensional discrete tokens, and uses a de-tokenizer to recover the original representations.

    Source: §1 · pp. 2–3, §3.2 · p. 4

  • Results

    DVD reports superior performance on 2D and 3D perception benchmarks while significantly reducing token count and inference latency.

    Source: §5 · p. 11

  • Takeaways & Limitations

    DVD offers an efficient, general framework for equipping MLLMs with perception capabilities across 2D and 3D tasks.

    Source: §5 · p. 11

    The separate DVD de-tokenizer may lead to suboptimal alignment between learned discrete tokens and the MLLM's semantic understanding.

    Source: §F.1 · p. 17

Abstract

from arXiv · show

Multimodal large language models have made remarkable progress in bridging vision and language, facilitating various perception tasks essential for human-machine interaction, robotics, and autonomous driving. However, existing MLLM-based perception methods predominantly rely on text-based coordinate representation, which suffers from excessive token overhead, or fixed-range quantization, which suffers from range and precision constraints, especially for 3D domains with unbounded spatial range and high localization accuracy requirements. To address these challenges, we propose a dynamic vector decoding method named DVD, which unifies the representation of 2D and 3D perception tasks. Specifically, we first transform diverse perceptual representation (i.e., 2D bounding boxes, 2D masks, and 3D bounding boxes) into 1D vector sequences, which are then mapped to compact discrete tokens in the high-dimensional space. Then, a lightweight de-tokenizer enables seamless integration with MLLMs by decoding output tokens back to original 2D and 3D perceptual representations. Extensive experiments on 2D and 3D perception benchmarks including RefCOCO series, SUN-RGBD, KITTI, Hypersim, nuScenes demonstrate that DVD achieves superior performance in 2D and 3D tasks and reduces significantly the token overhead and inference latency. DVD provides an efficient and general framework for integrating perception capabilities into MLLMs, overcoming the inherent limitations of existing methods.

1 Introduction

Existing MLLM perception representations trade token efficiency against the range and precision needed for 3D tasks. DVD unifies 2D and 3D representations as compact discrete tokens and reports strong benchmark performance with lower token overhead and latency.

1 Introduction

  • Text-based coordinates use many tokens, while fixed-range quantization constrains range and precision, particularly for 3D scenes with theoretically unbounded spatial range.The paper asks whether 2D and 3D representations can achieve unlimited range and high precision with few tokens.Source: §1 · p. 2
  • DVD converts 2D boxes, 2D masks, and 3D boxes into 1D sequences and projects them into a unified high-dimensional codebook of discrete tokens.A 1D encoder-decoder supports joint training across perception tasks.Source: §1 · pp. 2–3
  • Adaptive quantization avoids fixed-range constraints, while a de-tokenizer maps perception tokens in MLLM outputs back to original 2D and 3D representations.The codebook also compresses representations, reducing computational overhead relative to text-based coordinates.Source: §1 · p. 3
  • DVD outperforms existing methods in 2D and 3D grounding, maintains competitive 2D referring-segmentation performance, and reduces token overhead and inference latency.Evaluations cover RefCOCO, RefCOCO+, RefCOCOg, SUN-RGBD, KITTI, Hypersim, and nuScenes.Source: §1 · p. 3
  • A task-agnostic geometric loss supervises decoded geometry rather than sequence parameterization, helping the codebook capture structures shared across heterogeneous 2D and 3D tasks.The loss provides unified geometric supervision for both task types.Source: §1 · pp. 2–3

2 Related Work

Related work covers MLLM development and two approaches to perception: task-specific decoders and text-format prediction. The paper proposes unified adaptive compression and quantization for heterogeneous 2D and 3D representations.

2 Related Work

  • MLLM-based perception uses task-specific decoders [24] [55] [35] [13] [58] [57] [34] [8] [30] or converts perception tasks to text [4] [3] [40] [14] [21] [54] [27] [43].The former lack generality across tasks, while text-format methods are inefficient, especially for 3D scenes.Source: §2 · p. 3
  • Text-based coordinates require many tokens, and fixed-range quantization has precision constraints and is unsuitable for complex 3D scenes with unbounded spatial extent.Rex-Omni [21] uses next-point prediction with quantized coordinates from 0 to 999 for 2D grounding tasks.Source: §2 · p. 4
  • DVD unifies 2D and 3D representations as 1D vector sequences and learns a unified codebook for adaptive compression and quantization.The Figure 2 pipeline maps sequences to a high-dimensional space and decodes discrete tokens back to original representations.Source: §2 · p. 4, Figure 2 · p. 4

3 Method

DVD unifies 2D and 3D perceptual outputs as 1D vector sequences encoded by a shared discrete codebook. A lightweight de-tokenizer reconstructs the original representations from MLLM output tokens.

3.2 Perceptual Representation

  • DVD converts 2D boxes, 2D masks, and 3D boxes into 1D vector sequences for a unified representation.2D boxes use flattened corners; masks use uniformly sampled boundary points normalized by image dimensions; 3D boxes retain center, scale, and rotation values.Source: §3.2 · p. 4

3.3 Dynamic Vector Representation Learning

  • A 1D encoder, learnable codebook, and symmetric decoder map vector sequences to discrete tokens and reconstruct them.The shared codebook supports variable-length sequences across tasks without task-specific modifications.Source: §3.3 · p. 5
  • Reconstruction, VQ, and bidirectional geometric losses train the autoencoder to recover sequences while preserving point-set geometry.Geometric loss uses average minimum L1 distances between predicted and target 3D box vertices or 2D point sets.Source: §3.3 · pp. 5–6

3.4 Dynamic Vector Decoding

  • The MLLM adds codebook tokens to its vocabulary and uses a lightweight de-tokenizer to reconstruct outputs without modifying the backbone.The de-tokenizer reuses the pretrained 1D decoder to convert perception tokens into vector sequences and then original 2D or 3D representations.Source: §3.4 · p. 6

4 Experiments

DVD is evaluated on 2D and 3D perception benchmarks, where it compares favorably with existing methods and trades representation cost against accuracy. Ablations examine its codebook, geometric supervision, compression settings, and joint task training; visualizations show results across grounding and segmentation tasks.

4.2 Implementation Details

  • The unified autoencoder is trained on approximately 1.98M 3D boxes, 2.31M 2D boxes, and 1.52M 2D masks, using a 1:1:1 data ratio.The MLLM is fully fine-tuned for 264K iterations using 2.0M mixed 2D and 3D perception question-answer pairs.Source: §4.2 · pp. 6–7

4.3 Overall Performance

  • 40.0% AP3D on SUN-RGBD, 15.7% AP3D on Hypersim, and 19.8% AP3D on nuScenes let DVD-2B surpass state-of-the-art methods.DVD-8B reaches 40.3%, 18.4%, and 24.4% AP3D on those datasets, respectively.Source: Table 1 · p. 7, §4.3 · p. 7
  • DVD-2B outperforms VST-3B by 2.7% AP3D on SUN-RGBD and 10.7% AP3D on ARKitScenes; DVD-8B reaches 40.3% AP3D on SUN-RGBD and 24.4% on nuScenes.The evaluated 3D scenes include both indoor and outdoor settings.Source: Table 2 · p. 7, §4.3 · p. 7
  • DVD-2B scores 90.0% Precision on RefCOCOg test, 3.2% above Rex-Omni, while DVD-8B reaches 73.4% and 76.6% cIoU on RefCOCOg val and test.On RefCOCOg test, DVD-2B remains competitive with VistaLLM-7B at lower latency.Source: §4.3 · pp. 7–8, Table 3 · p. 8, Table 4 · p. 8

4.4 Ablation Study

  • A 4096-entry codebook achieves the best performance across all three tasks; size 2048 limits representational capacity (88.9% AP3D, 84.5% Precision), while 8192 complicates quantization and learning, degrading reconstruction (85.1% AP3D, 76.8% cIoU).Setting latent hidden size z=16 provides sufficient capacity with stable quantization and the best overall performance. Size z=8 drops to 83.0% AP3D on 3D grounding, while z=32 slightly degrades 3D reconstruction quality (88.4% AP3D).Source: §4.4 · pp. 9–10
  • Geometric guidance improves reconstruction by 10.1% AP3D and 10.0% cIoU, while K=2 halves token count and retains 90.8% AP3D and 100.0% Precision.K=4 causes severe information loss, especially for 2D grounding at 31.5% Precision; joint training improves 3D grounding and 2D RES while leaving 2D grounding stable.Source: Table 6 · p. 9, §4.4 · pp. 9–10, Table 7 · p. 9
  • On SUN-RGBD and RefCOCOg, text coordinates use 9× more tokens and 8.3× the latency of DVD, while special tokens fall to 17.5% AP3D.Text coordinates retain a slight 3D advantage, 37.5% versus DVD’s 34.1% AP3D; DVD offers the best accuracy-efficiency trade-off among the compared representations.Source: Figure 4 · p. 8, §4.4 · p. 9

4.5 Visualization.

  • Qualitative examples show 3D grounding in dark, complex scenes, precise 2D localization and segmentation, and simultaneous 2D/3D perception.Figure 5 presents those examples in four rows: 3D grounding, 2D grounding, 2D referring segmentation, and unified perception.Source: Figure 5 · p. 10, §4.5 · pp. 10–11

5 Conclusion

DVD converts 2D and 3D perceptual outputs into compact discrete tokens to address token overhead and fixed-range quantization constraints. The paper concludes that it achieves strong benchmark performance with reduced token counts and inference latency.

5 Conclusion

  • DVD unifies 2D and 3D perception by mapping their outputs to 1D vector sequences and compact discrete tokens.The conclusion frames this as an efficient, general framework for integrating perception into MLLMs.Source: §5 · p. 11
  • DVD is reported to achieve superior 2D and 3D benchmark performance while significantly reducing token count and inference latency.The conclusion identifies text-coordinate token overhead and fixed-range quantization constraints as the limitations addressed.Source: §5 · p. 11

A Technical Appendices

  • The technical appendices provide further autoencoder implementation details, dataset information, ablations, and visualizations.Source: A Technical Appendices · p. 15

B Implementation Details

The unified autoencoder encodes 1D vector sequences into codebook embeddings and reconstructs the sequences from quantized tokens. Its encoder and decoder use staged downsampling and upsampling, respectively.

B Implementation Details

  • The encoder downsamples 1D vector sequences through ResNet and attention blocks, then projects latent features to the codebook embedding dimension.The decoder reverses this process, projecting quantized tokens to the latent dimension and upsampling to reconstruct the original sequences.Source: B Implementation Details · p. 15
  • Vector quantization assigns each projected latent feature to its nearest entry in a learnable fixed-size codebook.The module uses Euclidean distance and straight-through gradient estimation; the reported codebook has 4096 entries and latent dimension 16.Source: B Implementation Details · p. 15

C Dataset Details

The dataset collection covers indoor and outdoor 3D grounding, standard 2D referring datasets and their 3D projections, and diverse 2D segmentation data. The appendix also identifies dataset, computation-cost, and codebook-utilization tables, plus latent-space visualizations.

C Dataset Details

  • The 3D grounding collection spans six datasets and 489K question-answer pairs, including indoor, synthetic, single-object, and outdoor driving scenes.SUN-RGBD and ARKitScenes cover indoor settings; Hypersim is synthetic, Objectron focuses on single objects, and KITTI and nuScenes cover driving scenes.Source: C Dataset Details · p. 15, Table 10 · p. 15
  • 2D grounding combines three referring datasets with projections of six 3D datasets for 810K question-answer pairs; 2D RES uses six datasets and 774K pairs.The supplied table captions identify additional dataset details, computation-cost comparisons, and codebook utilization; the t-SNE figure depicts latent spaces for 3D boxes and 2D masks.Source: C Dataset Details · p. 15, Table 10 · p. 15, Table 11 · p. 16

D More Ablation Studies

DVD reduces token counts and inference latency compared with text-based coordinate representations while retaining strong perception performance; special tokens reduce computation but constrain range and precision.

D.1 Comparison of different representations

  • DVD uses fewer tokens and achieves much faster inference than text-based methods, which use 512 tokens per mask and 74 per 3D bounding box.The passage reports significant reductions but provides no exact DVD token counts or latency values.Source: §D.1 · p. 16
  • Special tokens reduce computation cost but cause significant performance degradation because of limited range and precision.Compared with special tokens, DVD achieves a significant performance improvement in 3D scenes.Source: §D.1 · p. 16
  • DVD maintains strong perception performance while achieving remarkable efficiency across 3D grounding, 2D grounding, and 2D referring segmentation.Source: §D.1 · p. 16

D.2 Vector Latent Distribution

DVD forms clustered latent representations for perception tasks and uses a near-saturated codebook; qualitative examples show localization and segmentation across 2D and 3D tasks.

D.2 Vector Latent Distribution

  • DVD learns high-quality latent representations, with eight clusters for the eight tokens used to represent one 3D box.Source: §D.2 · p. 16
  • DVD produces three latent clusters in high-dimensional space for 2D and 3D perception tasks.Source: §D.2 · p. 17, Figure 7 · p. 17

D.3 The Codebook Utilization

  • 99.98% of codebook entries are active overall, with task-specific code subsets and no sign of codebook collapse.2D boxes use 72.2% of codes, while fine-grained masks spread over more codes with the highest usage uniformity.Source: §D.3 · p. 17

E Visualization

  • DVD achieves precise 3D localization in indoor and outdoor scenes, including rainy weather and cluttered scenes with dense objects.Source: E Visualization · p. 17
  • DVD outputs tightly aligned 2D boxes and accurately segments objects of varied shapes, including ambiguous boundaries and overlapping objects.Source: E Visualization · p. 17

F.1 Limitations

The authors identify reconstruction quality and semantic alignment as limitations, propose further optimization and broader task support, and note potential privacy risks from surveillance misuse.

F.1 Limitations

  • Reconstruction performance could improve even with a larger downsampling factor, and the separate de-tokenizer may misalign discrete tokens with MLLM semantic understanding.Source: §F.1 · p. 17

F.2 Future Works

  • Future work will optimize vector representation learning, extend the codebook to diverse tasks, and explore end-to-end training of the encoder-decoder and MLLM.Source: §F.2 · p. 19

G Broader Impacts

  • The authors identify possible misuse in closed-circuit television surveillance systems that could affect personal privacy.Source: G Broader Impacts · p. 19
Loading 2610.12266v1…