PaperlayerView exact revision on arXiv

Source-linked AI summary

Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution

Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, Yongming Rao

arXiv:2409.12961v4cs.CV

TL;DR

Existing MLLMs commonly standardize diverse visual inputs despite differing spatial, temporal, and task requirements. Oryx combines native-resolution OryxViT, on-demand dynamic compression, and specialized training, achieving strong performance across image, video, and 3D benchmarks.

  • Problem

    Existing MLLMs often treat diverse visual inputs uniformly, although visual sources and tasks require different resolutions, formats, and context lengths.

  • Method

    Oryx combines native-resolution OryxViT, Dynamic Compression, and curated training for long-context retrieval and spatial-aware multimodal understanding.

  • Results

    Oryx achieves outstanding performance across image, video, and 3D benchmarks, including competitive 7B video results and 32B results surpassing models up to 72B.

  • Takeaways & Limitations

    A unified on-demand framework can support spatial-temporal understanding across varied visual resolutions, temporal lengths, and modalities.

  • Takeaways & Limitations

    The paper notes that native or arbitrary-resolution processing in MLLMs remains comparatively underexplored, and motivates its approach partly from this assumption.

Abstract

from arXiv · show

Visual data comes in various forms, ranging from small icons of just a few pixels to long videos spanning hours. Existing multi-modal LLMs usually standardize these diverse visual inputs to a fixed resolution for visual encoders and yield similar numbers of tokens for LLMs. This approach is non-optimal for multimodal understanding and inefficient for processing inputs with long and short visual contents. To solve the problem, we propose Oryx, a unified multimodal architecture for the spatial-temporal understanding of images, videos, and multi-view 3D scenes. Oryx offers an on-demand solution to seamlessly and efficiently process visual inputs with arbitrary spatial sizes and temporal lengths through two core innovations: 1) a pre-trained OryxViT model that can encode images at any resolution into LLM-friendly visual representations; 2) a dynamic compressor module that supports 1x to 16x compression on visual tokens by request. These design features enable Oryx to accommodate extremely long visual contexts, such as videos, with lower resolution and high compression while maintaining high recognition precision for tasks like document understanding with native resolution and no compression. Beyond the architectural improvements, enhanced data curation and specialized training on long-context retrieval and spatial-aware data help Oryx achieve strong capabilities in image, video, and 3D multimodal understanding simultaneously. Our work is open-sourced at https://github.com/Oryx-mllm/Oryx.

1 INTRODUCTION

MLLMs need a unified way to handle increasingly diverse images, videos, and 3D data without treating every visual input identically. Oryx addresses this with native-resolution encoding, on-demand compression, and joint training for spatial-temporal understanding.

  • MLLMs increasingly need to process diverse images, videos, and 3D data across more challenging multimodal tasks.
  • Existing approaches often treat visual inputs uniformly, overlooking differences in content, resolution, task requirements, and source.
  • Native-resolution, on-demand processing can preserve image information while matching resolution and compression to task requirements.The paper links high resolution to text-relevant tasks and greater compression to long-video processing.
  • Oryx combines OryxViT for native-resolution visual representations, Dynamic Compression with 1x–16x compression, and enhanced training strategies.
  • Oryx achieves strong spatial and temporal understanding across image, video, and multi-view 3D benchmarks, with 32B results competitive against models up to 72B.

2 RELATED WORK

Related work develops visual encoders and multimodal systems for increasingly diverse contexts, but existing approaches retain limitations in flexibility and breadth of supported inputs.

  • MLLMs use attention-based query mechanisms, MLP connectors, or combinations of visual encoders to align visual features with language models.
  • Recent systems extend MLLMs from image understanding to video, long-form video, 3D environments, and interleaved multi-image inputs.

3 METHODS

Oryx uses native-resolution visual encoding, dynamic token compression, and specialized multimodal training to process images, videos, and 3D inputs with varied spatial and temporal demands.

  • 3 METHODS: The Oryx method has two components: an architecture for native and on-demand visual inputs, and a data curation and training pipeline.
  • 3.1 ARCHITECTURE: Fixed resizing and cropping can damage visual content, motivating representations that preserve original aspect ratios and resolutions.
  • 3.1.1 VISUAL REPRESENTATIONS WITH NATIVE RESOLUTION: OryxViT uses adaptive position embeddings and variable-length self-attention to process visual inputs with different sizes in batch mode.
  • 3.1.1 VISUAL REPRESENTATIONS WITH NATIVE RESOLUTION: Variable-length attention maintains a forward speed comparable to conventional fixed-resolution visual encoders.
  • 3.1.2 ON-DEMAND DYNAMIC COMPRESSION SUPPORTING LONG VISUAL CONTEXT: The Dynamic Compressor applies higher compression to longer contexts and unifies different compression ratios into a consistent visual-token pattern.
  • 3.1.2 ON-DEMAND DYNAMIC COMPRESSION SUPPORTING LONG VISUAL CONTEXT: Cross-attention lets low-resolution features interact with neighboring high-resolution patches, while a shared MLP maps compressed features into the language-model embedding space.
  • 3.2 DATA CURATION & TRAINING: Training includes long-form video data and needle-in-a-haystack tasks that retrieve or distinguish designated frames within extended movie sequences.
  • 3.2 DATA CURATION & TRAINING: Spatial-aware training addresses the absence of temporal or trajectory cues in multi-view 3D images, which can hinder spatial understanding.

4 EXPERIMENTS

Oryx is evaluated across general and long-form video, image, 3D spatial, and architectural ablation benchmarks. The experiments show strong performance across these settings and support native-resolution processing and dynamic compression as effective design choices.

  • General Temporal Understanding: Oryx achieves competitive general video understanding, reaching 64.2 and 74.9 mean accuracy on VideoMME with subtitles for its 7B and 32B variants.It also surpasses previous state-of-the-art results by 4.8% on NextQA and 7.1% on Perception Test.
  • Long-Form Temporal Understanding: Oryx-1.5-7B surpasses existing 7B model series on long-video benchmarks, while Oryx-1.5-32B improves over prior 72B state-of-the-art models by 5.9% on MLVU and 0.7% on LongVideoBench.The 32B model also outperforms GPT-4o on MLVU by 7.7%.
  • 2D & 3D Spatial Understanding: Oryx maintains strong image-understanding performance, including 86.3% mean accuracy on MMBench and 92.7% accuracy on DocVQA.The evaluated image benchmarks cover both general understanding and document, OCR, diagram, and text-specific tasks.
  • 2D & 3D Spatial Understanding: Oryx outperforms prior specialized 3D models, general open-source MLLMs, and a 3D-LLM on ScanQA validation-set spatial understanding.The evaluation compares Oryx with both 3D-specific and general multimodal baselines.
  • Analysis: Native-resolution inputs benefit all evaluated benchmarks, while text-relative benchmarks are more sensitive to resolution scale.Figure 3 compares fixed-size, native-size, and larger native-resolution inputs across varying resolutions.
  • Analysis: The ablations find that OryxViT improves visual-text alignment and that dynamic compression outperforms MLP connectors, with average pooling best among tested downsampling methods.The compression analysis attributes the average-pooling advantage to its parameter-free preservation of visual-feature distributions.

5 CONCLUSION

Oryx is presented as a unified multimodal framework for on-demand spatial-temporal understanding across diverse visual inputs, tasks, resolutions, and temporal lengths.

  • Oryx handles diverse visual inputs across varying tasks, temporal lengths, and resolutions in an on-demand manner.

A GENERATION RESULTS

Oryx demonstrates video capabilities spanning detailed summarization, reasoning, skill learning, and 3D spatial understanding. The examples show accurate outputs across these tasks, including reasoning with coarse correspondences.

  • Oryx generates comprehensive video captions that preserve essential information and accurately identify match details, player names, and player status.
  • Oryx answers video questions through analogy and generates well-reasoned responses.
  • Oryx learns to use Google Scholar to cite a paper by following the steps shown in a video.The example highlights potential for agent-based tasks and task execution.
  • Oryx improves 3D comprehension through coarse correspondences and generates accurate reasoning outcomes.In a challenging first-person-view direction task, LLaVA-OneVision provides incorrect conclusions.

B FAILURE CASES

The reported failure cases concern temporal perception and knowledge-based reasoning, while additional experiments examine video summarization, reasoning, skill learning, 3D understanding, and long-context retrieval. Oryx retrieves inserted images at up to 1.6k frames, but uniform sampling and incomplete knowledge remain limitations.

  • Generation Results: Oryx generates comprehensive video summaries and detailed captions that capture the main event while preserving essential information.
  • Generation Results: Oryx can reason through input videos, learn useful skills from demonstrations, and understand 3D spatial information through coarse correspondences.
  • Failure Cases: Uniform frame sampling can cause video models to lose temporal context, leading Oryx to misidentify timestamps and answer time-perception questions incorrectly.The paper suggests incorporating timestamps or avoiding frame sampling as possible directions.
  • Failure Cases: Oryx lacks knowledge of chemical reactions and faces challenges integrating chemical knowledge into a video’s main idea.The paper identifies knowledge-base expansion and stronger reasoning as possible directions.
  • Long-Context Retrieval: Oryx successfully retrieves inserted images and answers their questions with frame counts of 1.6k, whereas baselines trained with 32 frames fail to identify them.

C.2 INFERENCE SPEED AND EFFICIENCY

OryxViT provides native-resolution visual encoding with efficiency comparable to dynamic-partition approaches. Its memory cost grows approximately linearly with common image resolutions, while dynamic compression can further reduce visual-token memory.

  • Variable-length attention keeps inference throughput comparable while its memory overhead remains negligible after CUDA-kernel optimization.
  • OryxViT is only 7% slower than SigLIP with dynamic partitioning on 1280 × 1280 images.
  • 49.1G versus 48.7G: OryxViT and the dynamic-partition baseline show comparable maximum memory costs under the tested setting.
  • Memory cost increases with image resolution and is approximately linear among common resolutions.
  • Dynamic compression such as 2x or 4x downsampling can largely save memory, unlike previous methods that do not support this feature.

C.3 ABLATIONS ON TRAINING DATA

The ablations show that long-form temporal data benefits long-video benchmarks, while 3D data and coarse correspondence improve 3D understanding. The shared MLP design also avoids the video-performance degradation associated with separate image and video projectors.

  • Effects on Long-Form Data: Long-form temporal data benefits performance on long-video benchmarks.The ablation evaluates its effects on VideoMME and MLVU.
  • Effects on 3D-relevant Data: 3D data and coarse correspondence both contribute to improved understanding on 3D benchmarks.The study reports the coarse-correspondence approach as effective for 3D tasks and evaluates it on ScanQA.
  • MLP Adapter Design: Separate image and video MLPs negatively impact video benchmarks compared with a shared MLP.The authors attribute this to differing feature distributions for similar data under the dual-projector design.

C.5 ANALYSIS ON DOWNSAMPLING

The Dynamic Compressor outperforms conventional downsampling approaches, while fixed-length Q-former methods are unsuitable for increasingly long visual content. Within the compressor, average pooling performs best among the tested downsampling functions.

  • Overall Downsampling Architecture: The Dynamic Compressor outperforms average pooling and spatial convolution for downsampling visual content.The comparison references VideoMME and MLVU results.
  • Overall Downsampling Architecture: Fixed-length Q-former methods are unsuitable for long visual content because visual-token information capacity depends on token length.The authors identify longer visual content as a more complex case that fixed token lengths cannot adequately handle.
  • Downsampling Function: Average pooling outperforms DWConv and Conv-MLP within the Dynamic Compressor.The authors suggest that average pooling better preserves visual-feature distributions and avoids training difficulties affecting more complex layers.

D.1 IMPLEMENTATION DETAILS

Oryx combines arbitrary-resolution visual encoding with staged multimodal training across image, temporal, and 3D data. Its implementation uses multiple language models, pretraining, and supervised fine-tuning stages with progressively broader data and context requirements.

  • Model Configuration: Oryx integrates Qwen-2-7B and Yi-1.5-34B, while Oryx-1.5 uses Qwen-2.5-7B and Qwen-2.5-32B language models.The visual encoder is the pretrained OryxViT, which supports arbitrary-resolution visual inputs.
  • Pretraining: Pretraining uses 558k captioning examples while unfreezing the Dynamic Compressor module.The captioning data comes from LLaVA-1.5.
  • Pretraining: OryxViT is pretrained with Qwen2-0.5B, followed by LoRA fine-tuning of the language models.The implementation reports 0.6B trainable parameters and approximately 10 times faster training than main-stage supervised fine-tuning.
  • Stage 1: Stage 1 trains only the connector on 558k image-caption pairs while keeping the other parameters fixed.The stage uses a total batch size of 256 and a learning rate of 1e-3.
  • Stage 2: Stage 2 adds approximately 650k temporal and 3D examples to 600k image examples, producing 1.2M supervised fine-tuning examples.The image pixel restriction is increased to 15362 to accommodate longer temporal sequences.
Loading 2409.12961v4…