PaperlayerView exact revision on arXiv

Source-linked AI summary

Token Warping Helps MLLMs Look from Nearby Viewpoints

Phillip Y. Lee, Chanho Park, Mingue Park, Seungwoo Yoo, Juil Koo, Minhyuk Sung

arXiv:2604.02870v1cs.CV

TL;DR

MLLMs remain fragile when reasoning about nearby viewpoints, even with depth-based or 3D-aware methods. The paper tests token-level viewpoint transformations, comparing forward and backward warping, and finds backward token warping most robust on ViewBench. The approach consistently outperforms the evaluated baselines while requiring minimal inference-time warping computation.

  • Problem

    MLLMs struggle to reason about viewpoint transformations despite depth estimation, explicit 3D supervision, and 3D-aware features.

  • Method

    The paper warps ViT-based image tokens under viewpoint changes and compares forward warping with backward warping using nearest or adaptive fetching.

  • Results

    Backward token warping consistently outperforms other variants and evaluated baselines on ViewBench, including pixel-wise, specialist fine-tuned, and generative warping methods.

  • Takeaways & Limitations

    A regular, dense target-view token grid is crucial for robust MLLM viewpoint reasoning, while simple nearest fetching performs comparably to adaptive fetching.

  • Takeaways & Limitations

    The method assumes a source-view depth map and camera intrinsics and uses either ground-truth or estimated depth.

Abstract

from arXiv · show

Can warping tokens, rather than pixels, help multimodal large language models (MLLMs) understand how a scene appears from a nearby viewpoint? While MLLMs perform well on visual reasoning, they remain fragile to viewpoint changes, as pixel-wise warping is highly sensitive to small depth errors and often introduces geometric distortions. Drawing on theories of mental imagery that posit part-level structural representations as the basis for human perspective transformation, we examine whether image tokens in ViT-based MLLMs serve as an effective substrate for viewpoint changes. We compare forward and backward warping, finding that backward token warping, which defines a dense grid on the target view and retrieves a corresponding source-view token for each grid point, achieves greater stability and better preserves semantic coherence under viewpoint shifts. Experiments on our proposed ViewBench benchmark demonstrate that token-level warping enables MLLMs to reason reliably from nearby viewpoints, consistently outperforming all baselines including pixel-wise warping approaches, spatially fine-tuned MLLMs, and a generative warping method.

1. Introduction

MLLMs remain limited in viewpoint-aware spatial reasoning despite depth and 3D-aware methods. Motivated by part-level mental imagery and token robustness, the paper investigates token warping and finds backward token warping superior on ViewBench.

  • Depth estimation and explicit 3D supervision have produced little improvement in MLLMs’ ability to describe scenes from different viewpoints.
  • Object-centric transformations can support object-level relational reasoning but often lose fine-grained details and overall spatial coherence.
  • Mental-imagery theories motivate using image tokens as part-level structural units for viewpoint transformation.
  • Token-position perturbation experiments test whether MLLMs remain stable when patches are retrieved from slightly displaced centers.
  • Backward warping maps target-view grid centers back to the input image, whereas forward warping maps source-view tokens toward the target.
  • Backward token warping outperforms other variants on ViewBench and surpasses specialist spatially fine-tuned MLLMs and generative warping.

2. Related Work

Prior work finds that MLLMs remain limited in spatial and viewpoint-aware reasoning, while token representations offer a potential basis for more robust viewpoint perception. Pixel-wise warping can distort local geometry and degrade semantics, motivating token warping as a lightweight alternative.

  • MLLMs often struggle with basic spatial understanding despite their potential for real-world embodied tasks.
  • MLLMs are largely confined to the input camera’s viewpoint and struggle to adopt another person’s or object’s vantage point.
  • ViT-based models use patch-wise tokens as semantic primitives supporting local detail and global context across vision and multimodal tasks.
  • Pixel-wise warping often introduces local distortions and semantic degradation, affecting both forward and backward transformations.The source-view book appears significantly distorted after transformation.
  • This work proposes token warping as a lightweight, robust strategy for viewpoint-aware perception by treating tokens as primary semantic units in MLLMs.

3. Token Warping for Viewpoint Changes

The paper treats image tokens as part-level perceptual units for viewpoint transformation and evaluates how token warping should retrieve and arrange them. It finds that token representations tolerate positional noise, while backward warping and simple nearest fetching preserve coherent target-view inputs.

  • 3.1. Image Tokenization in MLLMs: ViT-based MLLMs encode images as localized, semantically meaningful tokens derived from non-overlapping patches and patch-center coordinates.Each patch is embedded and processed with positional information before projection into the LLM’s latent space.
  • 3.1. Image Tokenization in MLLMs: Token-level transformations are investigated as a finer-grained alternative to object-level representations for maintaining spatial and appearance details during viewpoint changes.The approach is motivated by part-level mental imagery representations and aims to preserve scene coherence under geometric noise.
  • 3.3. Designing Token Warping Functions: Token warping retrieves intact source-view tokens, whereas pixel-wise warping followed by patchification introduces local distortions that degrade MLLM understanding.The figure contrasts direct token retrieval with pixel reconstruction and subsequent patchification.
  • 3.2. Fetching Position Noise Sensitivity Test: Token representations remain stable under increasingly large fetching-position perturbations, showing only mild degradation at 19.0–20.0 pixels and greater robustness than pixel-level representations.The perturbation experiment varies maximum displacement from 0.0 to 20.0 while smoothing over 9 grid cells.
  • 3.3. Designing Token Warping Functions: Viewpoint warping uses source-image depth, camera intrinsics, and relative pose to transform token positions or retrieve source tokens for target-view coordinates.Forward warping projects source tokens into the target view, whereas backward warping maps target grid centers back to the source image plane.
  • 3.3. Designing Token Warping Functions: Backward warping preserves a dense, regular target-view grid, avoiding the irregular sparse distributions produced by forward projection.The paper compares nearest and adaptive fetching; nearest fetching performs comparably despite requiring less computation than adaptive re-patchification.

4. ViewBench

ViewBench evaluates whether MLLMs can reason about spatial relationships and describe objects from nearby target viewpoints using source–target image pairs. Its tasks, metrics, and examples cover text- and shape-based spatial reasoning alongside target-view object description.

  • 4. ViewBench: ViewBench measures MLLMs’ ability to imagine alternative viewpoints while transferring fine-grained details from an observed source view.The benchmark is built around source–target viewpoint pairs with overlapping fields of view.
  • Data: The benchmark uses adjacent-viewpoint pairs from real-world ScanNet scans, stratified by overlap ratios and assigned source and target poses.Questions are generated to require information available from the target viewpoint.
  • Tasks: ViewBench includes view-conditioned spatial reasoning and target-view object description tasks.Spatial reasoning is instantiated with text labels or simple geometric shapes whose left-right relationship reverses after viewpoint change.
  • Examples: The benchmark’s examples present source–target images, questions, and answers for Text, Shape, and Object evaluations of nearby-viewpoint understanding.These examples illustrate both reversed spatial relationships and object properties visible in the warped target view.
  • Metrics: Spatial reasoning is evaluated with accuracy, while target-view object descriptions receive average 1–10 ratings from Qwen2.5-14B.Oracle evaluation uses 571 Text pairs, 744 Shape pairs, and 300 Object pairs.

5. Evaluation

ViewBench evaluates warping choices for viewpoint-aware spatial reasoning and target-view description. Backward token warping consistently outperforms forward token warping and competing pixel-wise, specialized-MLLM, and generative baselines while avoiding severe pixel-warping artifacts.

  • Evaluation setup: ViewBench evaluates view-conditioned spatial reasoning and target-view object description across token-warping, pixel-wise, specialized-MLM, and generative-warping baselines.The framework uses Qwen2.5-VL-7B as its base MLLM and adds minimal inference-time warping overhead without extra fine-tuning.
  • View-conditioned spatial reasoning: Backward token warping consistently outperforms forward token warping across overlap ratios in ViewBench-Text and ViewBench-Shape.With ground-truth depth in the most challenging 5–15 overlap setting, Backward-Nearest improves accuracy by 14.57%p on ViewBench-Text and 12.4%p on ViewBench-Shape.
  • Baseline comparison: Token-wise warping consistently outperforms pixel-wise, specialized-MLLM, and generative-warping baselines.VLM-3R, despite incorporating CUT3R features, remains behind backward token warping.
  • Qualitative analysis: Pixel-wise warping introduces severe visual artifacts, whereas token-warping visualizations are pixelated only for display because the framework operates on token embeddings.The qualitative comparison attributes the visible artifacts to the pixel-wise baselines’ warped images entering the vision encoder.
  • Target-view object description: Backward token-warping approaches also outperform forward token warping, pixel-wise baselines, and the generative baseline on ViewBench-Object.The comparison uses higher scores from the MLLM evaluator for target-view object descriptions.

6. Conclusion

The paper explores token warping as a simple strategy for transferring source-view observations to nearby novel viewpoints. It finds that backward warping with a regular, dense target-view token grid is crucial for robust performance, while nearest fetching is comparable to adaptive fetching.

  • Conclusion: Token warping transfers source-view observations to nearby novel viewpoints using part-based image-token representations inspired by mental-imagery theories.The paper compares forward and backward warping directions and adaptive versus nearest backward fetching.
  • Conclusion: Backward warping is crucial because it constructs a regular, dense grid of tokens for robust MLLM performance.This design preserves regularly spaced target-view tokens during viewpoint transformation.
  • Conclusion: Nearest fetching performs comparably to adaptive fetching, providing a practical and efficient solution.The conclusion characterizes nearest fetching as simpler while retaining comparable performance.

Supplementary Material

The supplementary material extends baseline comparisons and examines robustness, extreme viewpoint conditions, geometry-based analyses, and qualitative examples. Additional results show backward token warping remains strongest across expanded baselines, while a 3D-feature model exhibits degraded behavior.

  • Supplementary scope: The supplement adds experiments on specialized MLLMs, estimated geometry, extreme viewpoint shifts and occlusion, geometry-based oracle analysis, and qualitative examples.It also documents backward-token-warping implementation details and the ViewBench data-construction pipeline.
  • Additional baselines: The extended comparison includes general-purpose MLLMs, spatially fine-tuned models, concurrent spatial-reasoning systems, and 3D-geometry-encoder baselines.The listed baselines include Qwen3-VL, InternVL3, VST variants, SpatialLadder, VG-LLM, and related systems.
  • Results: Backward-Nearest and Backward-Adaptive achieve the best performance on both ViewBench-Text and ViewBench-Shape, outperforming all newly added baselines.General MLLMs, MindCube, and SpatialLadder still struggle to internally shift viewpoint or underperform backward token warping.
  • Results: VG-LLM frequently outputs multiple-choice labels such as “A” or “B” when prompted to answer with “left” or “right.”The authors hypothesize that VGGT-based fine-tuning may have compromised the base MLLM’s general capabilities, whereas token warping leaves the underlying MLLM unchanged.

A.2. Robustness Analysis on Estimated Geometry

The robustness analysis tests token warping with estimated depth and jointly estimated depth and pose rather than ground-truth geometry. Token warping retains an advantage over pixel-wise warping and continues to outperform no-warping under estimated geometry.

  • Setup: The framework computes backward warping from a depth map and relative camera pose, then replaces ground-truth geometry with predictions from off-the-shelf models.The analysis evaluates estimated geometry on ViewBench-Shape.
  • Depth estimation: 65.84% with DA-V2 and 67.74% with DP versus 70.99% with GT depth shows that adaptive token warping remains effective with estimated depth.These values are reported for ViewBench-Shape, averaged across overlap levels.
  • Depth estimation: 60.49% with DA-V2 and 62.76% with DP versus 62.35% with GT depth shows the corresponding decline for pixel-wise backward warping.Despite the decline, estimated-geometry warping substantially outperforms the no-warping baseline, and token warping retains its advantage over pixel-wise warping.
  • Joint depth and pose estimation: 68.95% with VGGT-estimated geometry versus 63.58% for pixel-wise warping demonstrates a token-warping advantage when both depth and pose are estimated.With DUSt3R, both methods decline further, but token warping still outperforms pixel-wise warping.

A.3. Larger Viewpoint Shifts and Occlusion

Stress tests show that token warping remains advantageous under extremely low view overlap and full target-view occlusion. A geometry-only oracle further indicates that the warping pipeline is highly accurate, with residual errors linked to depth noise and projection edge cases.

  • Larger Viewpoint Shifts: 65.08% with GT depth and 66.14% with estimated depth, backward token warping with adaptive fetching outperforms pixel-wise backward warping and no warping at 2–5% overlap.Pixel-wise backward warping achieves 61.90% / 61.38%, while the no-warping baseline achieves 34.39%.
  • Larger Viewpoint Shifts: The consistent performance ordering across overlap levels supports backward token warping as robust to very large viewpoint shifts.The stress-test pairs share only 2–5% of visible scene content.
  • Occlusion: 46% accuracy with GT depth, compared with 38% for pixel-wise warping and 32% for base Qwen2.5-VL, under full target-view occlusion.The occlusion evaluation uses 50 ProcTHOR-based pairs with GT depth.
  • Occlusion: Despite lower absolute accuracy under full occlusion, token warping preserves the relative advantage over pixel-wise warping and the base model.The evaluation targets cases where a source-visible object becomes fully occluded in the target view.
  • Geometry-Based Oracle: Above 93% accuracy across all overlap levels on ViewBench-Text and ViewBench-Shape, achieved by the geometry-based oracle.The oracle directly compares warped source keypoint x-coordinates rather than querying the MLLM.
  • Geometry-Based Oracle: The oracle’s small gap from 100% is attributed to depth noise near object boundaries and projections whose target-view x-coordinates are nearly identical.These cases explain residual geometric errors independently of MLLM reasoning.

A.5. Additional Qualitative Results

Additional qualitative examples visualize how token and pixel warping preserve or distort spatial information during viewpoint changes. The examples cover spatial relationships, object ordering, and object descriptions from target viewpoints.

  • Comparison Setup: The qualitative comparisons provide source images, depth maps, relative camera poses, and intrinsics for single-view VQA under viewpoint changes.The visualizations compare backward token warping with pixel-wise and forward token warping baselines.
  • Qualitative Samples: Sample 1 asks for the target-view relationship between a photo frame and a pillow while visualizing each token’s source-image position by color.The color coding preserves source coordinates after warping.
  • Qualitative Samples: Sample 2 asks for the target-view left-to-right order of toys and illustrates local pixel distortions under pixel-wise backward warping.Source-token colors indicate where warped tokens originated.
  • Qualitative Samples: Samples 3 and 4 ask target-view object-description questions involving an omelet and a white mug, respectively, while showing distortions from pixel-wise forward warping.The examples use color-coded tokens to reveal source locations after warping.

B. Implementation Details

The implementation constructs a target-view grid, maps each target location backward through a depth-derived source proxy, and fetches source tokens at the mapped coordinates. It supports nearest and adaptive fetching, with adaptive fetching requiring re-patchification rather than the original fixed-grid partitioning.

  • Target Grid: A regular target patch grid of size l × l yields M = (HW)/l^2 patches, with one token assigned to each target-grid center.The implementation assumes the target and source images have the same resolution and that H and W are divisible by l.
  • Source Proxy from Depth: The source proxy is a lightweight 3D triangle mesh built by unprojecting depth-associated source pixels with the camera intrinsics and triangulating each 2×2 cell.The mesh is formed in the source camera frame.
  • Backward Mapping via Ray Casting: For every target-grid center, ray casting intersects the target-camera ray with the source proxy and projects the hit point back into the source image.The relative target-to-source pose and perspective projection produce the backward-warped source coordinate.
  • Backward Mapping: Invalid target-to-source intersections, such as those caused by occlusion or field-of-view mismatch, are marked invalid and their corresponding patches are omitted.Valid mapped coordinates are collected across all target-grid centers before token fetching.
  • Nearest Fetching: Nearest fetching reuses the fixed source patch whose center is closest to each backward-warped coordinate, preserving efficient original-grid patchification despite a small positional mismatch.The mismatch occurs because mapped coordinates generally do not coincide with existing source-grid centers.
  • Adaptive Fetching: Adaptive fetching re-patchifies the source image so each patch is centered exactly at its backward-warped coordinate, more faithfully following the mapping.This approach incurs the cost of re-patchifying rather than using the original efficient fixed-grid partitioning.

C.1. Benchmark Construction

ViewBench is built from ScanNet RGB-D scenes and controlled source–target view pairs, using visibility and overlap to construct viewpoint-dependent questions and object-description samples. An LLM evaluator scores the open-ended object-description responses against oracle target-view descriptions.

  • ViewBench uses ScanNet frames with RGB images, depth maps, camera extrinsics, and camera intrinsics.
  • Visibility is defined by projections falling inside the frame and not being occluded according to the depth map.
  • Pairs are retained at approximately 5–35% visible-point overlap and sampled across overlap bins to reduce bias toward small overlaps.
  • ViewBench-Text and ViewBench-Shape mark corresponding co-visible 3D points with labels or geometric symbols, then ask target-view left–right questions whose relation flips across views.
  • The benchmark contains 571 text questions, 744 shape questions, and 300 object-description samples validated with target-view oracles and co-visibility constraints.
Loading 2604.02870v1…