PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Comprehensive 3D Grounding: Orientation Grounding through Vision-Language Models

Tuo Liang, Disheng Liu, Nengbo Wang, Vipin Chaudhary, Yu Yin

arXiv:2609.33109v1cs.CVcs.AI

TL;DR

Localization grounding usually identifies where a referred object is, while many spatial tasks also require its object-centric orientation and symmetry. The paper introduces orientation grounding, builds ReferOri, and adapts a 3D VLM as OG-VLM; experiments report gains across single-view and multi-view benchmarks over orientation-aware baselines and object-level orientation models.

  • Problem

    Most grounding focuses on object location, although many spatial tasks also require object-centric orientation, which is not captured explicitly by an enclosing box’s pose.

    Source: §1 · p. 1

  • Method

    The paper defines orientation grounding for language or box queries, constructs ReferOri, and adapts a 3D VLM with structured outputs and geometry-aware supervision.

    Source: §1 · p. 2

  • Results

    Experiments show substantial gains over orientation-aware VLMs and object-level orientation foundation models across single-view and multi-view benchmarks.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The results support orientation grounding as a distinct, learnable capability beyond localization and show benefits for orientation-related spatial reasoning.

    Source: §6 · p. 9

    Reconstruction ambiguity can leave residual annotation errors, especially for ambiguous or weakly directional objects, and the VLM remains behind reconstruction-based object-level orientation estimates.

    Source: A LIMITATIONS · p. 13, §B.8 · p. 16

Abstract

from arXiv · show

Grounding is a core capability of spatial vision-language models, yet most existing work focuses only on where a referred object is. Many 3D tasks also require knowing how it is oriented. Although existing 3D VLMs may predict oriented boxes, box pose does not explicitly capture object-centric orientation or symmetry-induced ambiguities. We introduce orientation grounding, a referring grounding task that predicts an object's 6D orientation and axial symmetry from a language or box query in single-view or multi-view scenes. To support this task, we construct ReferOri, with 331K multi-view and 387K single-view orientation-grounding queries obtained through scalable reconstruction, consistency checking, and human verification. We further present OG-VLM, which adapts a 3D VLM with structured box/orientation outputs, sign and symmetry tokens, and geometry-aware auxiliary losses. Across single-view and multi-view benchmarks, OG-VLM substantially outperforms orientation-aware VLM baselines and surpasses object-level orientation foundation models on scene-level referring benchmarks, showing that explicit orientation grounding is a distinct and learnable capability beyond localization. Downstream results validate its benefit for orientation-related spatial reasoning.

1 INTRODUCTION

Most grounding work locates referred objects, but many spatial tasks also need object-centric orientation, which is not equivalent to an enclosing box’s pose. The paper defines orientation grounding and introduces ReferOri and OG-VLM to predict orientation and symmetry alongside object position.

Motivation

  • Localization-only grounding misses object-centric direction, which can depend on part structure, local shape, symmetry, and the environment rather than box geometry.Existing oriented-box supervision couples orientation to box geometry, coordinate conventions, and localization quality.Source: §1 · p. 1

Definition and challenges

  • Orientation grounding predicts a referred object’s 6D orientation and axial symmetry from a language or box query in single-view or multi-view scenes.The formulation addresses scarce labels, symmetry-induced ambiguities, and sign flips or inconsistent binary predictions in autoregressive VLM outputs.Source: §1 · p. 2

Contributions

  • ReferOri provides 331K multi-view and 387K single-view queries, while OG-VLM adds structured box/orientation outputs, sign and symmetry tokens, and geometry-aware heads and losses.The dataset links existing referring data with scalable scene and object reconstruction, multi-view consistency checks, and human verification.Source: §1 · p. 2
  • Experiments show gains over orientation-aware VLM baselines and object-level orientation foundation models, supporting orientation grounding as distinct and learnable beyond localization.Downstream evaluations further show benefits for orientation-related spatial reasoning.Source: §1 · p. 2

2 RELATED WORK

Prior grounding work largely targets object location, while orientation datasets and estimators generally do not address language-conditioned, scene-level referring orientation grounding. This paper places orientation and symmetry in that referring-grounding setting.

2.1 3D REFERRING GROUNDING

  • 3D referring grounding predicts target boxes and has also become an intermediate representation for spatial reasoning and 3D question answering, but mainly captures position and extent.This work retains the referring setting while extending the target to object-centric orientation and symmetry.Source: §2.1 · p. 2

2.2 ORIENTATION DATASET

  • Existing orientation datasets cover object-level geometry, single-view pose, or multi-view boxes and poses, but generally lack the referring orientation-grounding setup.ReferOri constructs orientation and symmetry annotations under referring settings for both single-view and multi-view data.Source: §2.2 · p. 3

2.3 ORIENTATION ESTIMATION

  • Prior orientation estimators focus on CAD alignment, pose estimation, or geometry recovery, while VLM approaches rarely treat scene-level referring orientation as a first-class target.VG-LLM couples rotation to box geometry and Euler-angle parameterization; SOFAR focuses on object-level semantic directions.Source: §2.3 · p. 3

3 METHODOLOGY

The method defines referring orientation grounding over object boxes, object-centric rotations, and axial symmetry, then constructs ReferOri and adapts a 3D VLM with structured outputs and geometry-aware supervision.

3.1 TASK FORMULATION

  • Given a scene and language or box query, orientation grounding predicts the referred object’s 3D box, world-frame object-centric rotation, and canonical-axis symmetry labels.The formulation treats orientations equivalent under object symmetry as ambiguous rather than distinct errors.Source: §3.1 · pp. 3–4

3.2 DATA CONSTRUCTION

  • ReferOri attaches orientation and axial-symmetry labels to RefCOCO single-view data and ScanNet-based multi-view referring data, with human verification and consistency checking.Single-view verification found 96% correctness; an audit of 800 images found 97.5% inter-annotator agreement, while all approximately 10K multi-view object annotations were manually inspected and corrected when needed.Source: §3.2 · p. 4

3.3 OG-VLM OVERVIEW

  • OG-VLM predicts boxes, orientation, and symmetry from single- or multi-view input or point clouds, conditioned on a language or box query.It builds on GS-Reasoner and separates box and orientation outputs with geometry-aware supervision.Source: §3.3 · pp. 4–5

3.3.1 SPECIAL TOKEN DESIGN FOR ORIENTATION GROUNDING

  • Separate <bbox> and <ori> spans, explicit sign tokens, and dedicated symmetry tokens structure numeric outputs and reduce sign-flip errors and unstable free-form binary predictions.The representation factorizes each box and orientation scalar into sign and magnitude.Source: §3.3.1 · p. 5

3.3.2 GEOMETRY-AWARE LOSS FOR GROUNDING

  • Lightweight task-specific heads supplement token-level loss with box, orientation, and symmetry supervision, including axis alignment and mutual orthogonality constraints.At inference, Gram–Schmidt orthogonalization converts predicted axes into a valid rotation matrix.Source: §3.3.2 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate orientation grounding across single-view and multi-view benchmarks, comparing OG-VLM with existing spatial VLMs and orientation foundation models. OG-VLM achieves large gains across both benchmark settings and performs strongly on in-domain and OOD evaluations.

4.1 EXPERIMENTAL SETTINGS

  • The experiments use ReferOri’s 387K single-view and 331K multi-view queries to compare data-only, special-token, and geometry-aware-loss settings.The settings progressively add the special-token representation and then geometry-aware losses.Source: §4.1 · p. 6
  • Evaluation covers single-view and multi-view orientation-grounding benchmarks, with Ori-COCO and ARKitScenes used exclusively for OOD and zero-shot evaluation.The benchmark suite also includes datasets whose orientation annotations were generated by the authors’ pipeline.Source: §4.1 · p. 6
  • Continuous-orientation benchmarks use Acc@15 and Acc@30, scored with a shared symmetry-aware SO(3) geodesic metric after converting native representations to rotation matrices.Ori-COCO instead uses its native eight-sector classification accuracy.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • OG-VLM consistently achieves large gains over existing spatial VLM baselines across single-view and multi-view benchmarks.The comparison indicates that localization-focused grounding and implicit orientation cues do not reliably yield object-centric orientation grounding.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 COMPARISON WITH ORIENTATION FOUNDATION MODEL

  • OG-VLM achieves superior performance across in-domain and OOD benchmarks, including transfer to ARKitScenes after training exclusively on ScanNet-based multi-view data.The authors attribute this advantage to explicit grounding supervision and scene-aware reasoning.Source: §4.3 · p. 7, Table 2 · p. 7

5 ANALYSIS

Axis-wise analysis finds the main improvements in the harder horizontal directions, while symmetry adjustment and localization constraints affect evaluation differently. Downstream evaluations report improved spatial reasoning on 3DSRBench and MSQA without benchmark-specific training.

5.1 AXIS-WISE ERROR ANALYSIS

  • OG-VLM improves the harder X and Y orientation directions, while Z-axis error is smaller and varies little across settings.The authors relate the easier up axis to real-scene objects being approximately upright; structured outputs and geometry-aware supervision improve fine-grained alignment.Source: §5.1 · pp. 7–8, Table 3 · p. 7

5.2 WHAT AFFECTS ORIENTATION GROUNDING?

  • The analysis examines how input depth, object symmetry, localization quality, and the number of views affect orientation grounding.The number-of-views analysis is reported in Appendix D.2.Source: §5.2 · p. 8
  • Removing symmetry adjustment lowers Acc@15 by 2.7 points on ScanRefer∗ and 7.2 points on Multi3DRefer∗, but by less than 0.2 points on the RefCOCO family.Jointly requiring localization correctness causes a larger drop, as localization and orientation errors accumulate; symmetry modeling remains useful for intrinsically ambiguous orientations.Source: §5.2 · p. 8, Table 6 · p. 9

5.3 DOES ORIENTATION GROUNDING BENEFIT DOWNSTREAM SPATIAL REASONING?

  • Without 3DSRBench-specific training, OG-VLM raises overall accuracy on its Orientation subset from 30.0% to 40.0%.The authors report improved MSQA EM@1 and EM@1-Strict as well, suggesting transfer to situated spatial reasoning.Source: §5.3 · p. 9, Figure 4 · p. 9, Table 7 · p. 9
  • The authors characterize orientation grounding as a complementary cue for spatial reasoning, providing a query-conditioned object-centric frame that localization alone cannot recover.They highlight symmetric objects and orientation-dependent referring expressions as cases where valid orientations depend on the query.Source: §5.3 · p. 9

6 CONCLUSION

The paper introduces orientation grounding and ReferOri as a single- and multi-view benchmark and training resource. It reports that explicit supervision, structured outputs, and geometry-aware losses support orientation-grounding gains and downstream orientation-related reasoning.

6 CONCLUSION

  • Orientation grounding extends referring 3D grounding from localization to object-centric orientation and axial symmetry.ReferOri is presented as a scalable single- and multi-view benchmark and training resource.Source: §6 · p. 9
  • The authors report that explicit supervision, structured outputs, and lightweight geometry-aware losses let existing spatial VLMs acquire strong orientation-grounding capability.Experiments show gains over orientation-aware VLMs and object-level orientation foundation models.Source: §6 · p. 9
  • Downstream evaluations show benefits for orientation-related spatial reasoning.The authors identify richer spatial reasoning datasets as a future direction for studying grounding-to-reasoning transfer.Source: §6 · p. 9

AI USE STATEMENT

The authors report that language models were used for language polishing and writing clarity, not for research design, experiments, results, or scientific conclusions.

AI USE STATEMENT

  • Language models were used only to polish language and improve writing clarity.The authors state they were not used for research design, experiments, results, or scientific conclusions.Source: AI USE STATEMENT · p. 13
  • The authors state that all assisted text was reviewed by the authors.Source: AI USE STATEMENT · p. 13
  • The statement excludes research design, experiments, results, and scientific conclusions from language-model use.Source: AI USE STATEMENT · p. 13

A LIMITATIONS

The appendix documents the ReferOri construction and validation pipeline alongside limits on annotation quality and evaluation scope. Reconstruction-based labels are checked through consistency procedures and human review, but residual noise remains.

A LIMITATIONS

  • Residual annotation noise persists; single-view object estimators remain stronger, base-model capability retention is unevaluated, and embodied manipulation and navigation remain untested.The authors also report residual reliance on upright-scene and language priors.Source: A LIMITATIONS · p. 13

B DATA CONSTRUCTION DETAILS; B.1 SINGLE-VIEW CONSTRUCTION PIPELINE

  • Single-view labels start from referring boxes, filter blurred images, reconstruct metric scene and object geometry, and transform the object's canonical frame into world orientation.The canonical coordinate system of the reconstructed object serves as an orientation proxy.Source: §B.1 · pp. 13–14

B.2 MULTI-VIEW CONSTRUCTION PIPELINE

  • For multi-view labels, the pipeline selects the two largest visible views, estimates orientation independently, and sends estimates differing by at least 5° for human correction.Estimates below the 5° threshold are accepted directly, reducing manual effort.Source: §B.2 · p. 14

B.3 SYMMETRY ESTIMATION

  • Axial symmetry labels come from point-cloud reflection consistency, but estimates can become noisy with severe reconstruction artifacts, heavy occlusion, or near-symmetric shapes.The method tests symmetry along each canonical axis.Source: §B.3 · p. 14

B.4 COORDINATE ALIGNMENT AND SANITY CHECKS

  • Cross-module checks inspect reconstructed 3D scenes as well as images, helping detect coordinate mismatches such as left-right inversion or an incorrect up-axis.The authors report these sanity checks are important when combining independently trained reconstruction modules.Source: §B.4 · pp. 14–15

B.5 ANNOTATION MODULE VALIDATION; B.6 HUMAN VERIFICATION PROTOCOL

  • SAM3D reaches 96.3% Ori-COCO accuracy; human review judges 96% correct, and the independent 800-image audit finds 97.5% overall agreement.The human-verified correctness rate averages five rounds of 100 sampled annotations.Source: §B.5 · p. 15, §B.6 · p. 15, Table 8 · p. 15

B.7 ADVANTAGES OF THE PIPELINE

  • The pipeline reuses existing referring data and reconstruction modules to scale annotation without exhaustive manual labeling, and can incorporate improved reconstruction models.It supports both single-view and multi-view settings.Source: §B.7 · p. 16

B.8 FAILURE CASES AND LIMITATIONS

The pipeline's annotations have bounded noise, with reconstruction ambiguity, segmentation, occlusion, and cross-module inconsistencies as sources of error. The benchmark also standardizes orientation evaluation by converting outputs to a shared frame and metric.

B.8 FAILURE CASES AND LIMITATIONS

  • The canonical reconstruction frame may diverge from human-defined intrinsic orientation, especially for symmetric or weakly directional objects; segmentation, occlusion, and cross-module errors add bounded residual noise.The authors characterize the annotations as scalable supervision rather than perfectly noise-free labels.Source: §B.8 · p. 16

C.1 UNIFIED ROTATION CONVERSION

  • Methods' native outputs, including OG-VLM's 6D rotation orthogonalized via Gram–Schmidt, are mapped to a shared right-handed 3 x 3 frame and evaluated with one SO(3) geodesic error.Source: §C.1 · p. 16

C.2 SYMMETRY-AWARE EVALUATION

This section evaluates orientation with symmetry-aware metrics, defines benchmark-specific splits and protocols, and reports sensitivity metrics for image-plane rotation.

C.2 SYMMETRY-AWARE EVALUATION

  • Symmetry adjustment yields dataset-dependent improvements, but modest changes on most benchmarks indicate orientation performance is not primarily driven by symmetry handling.OG-VLM uses predicted axial symmetry labels for equivalence sets; baselines without compatible predictions use the identity set, so ground-truth symmetry never relaxes errors.Source: §C.2 · p. 17

C.3 BENCHMARK SPLITS AND METRICS

  • RefCOCO-family datasets, ScanRefer, and Multi3DRefer use converted test splits, while Ori-COCO and ARKitScenes use their specified existing benchmarks.Ori-COCO reports eight-sector classification accuracy rather than Acc@15/Acc@30; continuous benchmarks use a common symmetry-aware SO(3) geodesic metric.Source: §C.3 · p. 17

C.4 DOWNSTREAM EVALUATION PROTOCOL

  • 3DSRBench evaluates three Orientation relations with CircularEval and reports per-relation and aggregate accuracy; MSQA reports EM@1 and EM@1-Strict.Neither model receives benchmark-specific QA supervision or task-specific fine-tuning on these benchmarks.Source: §C.4 · p. 17

D.1 SENSITIVITY TO IMAGE-PLANE ROTATION

  • Table 10 reports axis-wise and overall mean orientation errors in degrees and Acc@15/Acc@30 in percent for image-plane rotation sensitivity.The supplied caption does not state the table's outcome values.Source: Table 10 · p. 18

D.2 INPUT FRAMES ABLATION

The ablations examine how visual input relates to orientation grounding, how grounded orientation connects to spatial reasoning, and how orientation supervision relates to localization and qualitative predictions.

D.2 INPUT FRAMES ABLATION

  • Performance drops from multi-view to single-view and further to language-only input, indicating that orientation grounding depends strongly on visual and geometric evidence.Multi-view cues help resolve occlusion and structural ambiguity, while a single view provides limited local appearance and shape information.Source: §D.2 · p. 18, Table 11 · p. 18
  • Language-only orientation predictions mainly reflect pretrained category- and scene-level priors, which are insufficient for robust grounding in ambiguous or non-canonical cases.The ablation quantifies the additional benefit of visual evidence, especially multi-view observations, rather than seeking to eliminate language priors.Source: §D.2 · p. 18

D.3 DOWNSTREAM SPATIAL REASONING ANALYSIS

  • Consistent gains across In front of, On the left, and Viewpoint suggest orientation grounding benefits multiple forms of orientation-dependent spatial reasoning.Viewpoint additionally requires relating the object's intrinsic frame to the camera frame, making it a more challenging reference-frame transformation.Source: §D.3 · p. 18

D.4 LOCALIZATION RESULTS

  • Localization results test whether explicit orientation supervision preserves the base model's localization capability; Table 12 covers Acc25% on five referring benchmarks.The supplied passages describe the evaluation purpose and table scope, not its results.Source: §D.4 · p. 18, Table 12 · p. 19

D.5 QUALITATIVE COMPARISON WITH BASELINES

  • On RefCOCO+ and ScanNet, OG-VLM predictions align better with ground-truth orientations than SpatialReasoner and VG-LLM, respectively, across single- and multi-view settings.Figure 5 encodes RefCOCO+ ground truth in gray, OG-VLM predictions in orange, and SpatialReasoner predictions in red; ScanNet uses solid ground-truth and dashed prediction axes.Source: §D.5 · p. 18, Figure 5 · p. 19

E TRAINING AND GEOMETRY-AWARE LOSS DETAILS

The appendix details ReferOri construction, OG-VLM prediction heads, geometry-aware losses, and implementation settings for training.

E.1 DATASET CONSTRUCTION AND TRAINING DETAILS

  • ReferOri uses existing referring datasets to create language and box queries, yielding 331K multi-view and 387K single-view queries at an approximately 1:1 ratio between query types.Training uses GS-Reasoner as the base model, with full runs taking approximately 120 hours on 4×H200 GPUs.Source: §E.1 · p. 20

E.2 PREDICTION HEADS

  • Lightweight heads decode boxes, continuous 6D orientation, and symmetry from task-specific hidden states, decoupling localization and orientation supervision.The box and orientation heads use the </bbox> and </ori> token states, respectively; the decoupling reduces interference between the tasks.Source: §E.2 · p. 20

E.3 DETAILED BOX LOSS

  • The box loss has three terms: 3D generalized IoU, relative scale discrepancy, and normalized center discrepancy.The passages identify the predicted and ground-truth boxes and the smallest enclosing box used for generalized IoU.Source: §E.3 · pp. 20–21

E.4 DETAILED ORIENTATION LOSS

  • The orientation head regresses two 3D vectors, orthogonalizes them into predicted right and front axes, and uses alignment and orthogonality losses.The alignment exponent preserves optimization sensitivity at low error, while the orthogonality term penalizes non-orthogonal predictions and stabilizes the recovered basis.Source: §E.4 · p. 21

E.5 ADAPTIVE EXPONENT SCHEDULE

  • The alignment exponent decreases across predefined loss thresholds, shifting optimization from coarse orientation alignment toward fine-grained directional refinement.The schedule uses exponents 1, 1/2, and 1/4 as alignment loss crosses the specified thresholds.Source: §E.5 · p. 21

E.6 DETAILED SYMMETRY LOSS

  • A cosine-based symmetry loss regularizes the predicted symmetry vector in continuous feature space, complementing token-level discrete symmetry decoding.The loss compares predicted and ground-truth symmetry vectors.Source: §E.6 · p. 22

E.7 HYPERPARAMETERS AND IMPLEMENTATION DETAILS

  • Default box-loss coefficients are 0.5, 0.25, and 0.25, while orientation alignment and orthogonality coefficients are 0.8 and 0.2.Overall box, orientation, and symmetry weights are linearly warmed up; after step 2000, tokens inside <bbox> and <ori> spans receive language-modeling weight 3 versus 1 elsewhere.Source: §E.7 · p. 22

Related papers

Loading 2609.33109v1…