Source-linked AI summary
SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation
Zekun Qi, Wenyao Zhang, Yufei Ding, Runpei Dong, Xinqiang Yu, Jingwen Li, Lingyun Xu, Baoyu Li, Xialin He, Guofan Fan, Jiazhao Zhang, Jiawei He, Jiayuan Gu, Xin Jin, Kaisheng Ma, Zhizheng Zhang, He Wang, Li Yi
TL;DR
Current spatial reasoning methods often overlook object orientation, despite its importance for fine-grained 6-DoF manipulation. The paper introduces language-grounded semantic orientation, OrienText300K, PointSO, and SOFAR to support orientation-aware scene reasoning and robotic actions. SOFAR outperforms baselines across simulated and real-world tasks, although decoupled execution remains vulnerable to sub-module errors.
Problem
Existing VLMs struggle to translate language descriptions into object orientations needed for 6-DoF manipulation.
Method
The paper defines semantic orientation, constructs OrienText300K, develops PointSO, and integrates it with segmentation and VLM reasoning in SOFAR.
Results
SOFAR significantly outperforms state-of-the-art VLMs and VLA models across simulated and real-world tasks.
Takeaways & Limitations
Semantic orientation provides open-vocabulary, template-free orientation understanding for unseen objects and supports orientation-aware robotic manipulation.
Takeaways & Limitations
Decoupled SOFAR execution may fail when sub-modules produce unstable grasps or inaccurate visual perception.
Abstract
from arXiv · showhide
While spatial reasoning has made progress in object localization relationships, it often overlooks object orientation-a key factor in 6-DoF fine-grained manipulation. Traditional pose representations rely on pre-defined frames or templates, limiting generalization and semantic grounding. In this paper, we introduce the concept of semantic orientation, which defines object orientations using natural language in a reference-frame-free manner (e.g., the "plug-in" direction of a USB or the "handle" direction of a cup). To support this, we construct OrienText300K, a large-scale dataset of 3D objects annotated with semantic orientations, and develop PointSO, a general model for zero-shot semantic orientation prediction. By integrating semantic orientation into VLM agents, our SoFar framework enables 6-DoF spatial reasoning and generates robotic actions. Extensive experiments demonstrated the effectiveness and generalization of our SoFar, e.g., zero-shot 48.7% successful rate on Open6DOR and zero-shot 74.9% successful rate on SIMPLER-Env.
1 Introduction
The paper argues that orientation is essential for 6-DoF manipulation but remains difficult for VLMs to connect with language. It introduces semantic orientation, PointSO, OrienText300K, and SOFAR to support orientation-aware spatial reasoning and robotic manipulation.
- Motivation: 6-DoF manipulation requires orientation awareness in addition to object position relationships.Examples include blade, glass-top, and insertion directions.
- Core concept: Semantic orientation links object orientation vectors to open-vocabulary descriptions such as “handle” and “plug-in” directions.This replaces reliance on fixed reference frames or template models.
- Resources: PointSO predicts semantic orientations from 3D objects and language, while OrienText300K provides over 350K 3D models with diverse orientation-text pairs.The dataset and model are designed for open-world orientation understanding.
- System: SOFAR combines PointSO with foundation models to build an orientation-aware 3D scene graph for spatial reasoning and manipulation actions.RGB-D input is segmented, oriented, and passed with the image to a VLM.
- Evaluation: SOFAR significantly outperforms state-of-the-art VLMs and VLA models across simulated and real-world tasks.The paper also introduces Open6DOR V2 and 6-DoF SpatialBench for evaluation.
- Conclusion: Semantic orientation enables open-vocabulary, template-free orientation understanding for unseen objects and supports generalization across embodiments and tasks without task-specific fine-tuning.The reported scope includes manipulation, navigation, and visual question answering.
2 Semantic Orientation: Connecting Language and Object Orientation
Semantic orientation represents a language-described object direction as a reference-free unit vector, enabling multiple task-relevant orientations per object. The paper supports prediction with OrienText300K and PointSO, then uses these predictions in SOFAR for scene-level reasoning.
- Definition: Semantic orientation is a unit vector capturing the direction corresponding to a language description for an object.Descriptions can refer to directions, object parts, or interactions.
- Definition: An object can have multiple semantic orientations by varying the language input, forming a semantic basis for describing and transforming its rotation.Examples include “front,” “handle,” and “plug-in.”
- Dataset: OrienText300K is built from Objaverse 3D models using filtering and language-guided annotation to provide diverse orientation labels.GPT-4o is used to filter large-scale data and support annotation.
- Dataset validation: GPT-4o achieves 88.3% accuracy on filtering and 97.1% accuracy on annotation in a 208-sample validation set.These results are reported as validation of OrienText300K annotation quality.
- PointSO: PointSO takes an object’s 3D point cloud and a language description as inputs and predicts the corresponding semantic orientation.Its architecture uses cross-modal 3D-language fusion.
- SOFAR: SOFAR combines segmentation, PointSO orientation estimation, and VLM reasoning to construct an orientation-aware 3D scene graph.The graph supports positional and orientational planning for downstream manipulation.
3 SOFAR: Semantic Orientation Bridges Spatial Reasoning and Object Manipulation
SOFAR combines semantic orientation estimation with a 6-DoF scene graph and VLM reasoning to plan object manipulation from RGB-D inputs. Its pipeline extracts task-relevant objects, predicts semantic orientations, reasons over desired transformations, and executes collision-free motions.
- Scene-level reasoning: SOFAR integrates PointSO and SAM with a VLM to extend object-centric orientation understanding into scene-level spatial reasoning.The system uses foundation models to segment scenes, estimate orientations, and support positional and orientational planning.
- Position and orientation extraction: Given a language query, the system extracts relevant object phrases, segments their 3D point clouds, and predicts task-specific semantic orientations.PointSO infers a semantic orientation set for each related object after language-conditioned segmentation.
- 6-DoF scene graph: The 6-DoF scene graph represents object identity, centroid position, bounding-box size, semantic orientations, and pairwise spatial relations.Graph nodes encode semantic and spatial attributes, while edges capture relative translation and size ratios.
- VLM reasoning: The VLM receives the encoded scene graph alongside the RGB image and query, enabling joint positional and orientational reasoning.This enriched representation supports chain-of-thought planning for transformations of rigid objects.
- Motion execution: SOFAR computes translation from target-minus-initial position and estimates rotation from semantic orientation correspondences before generating collision-free trajectories.Low-level execution combines grasp generation, transformation planning, and OMPL-based motion planning.
4 Experiments
Experiments evaluate SOFAR on semantic orientation prediction, simulated and real-world 6-DoF manipulation, navigation, and spatial VQA. SOFAR consistently outperforms baselines, with especially strong performance on orientation and 6-DoF tasks and robust orientation prediction under fine angular thresholds.
- Evaluation setup: 60 real-world tasks involving over 100 objects are organized into position, orientation, and comprehensive 6-DoF tracks with simple and hard variants.The orientation track tests part-level orientation and fine-grained angle estimation.
- Manipulation: SOFAR consistently outperforms baselines across real-world tracks, especially on orientation and 6-DoF tasks, while maintaining low planning overhead.The system also generalizes across end-effectors including dexterous hands and suction cups.
- Semantic orientation prediction: 60% accuracy at a 5° threshold demonstrates PointSO’s semantic orientation prediction capability on the OrienText300K validation split.The evaluation uses 128 manually annotated objects and angular thresholds from 45° to 5°.
- Robustness: PointSO is evaluated under random rotations, partial single-sided observations, and Gaussian noise to measure robustness to incomplete or corrupted point clouds.Table 3 reports accuracy at the 45° threshold for individual and combined corruptions.
- Navigation: Orientation-aware navigation requires the robot to reach both the correct position and the appropriate facing direction before manipulation.The evaluation uses a quadruped robot and emphasizes access to an object’s functional side.
- Spatial reasoning benchmarks: 6-DoF SpatialBench evaluates positional and orientational comprehension with 223 human-annotated RGB-image multiple-choice samples.The benchmark provides fine-grained quantitative annotations rather than relying only on coarse positional or relative metrics.
5 Limitations & Conclusions
The paper concludes that semantic orientation, OrienText300K, PointSO, and SOFAR support open-vocabulary 6-DoF reasoning across robotic settings. It also identifies execution failures from unstable grasping or inaccurate visual perception as a limitation of the decoupled system.
- Limitation: Execution may fail when a decoupled sub-module introduces transformation errors through unstable grasping or inaccurate visual perception.Such errors can place target objects in unexpected poses during execution.
- Contributions: Semantic orientation links object orientations with intuitive language descriptors, while OrienText300K and PointSO support semantic orientation understanding.The conclusion presents these components as foundations for SOFAR’s simulated and real-world manipulation results.
- Experimental platforms: SOFAR is evaluated with Franka Panda, UR, Flexiv, dexterous-hand, suction-tool, and quadruped setups across manipulation and navigation experiments.The reported setups use different robot arms, end-effectors, cameras, and navigation platforms.
- Articulated manipulation: SOFAR’s articulated-object evaluation uses the PartNet-Mobility Dataset in SAPIEN without fine-tuning on the SAPIEN samples.The setup uses SAM segmentation centers, PointSO contact directions, and VLM planning.
B.1 Articulated Objects Manipulation Evaluation
SOFAR is further evaluated for articulated-object manipulation, spatial VQA, dexterous-hand manipulation, and long-horizon tasks. These experiments emphasize broad task and embodiment coverage, including decomposition of complex instructions into subtasks.
- Articulated objects: SOFAR evaluates articulated-object manipulation in SAPIEN using PartNet-Mobility and directly uses SAM segmentation centers with PointSO-generated contact directions.A VLM determines subsequent motion directions under the stated evaluation setup.
- Spatial VQA: SOFAR achieves over 20% improvement in overall performance on EmbSpatial-Bench relative to all baseline methods.The result is presented as evidence of stronger spatial understanding in complex visual scenes.
- Cross-embodiment manipulation: SOFAR plans grasp poses and target transformations without trajectory data specific to a robotic arm, supporting embodiment-agnostic manipulation.Experiments include dexterous hands with position-related and rotation-related tasks.
- Long-horizon tasks: VLMs decompose long-horizon instructions such as making breakfast and cleaning a desktop into executable subtasks.The reported experiment includes uncommon objects such as Aladdin’s lamp and puppets.
B.5 Close-Loop Execution Experiment
SOFAR supports close-loop 6-DoF manipulation across varied objects, views, and instructions. PointSO’s in-the-wild evaluation and SOFAR’s cross-view experiments indicate generalization beyond fixed camera perspectives.
- Close-loop execution: Figure 13 presents SOFAR’s close-loop execution pipeline.
- In-the-wild PointSO evaluation: PointSO achieves good semantic-orientation performance across different views, objects, and instructions in in-the-wild evaluation.Objects are segmented with Florence-2 and SAM before single-sided point clouds are passed to PointSO.
- Cross-view generalization: SOFAR obtains world-coordinate point clouds from an RGB-D camera to generate grasping poses without requiring a fixed camera perspective.PointSO also uses random camera views during training to augment partial point clouds.
B.8 Failure Case Distribution Analysis
The failure analysis attributes SOFAR’s real-world failures primarily to grasping, semantic-orientation prediction, and object analysis or detection. The analysis also identifies motion-planning and task-planning issues as additional failure sources.
- Analysis figures: Figure 15 presents SOFAR’s cross-view generalization, while Figure 16 presents its failure-case distribution analysis.
- Failure sources: 31% of failures originated from grasping issues, including small objects, unreasonable grasping poses, and instability after grasping.
- Failure sources: 23% of failures were due to incorrect or inaccurate Semantic Orientation prediction.Upright or upside-down tasks require angle estimation below 5° for smooth execution.
- Failure sources: Approximately 20% of errors involved object analysis and detection.The passage attributes these errors to instability in open-vocabulary detection modules for out-of-distribution objects or parts.
C.1 Semantic Orientation Ablation
The ablation studies examine whether semantic orientation, chain-of-thought reasoning, data scale, multimodal fusion, and perception modules contribute to SOFAR and PointSO performance. The supplied evidence reports data volume as the most significant factor driving PointSO improvement and addition as the best fusion method.
- Semantic orientation ablation: The SOFAR ablation varies semantic orientation in the scene graph and chain-of-thought reasoning to distinguish their methodological contributions.
- Multimodal fusion: Simple addition achieves the best performance among the tested PointSO multimodal fusion methods.The comparison includes cross-attention, multiplication, addition, and concatenation.
- Open-vocabulary detection: Florence-2 achieves the best performance among the compared detection modules on Open6DOR perception tasks.The comparison includes Florence-2, Grounding DINO, and YOLO-World.
- Experimental setting: The real-world experiments use more than 100 diverse objects in 6-DoF rearrangement experiments.
D.1 Detail Real World Experiment Results
The real-world evaluation is designed to test SOFAR across diverse objects, task types, difficulty levels, and model configurations. The pipeline combines open-vocabulary detection, scene-graph processing, visual instruction tuning, and 6-DoF manipulation training data.
- Evaluation design: SOFAR’s real-world evaluation includes 60 tasks covering more than 100 diverse objects.Tasks are divided into position-track, orientation-track, and comprehensive & 6-DoF-track categories, each with simple and hard levels.
- PointSO implementation: PointSO uses FPS and KNN patchification, a PointNet patch encoder, a Transformer encoder, and a linear layer mapping outputs to a three-dimensional vector space.
- Results: The real-world results are reported in a detailed zero-shot 6-DoF rearrangement-results table.
- SOFAR-LLaVA: SOFAR-LLaVA receives a JSON-formatted 6-DoF scene graph and a refined image for supervised fine-tuning of an LLM.The training data includes approximately 3,000 6-DoF object-manipulation instructions.
- SpatialBench: The study also introduces 6-DoF SpatialBench for orientation-aware spatial visual question answering.The benchmark contains diverse internet images and questions involving one or multiple objects with varying uncertainty in image resolution.
E.2 Open6DOR V2 Analysis
Open6DOR V2 is presented as a revised benchmark for 6-DoF object rearrangement, with task statistics and visual examples supporting analysis of spatial reasoning and manipulation. The surrounding discussion connects this benchmark to semantic orientation, scene-level reasoning, and future extensions.
- Benchmark revisions: The benchmark removes incorrectly labeled data and manual evaluation metrics while integrating assets and metrics into Libero for closed-loop policy evaluation.
- Benchmark structure: Open6DOR V2 comprises three independent tracks with diverse tasks and carefully annotated instruction levels.
- Manipulation example: SOFAR predicts target and relative-object coordinates and produces grasp poses for a “move near” task in SIMPLER.
- Spatial reasoning evaluation: 6-DoF SpatialBench includes quantitative and qualitative questions across indoor and outdoor scenarios, including motion-based reasoning problems.
2. Limitations
The supplied material records a dedicated limitations discussion and emphasizes that claims, assumptions, scope, and reproducibility should be explicitly documented. It also states that detailed experimental information is provided for reproduction.
- Limitations discussion: The paper identifies Section 5 as the location where limitations are discussed.
- Assumptions and scope: The limitations guidance calls for stating strong assumptions, robustness to violations, and the scope of empirical claims.
- Reproducibility: The paper states that detailed experiment information is provided in Section D for reproducing the main results.
- Open materials: The paper states that data and code are provided in the supplemental material.
- Experimental details: Training and test details, including hyperparameters, are reported as necessary to understand and reproduce the results.
7. Experiment statistical significance
The paper reports error bars for the real-world experiment in Fig. 7 and states that detailed computer resources are provided for reproduction. It also documents ethical standards, societal impacts, asset licensing, and new-asset documentation.
- Error bars are reported for the real-world experiment depicted in Fig. 7.
- Detailed computer resources are provided in Section D to support reproducibility.
- The paper states that it ensured anonymity and ethical standards.
- Potential positive and negative societal impacts are discussed in Section I.
- The paper states that public baselines and datasets are credited and that documentation for new assets appears in the supplementary materials.