Source-linked AI summary
Object-Uni: A Unified Model for Object-Centric Spatial Understanding and Controllable Generation
Mining Tan, Yinuo Wang, Ziqi Zhou, Weize Quan, Sifei Li, Jingdong Chen, DanDan Zheng, Libin Wang, Weiming Dong
TL;DR
Unified visual models still struggle to represent continuous object poses and generate geometrically consistent target-view images. Object-Uni unifies pose understanding and controllable generation through explicit geometric pose variables, viewpoint-based orientation abstraction, UniSpatial-80K, and object-token grounding; experiments report stronger object-level pose understanding and controllable generation.
Problem
Existing unified models struggle to precisely represent continuous object poses and generate geometrically consistent images under target viewpoints.
Method
Object-Uni unifies pose perception, spatial reasoning, pose-conditioned generation, and novel view synthesis using explicit pose variables, viewpoint-based orientation abstraction, UniSpatial-80K, and object-token grounding.
Results
Object-Uni achieves stronger overall performance than SceneDesigner across datasets and settings, including improved spatial grounding and pose controllability.
Takeaways & Limitations
Object-Uni provides a unified framework for object-centric spatial understanding and controllable generation under object-level pose constraints.
Takeaways & Limitations
Object-Uni remains limited in generating fine-grained text and human details, partly due to limitations inherited from its generative backbone.
Abstract
from arXiv · showhide
Unified models for visual understanding and generation have made rapid progress, yet they still lack the ability to understand and manipulate the spatial states of object instances. Existing models can describe objects in natural language, but they struggle to precisely represent continuous object poses and generate geometrically consistent images under target viewpoints. To mitigate this, we propose \emph{Object-Uni}, a unified model for object-centric spatial understanding and controllable generation. Specifically, we formulate object-centric spatial intelligence as a unified problem connecting pose perception, spatial reasoning, pose-conditioned generation, and object-centric novel view synthesis. We treat object pose as an explicit geometric variable shared by understanding and generation, rather than merely a prediction label or control signal. To make pose usable by multimodal large language models, we propose a viewpoint-based orientation abstraction that maps orientation into structured viewpoint descriptions while preserving continuous geometric supervision. We further construct an object-centric spatial benchmark (UniSpatial-80K) and train a unified model with an object-token-grounded pose anchor to associate each instance with its pose state. Experiments show that our model improves object-level pose understanding and pose-controllable generation, moving unified models from describing objects toward manipulating spatial states.
1 Introduction
Object-Uni frames object-centric spatial intelligence as a unified understanding-and-generation problem, using object pose as a shared geometric variable. It introduces viewpoint-based orientation descriptions, an 80K+ benchmark, and object-token grounding to connect pose reasoning with controllable generation.
- Object-centric spatial intelligence requires reasoning about object locations, poses, relative relations, pose-conditioned generation, and novel view synthesis.
- Existing methods use flexible language descriptions or separate numerical pose estimation and pose-conditioned generation, lacking stable geometric constraints and a unified formulation.
- UniSpatial-80K contains 80K+ samples spanning spatial relations, viewpoint reasoning, orientation prediction, pose-conditioned generation, and novel view synthesis.
- Object-Uni treats object pose as an explicit geometric variable connecting pose perception, spatial reasoning, and controllable generation.
- Viewpoint-based orientation abstraction converts continuous 3D orientations into structured natural-language viewpoint descriptions for MLLM reasoning.
- An object-token-grounded pose anchor associates each predicted orientation with its corresponding object instance, improving orientation prediction in multi-object scenes.
2 Related Work
Related work includes object pose understanding, spatially controlled generation, and unified multimodal models. Object-Uni differs by making object orientation a shared variable for multimodal reasoning and controllable generation.
- Object Pose Understanding from Images: Prior object-pose benchmarks and models estimate object position or orientation, including multi-axis orientation from single images and relative rotation.
- Unified Models: Object-Uni makes object orientation a shared variable for multimodal reasoning and controllable generation rather than only predicting pose.
- Spatially Controlled Generation: Spatially controlled generation methods explore continuous 3D attributes, subject viewpoint control, and multi-object orientation control.
- Unified Models: Recent unified multimodal models combine visual understanding and generation through varied architectures or training objectives for general multimodal tasks.
3 Object-Centric Understanding and Controllable Generation
Object-Uni unifies object-centric pose understanding, spatial reasoning, pose-conditioned generation, and novel view synthesis through shared geometric representations and instance-grounded pose prediction.
- Viewpoint-based Orientation Abstraction: Object-Uni maps continuous 3D orientations into structured viewpoint descriptions while retaining continuous angles for metric supervision and controllable generation.Azimuth, polar angle, and in-plane rotation are represented both geometrically and linguistically.
- Object-Centric Spatial Representation: The model represents each object with location, size, orientation, image-space position, and pairwise spatial relations.Spatial descriptions include viewer-centric and object-centric relations in multi-object scenes.
- Object-Centric Unified Multimodal Model: Object-Uni uses an object-token-grounded pose anchor to bind azimuth, polar, and rotation predictions to the corresponding object instance.Object-region features are projected into the multimodal language model and used by lightweight pose-prediction heads.
- Pose Prediction: Orientation prediction is trained as soft cross-entropy over discretized azimuth, polar, and rotation distributions, with circular distance handling periodic angles.The loss is averaged over valid objects.
- UniSpatial-80K Task Formulation: UniSpatial-80K organizes interconnected tasks spanning spatial description and orientation prediction, pose-conditioned image generation, spatial reasoning, and novel view synthesis.The benchmark integrates object-level 3D and pose-annotated sources across street-view, indoor, and general scenes.
- Pose-Controllable Generation: Generation uses target object poses as CNOCS maps that provide dense image-space geometric guidance to the diffusion model.The CNOCS map is encoded through the VAE encoder as a generation-side geometric condition.
4 Experiments
The experiments evaluate Object-Uni on object orientation estimation, pose-controllable generation, and object-centric novel view synthesis. Results show stronger orientation awareness, spatial control, and multi-object pose handling, while ablations support the value of object-token grounding.
- Experimental Setup: The evaluation studies orientation estimation, pose-controllable generation, and whether PoseAnchor improves multi-object orientation prediction.Experiments use UniSpatial-80K subsets and compare unified and specialist baselines across single- and multi-object settings.
- Pose Understanding: Object-Uni achieves the best performance across most orientation-estimation datasets, splits, and metrics, with especially strong gains for azimuth.The model is more stable than general-purpose MLLMs in multi-object scenes and remains competitive with specialist orientation models.
- Pose-Controllable Generation: Object-Uni improves Objectron mIoU from 40.77 to 78.23 for single-object generation and from 27.25 to 69.08 for multi-object generation.The generation evaluation measures spatial grounding, pose controllability, and semantic fidelity using localization, angular-error, AUC, and CLIP metrics.
- Pose-Controllable Generation: Object-Uni generally reduces angular errors and improves AUC@10/30 for azimuth, polar angle, and rotation, especially on ImageNet3D and Objectron.The advantage is more pronounced in multi-object scenes, where each target orientation must be associated with its corresponding object instance.
- Ablation Study: Removing explicit object tokens weakens orientation estimation, particularly azimuth prediction in multi-object scenes.The ablation compares Object-Uni with a base model trained on the same tasks without explicit object tokens for orientation prediction.
- Novel View Synthesis: Object-Uni performs object-centric novel view synthesis by editing an object to match a target pose while preserving category, local texture, and surrounding context.The generated result is described as remaining consistent with the input appearance while matching the desired viewpoint.
5 Conclusion
Object-Uni unifies object-centric spatial understanding and controllable generation through object-level pose constraints. The paper reports strong spatial understanding and generation, while identifying fine-grained text and human details as remaining limitations.
- Conclusion: Object-Uni jointly models object semantics, layout, pose, and pose-conditioned generation using object-centric spatial supervision.The framework is built on UniSpatial-80K and targets object-level pose constraints.
- Limitations: The model still has limitations in generating fine-grained text and human details, partly inherited from its generative backbone.These issues are identified as important directions for future improvement.
A Orientation to viewpoint description
The paper converts continuous object orientation into structured viewpoint descriptions for multimodal reasoning. Orientation is represented by azimuth, polar angle, and in-plane rotation, with quantized angles mapped to semantic view terms.
- Orientation Abstraction: Each object orientation is represented by azimuth, polar angle, and in-plane rotation, then quantized into coarse semantic viewpoint descriptions.The abstraction is introduced as intermediate supervision bridging continuous poses and MLLM spatial reasoning.
- Orientation Abstraction: Table 3 specifies the mapping between continuous orientation angles and viewpoint descriptions.Azimuth terms include front, quarter views, side views, and back.
B.1 Dataset statistics.
UniSpatial-80K contains 83,252 images and 91,392 annotated objects from 122 categories. It combines street-view, indoor object-centric, and general rigid-object sources across single- and multi-object scenarios.
- Dataset Statistics: UniSpatial-80K contains 83,252 image entries and 91,392 annotated objects from 122 categories.The dataset averages 1.10 objects per image.
- Dataset Statistics: The dataset integrates street-view, indoor object-centric, and general rigid-object sources for single- and multi-object spatial reasoning.Its category distribution is broad but moderately long-tailed, with the top eight categories comprising 55.16% of object instances.
B.2 Prompt template
The prompt template specifies how to estimate and serialize object orientations using three viewpoint angles: azimuth, polar, and rotation.
- Prompt format: The task asks for orientation estimates only for listed objects, returned in their specified order.Objects are identified by labels and bounding boxes in the image.
- Orientation representation: Azimuth represents the horizontal viewpoint around an object, with 0° front, 90° right, 180° back, and 270° left.Quarter views are mapped to nearby intermediate angles.
- Orientation representation: Polar represents vertical viewpoint and rotation represents in-plane image rotation, each using explicitly defined degree conventions.Polar spans top to bottom viewpoints, while rotation distinguishes upright, clockwise, upside-down, and counter-clockwise orientations.
C Additional results
Figure 4 presents additional results for pose-controllable generation. The results address both single-object and multi-object scenarios.
- Additional results: Additional pose-controllable generation results are presented in Figure 4.The figure provides supplementary visual evidence for the generation capability.
- Additional results: Figure 4 focuses on generation controlled by object pose.Its subject is pose-controllable generation rather than orientation estimation alone.