PaperlayerView exact revision on arXiv

Source-linked AI summary

ORIGEN: Zero-Shot 3D Orientation Grounding in Text-to-Image Generation

Yunhong Min, Daehyeon Choi, Kyeongmin Yeo, Jihyun Lee, Minhyuk Sung

arXiv:2503.22194v3cs.CVcs.LG

TL;DR

Existing spatial grounding mainly controls 2D positioning, leaving 3D orientation grounding open. ORIGEN uses test-time, reward-guided Langevin sampling with a pretrained orientation discriminator, and reports stronger results than prior methods while supporting multi-object grounding.

  • Problem

    Image-generation spatial grounding has mainly addressed 2D positioning, leaving 3D orientation control an open challenge.

    Source: §1 · p. 2

  • Method

    ORIGEN guides a one-step text-to-image model at test time using an orientation reward and Langevin-based sampling with reward-adaptive time rescaling.

    Source: §1 · p. 2

  • Results

    ORIGEN outperforms fine-tuning-based baselines and orientation-conditioned models, and reports gains in benchmark and user-study comparisons.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    ORIGEN uniquely supports conditioning on multiple open-vocabulary objects across diverse categories.

    Source: §5 · p. 9

    Performance is inherently bounded by the quality of the pretrained discriminator.

    Source: §5 · p. 10

Abstract

from arXiv · show

We introduce ORIGEN, the first zero-shot method for 3D orientation grounding in text-to-image generation across multiple objects and diverse categories. While previous work on spatial grounding in image generation has mainly focused on 2D positioning, it lacks control over 3D orientation. To address this, we propose a reward-guided sampling approach using a pretrained discriminative model for 3D orientation estimation and a one-step text-to-image generative flow model. While gradient-ascent-based optimization is a natural choice for reward-based guidance, it struggles to maintain image realism. Instead, we adopt a sampling-based approach using Langevin dynamics, which extends gradient ascent by simply injecting random noise--requiring just a single additional line of code. Additionally, we introduce adaptive time rescaling based on the reward function to accelerate convergence. Our experiments show that ORIGEN outperforms both training-based and test-time guidance methods across quantitative metrics and user studies.

1 Introduction

Prior spatial grounding mainly addressed 2D positioning, leaving 3D orientation control open; ORIGEN introduces zero-shot multi-object grounding with reward-guided sampling and reports gains over existing methods.

1 Introduction

  • Prior spatial-grounding work mainly addressed 2D positioning, while 3D orientation grounding remained largely unexplored and an open conditioning challenge.Orientation defines an object's spatial pose [10] [11] [13] [14] [16].Source: §1 · pp. 1–2
  • Existing orientation-conditioned methods have constrained scope: relative control, single-object settings, limited azimuths, or synthetic-data training.Zero-1-to-3 [17] requires a reference image, Continuous 3D Words [18] supports single-object images and half-front azimuth, and OrientDream [19] supports four primitive azimuths for single objects.Source: §1 · p. 2
  • ORIGEN introduces zero-shot orientation grounding for multiple objects across diverse categories, using test-time guidance from a pretrained 3D orientation discriminator and a one-step text-to-image model.The method searches for a latent whose generated image receives a high reward from the discriminator.Source: §1 · p. 2
  • Langevin-based sampling adds random noise to reward-driven gradient ascent to balance reward maximization with adherence to the prior; reward-adaptive timestep rescaling accelerates convergence.The Euler–Maruyama update is described as implementable by adding a single line of code.Source: §1 · p. 2
  • ORIGEN significantly outperforms prior orientation-conditioned image generative models [18] [17] on the benchmark and user studies; comparisons were conducted under single-object conditioning because prior models cannot condition on multiple objects.Source: §1 · p. 2

2 Related Work

Prior work controls global viewpoints or individual-object orientation, while training-free and training-based guidance methods face limitations in reward estimation, realism, data availability, or computational cost.

2 Related Work

  • Global-viewpoint methods [26] [27] do not individually control foreground-object orientation, while orientation-specific methods [17] [18] [19] rely on single-object synthetic training images that limit generalization.Depth conditioning [28] [29] [30] and 3D bounding boxes [31] also do not directly control orientation; boxes can have front-back ambiguities.Source: §2 · pp. 2–3
  • Training-free diffusion guidance methods [32] [33] can rely on blurry posterior means for future-reward estimates, while straightened distilled trajectories [38] [39] [40] [41] [42] yield insufficiently small intermediate image updates.Initial-noise optimization methods [22] address intermediate-noise limitations, but gradient ascent can encounter local optima and realism deviations even with heuristic regularization.Source: §2 · p. 3
  • Training-based control methods such as ControlNet [28] and IP-Adapter [47] require training data, while reward fine-tuning [48? –51] uses substantial computation and can degrade image quality and diversity.The paper therefore proposes test-time reward-guided generation using a discriminative foundational model.Source: §2 · p. 3

3 ORIGEN

ORIGEN is a zero-shot method for grounding 3D orientations of multiple objects across open-vocabulary categories. It guides latent sampling with an orientation reward, Langevin dynamics, and reward-adaptive time rescaling.

3.1 Problem Definition and Overview

  • ORIGEN uses test-time guidance with a one-step text-to-image model to ground orientations for multiple objects across open-vocabulary categories.The method uses GroundingDINO for object detection and OrientAnything for orientation estimation to form its reward.Source: §3 · p. 3, §3.1 · pp. 3–4
  • The method samples latent vectors from a target distribution that maximizes expected orientation reward while keeping the distribution close to the Gaussian prior.The objective’s regularization strength is controlled by α.Source: §3.1 · p. 4

3.2 Multi-Object Orientation Grounding Reward

  • The orientation reward compares predicted and target distributions over one-degree bins for each Euler angle, summing negative KL divergence across angles and objects.GroundingDINO detects the described objects, and OrientAnything estimates their orientations.Source: §3.2 · pp. 4–5

3.3 Reward-Guided Langevin Dynamics

  • Reward-guided Langevin dynamics adds stochastic exploration to latent sampling; its stationary distribution coincides with the objective’s optimal distribution.The discrete update adds noise and a scaling factor to gradient ascent, while explicitly keeping samples near the prior.Source: §3.3 · p. 5

3.4 Reward-Adaptive Time Rescaling

  • Reward-adaptive time rescaling assigns smaller steps at high reward and larger steps at low reward to improve convergence speed and accuracy.The monitor function’s experimental parameters are smin = 1/3, smax = 4/3, and k = 0.3.Source: §3.4 · p. 6
  • In the toy experiment, ORIGEN aligns with the target distribution while ReNO fails to capture it, and reward-adaptive time rescaling further accelerates convergence.All methods use the same total number of sampling steps.Source: §3.4 · p. 6, Figure 2 · p. 6

4 Experiments

The experiments introduce ORIBENCH for single- and multi-object orientation grounding and evaluate ORIGEN against training-based and guided-generation baselines. ORIGEN is reported to outperform the baselines in orientation grounding, receive the strongest user preference, and reach the stated accuracy threshold in fewer iterations than the full run.

4.1 Datasets

  • ORIBENCH uses MS-COCO-derived prompts, images, and boxes; its single-object split has 1K samples across 25 classes, and its multi-object split has 371 samples.The single-object split filters out unclear orientations and captions without explicit object references, then mixes captions and orientations; multi-object prompts combine object phrases.Source: §4.1 · pp. 6–7

4.2 Evaluation Metrics

  • Orientation alignment is evaluated with Acc.@22.5° and absolute azimuth error, while CLIP Score, VQA-Score, and PickScore measure text-to-image alignment.OrientAnything estimates orientations from generated images; a user study is also used to validate the results.Source: §4.2 · pp. 7–8

4.5 Results

  • ORIGEN significantly outperforms baselines in single-object orientation alignment with comparable text-to-image alignment, and consistently outperforms guided-generation baselines on multi-object grounding.The qualitative comparisons also report the most realistic images and best alignment with grounding conditions on both benchmarks.Source: Figure 3 · p. 7, Figure 4 · p. 7, Table 1 · p. 8

4.3 Baselines

  • Comparisons include orientation-to-image training methods, multi-step guided-generation methods, one-step ReNO, and ORIGEN without reward-adaptive time rescaling.The multi-step baselines are DPS, MPGD, and FreeDoM; the training-based baselines are C3DW and Zero-1-to-3.Source: §4.3 · p. 8

4.4 Implementation Details

  • ORIGEN and ReNO use FLUX-Schnell, while the multi-step guidance baselines use FLUX-Dev; non-training-based methods are matched at 50 function evaluations.Experiments were run on a single NVIDIA 48GB VRAM A6000 GPU.Source: §4.4 · p. 8

4.6 User Study

  • In a user study with 100 Amazon Mechanical Turk participants, ORIGEN was preferred by 58.18% and significantly outperformed Zero-1-to-3 and C3DW.Guided-generation baselines were excluded because ORIGEN outperformed them in the quantitative comparison.Source: Table 2 · p. 9, §4.6 · p. 9

4.7 Computation Time

  • The average run takes approximately 52.7 seconds per image, while cases reaching ±22.5° accuracy averaged 12.8 iterations and 13.5 seconds.The method may also serve as a post-refinement method for existing models, potentially reducing computation time further.Source: §4.7 · p. 9

5 Conclusion

ORIGEN provides zero-shot 3D orientation grounding across multiple objects and categories using Langevin-based sampling with reward-adaptive time rescaling. It outperforms fine-tuning-based baselines, supports conditioning on multiple open-vocabulary objects, and is bounded by the quality of its pretrained discriminator.

5 Conclusion

  • ORIGEN outperforms fine-tuning-based baselines and uniquely supports conditioning on multiple open-vocabulary objects.The method is presented as the first 3D orientation grounding method for text-to-image generation across multiple objects and categories.Source: §5 · p. 9
  • Langevin-based sampling with reward-adaptive time rescaling enables test-time guidance with a pretrained discriminator and faster convergence.Source: §5 · p. 9
  • Performance is inherently bounded by the quality of the pretrained discriminator.The authors also note that their method achieved state-of-the-art results by a significant margin.Source: §5 · p. 10
  • As a generative AI technique, ORIGEN may be misused to produce inappropriate or harmful content.The authors underscore responsible development and deployment.Source: §5 · p. 10

A Analysis on Norm-Based Regularization

The analysis characterizes norm-based regularization as mode-seeking and describes how it can lead to overoptimization and local maxima. It connects the update to deterministic gradient flow, which may drift from the original latent distribution and converge to suboptimal solutions.

A Analysis on Norm-Based Regularization

B Proofs

The appendix analyzes reward-adaptive time rescaling: direct rescaling changes the target stationary distribution, while a correction drift preserves it and improves convergence speed.

B Proofs

  • The proof invokes the overdamped Langevin stationary-distribution result and identifies q(x) as a standard Gaussian.These statements establish the distributional basis used in the proof.Source: B Proofs · pp. 14–15

C.1 Non-Convergence of Direct Time-Rescaled SDE

  • Direct time rescaling with monitor G(x) does not preserve q*: the rescaled Fokker–Planck operator does not annihilate the desired distribution.The analysis concludes that the rescaled SDE therefore does not converge to the target distribution.Source: §C.1 · pp. 15–16

C.2 Time-Rescaled SDE with Correction Drift Term

  • Adding a correction drift of 1/2∇G makes the rescaled process preserve the original invariant distribution.The corrected Fokker–Planck operator annihilates q*, leaving it in the operator’s kernel.Source: §C.2 · p. 16
  • 12.8 iterations with ORIGEN versus 14.2 with ORIGEN* were required to reach the desired reward level.ORIGEN* omits adaptive time rescaling; the reported speedup did not degrade image quality, and the added computational overhead was negligible.Source: §C.2 · p. 16, Table 3 · p. 16

D Setup for the Toy Experiment

The toy experiment uses a distilled rectified-flow model on a two-dimensional Gaussian-mixture task, while the benchmark class list retains objects with clear orientation cues.

D Setup for the Toy Experiment

  • The toy task trains a rectified-flow model from a standard-Gaussian source toward a mixture of two Gaussians, then distills the model twice.The resulting model is a 3-rectified-flow model.Source: D Setup for the Toy Experiment · pp. 16–17
  • All methods in the toy comparison use the same total number of sampling steps.This controls the sampling-step budget across methods.Source: D Setup for the Toy Experiment · p. 17

E Selected Classes

  • The benchmark selects 25 of MS-COCO’s 80 validation classes, excluding ambiguous orientations and retaining classes with clear orientation cues.The detailed selected-class list is provided in Table 4.Source: E Selected Classes · p. 17

F Implementation Details

Implementation uses FLUX-Schnell with OrientAnything for ORIGEN, compares guided-generation baselines under matched evaluation budgets, and adds a prompt adjustment for back-facing cases.

Evaluation Settings

  • Image quality was assessed with CLIP Score, VQA-Score, and Pickscore, using their specified CLIP-ViT-L-14-336, LLaVA-v1.5-13b, and Pickscorev1 models.Orientation distributions used standard deviations of 20, 2, and 1 for azimuth, polar, and rotation, respectively.Source: F Implementation Details · p. 17

About ORIGEN

  • FLUX-Schnell and OrientAnything (ViT-L) provide ORIGEN’s generation and orientation-reward models; the monitor assigns smaller steps at high reward and larger steps at low reward.Figure 5 plots the monitor function; ORIGEN uses η = 0.8, with γ = 0.3 for ORIBENCH-Single and 0.2 for ORIBENCH-Multi.Source: F Implementation Details · p. 18, Figure 5 · p. 18

About Guided Generation Methods

  • Training-free one-step and multi-step guidance methods were matched at 50 function evaluations for fair comparison.The multi-step methods used FLUX-Dev, while ReNO used FLUX-Schnell as its base model.Source: F Implementation Details · p. 18

About Zero-1-to-3 and C3DW

  • The Zero-1-to-3 baseline used front-prompted base images and composited azimuth-conditioned foregrounds, while C3DW controlled only half-front azimuth views.The C3DW implementation mapped scalar orientation inputs to azimuth values spanning 90° to −90°.Source: F Implementation Details · p. 18

Handling Back-Facing Generation

  • Adding “facing back” to back-view prompts helped the base model generate both front- and back-facing images while maintaining high-quality outputs.The authors report this adjustment for cases with azimuth between 90° and 270°.Source: F Implementation Details · p. 18

G Additional Results on the Extended ORIBENCH-Single Benchmark

The extended benchmark tests broader azimuth, polar, and rotation conditions, as well as text-only prompting for four primitive views; ORIGEN outperforms the reported baselines in both settings.

G.1 Orientation Grounding in more general and complex scenario.

  • On the 1K-sample, 25-class extended benchmark, ORIGEN outperforms all other training-free guidance methods on general orientation conditions.The benchmark includes azimuth, polar, and rotation without filtering the front azimuth range; Figure 6 shows qualitative comparisons.Source: §G.1 · p. 19, Figure 6 · p. 20

G.2 Text-to-Image Models.

  • 82.4% vs. 31.2% in 3-view alignment and 86.6% vs. 42.4% in 4-view alignment: ORIGEN exceeds FLUX-Schnell on both settings.The comparison evaluates orientation alignment when direction is specified in text prompts.Source: §G.2 · p. 21
  • Vanilla text-to-image models frequently miss the requested orientation, whereas ORIGEN consistently aligns images with the specified primitive directions.The evaluation provides the orientation condition through the input text prompt.Source: G Additional Results on the Extended… · p. 19, §G.2 · p. 21

H User Study Examples

The appendix describes the AMT user study and its participant screening, then reports the setup used to compare computational costs across orientation-grounding methods.

H User Study Examples

  • The AMT study compared Zero-1-to-3, C3DW, and ORIGEN on ORIBENCH-Single, selecting one sample per object class for 25 questions.Participants chose the image that best satisfied all orientation and prompt conditions.Source: H User Study Examples · pp. 21–22
  • 55 of 100 participants were valid after correctly answering at least three of five randomly interspersed vigilance tests.No additional eligibility restrictions were imposed.Source: H User Study Examples · p. 22

I Computational Costs

  • Computational-cost measurements used 50 NFEs on one 80GB A100, and guided methods were generally slower because they require reward backpropagation.The base model and guided methods generated 512×512 images; the authors note that fewer NFEs may partially reduce cost.Source: I Computational Costs · p. 22, Table 7 · p. 22

J Extended Analysis of the Reward-Adaptive Monitor Function

The appendix examines the reward-adaptive monitor, adds training-based comparisons, and tests reward-guided sampling on layout and depth tasks. It also provides further qualitative orientation-grounding comparisons.

J Extended Analysis of the Reward-Adaptive Monitor Function

  • All tested reward-adaptive monitor settings show negligible metric variation and outperform the fixed-step ORIGEN* baseline, despite defaults chosen by coarse grid search.The monitor varies step size with reward; k controls the slope, while smin and smax bound the step-size range.Source: J Extended Analysis of the… · pp. 22–23

K.1 Baselines.

  • Training-based comparisons include DDPO fine-tuning on 50k orientation-caption pairs, plus multi-view methods evaluated using the image at the target azimuth.DDPO uses policy-gradient updates on a learned reward; SV3D and MV-Adapter generate multiple views.Source: §K.1 · p. 23, K More Comprehensive Comparisons with… · p. 23

K.2 Results.

  • ORIGEN achieves substantially higher orientation alignment accuracy than all training-based baselines while maintaining competitive text-to-image alignment.The comparison is summarized in Table 9.Source: §K.2 · p. 23, Table 9 · p. 24

L.2 Results

  • ORIGEN outperforms FreeDoM and ReNO in condition alignment and image quality on both layout-to-image and depth-guided generation.The quantitative comparison is in Table 10, with qualitative examples in Figure 9.Source: Table 10 · p. 24, §L.2 · p. 24, Figure 9 · p. 25

L.1 Experimental Setup

M More Qualitative Comparisons on Single Object Orientation Grounding

N More Qualitative Comparisons on Multi Object Orientation Grounding

Loading 2503.22194v3…