PaperlayerView exact revision on arXiv

Source-linked AI summary

Flow3D-OPD: Multi-Teacher On-Policy Distillation for 3D Geometry Generation with Flow-Matching Diffusion Transformer

Zhiwei Ning, Zhen Zhou, Puhua Jiang, Xintong Han, Gengming Zhang, Jie Yang, Zhonglong Zheng, Yuanjie Zheng, Wei Liu, Chunchao Guo

arXiv:2609.07137v1cs.CVcs.AI

TL;DR

Post-training for flow-matching 3D geometry generation remains underexplored, particularly multi-teacher distillation. Flow3D-OPD trains specialized teachers and distills them into one student, improving quality across dimensions and outperforming all teachers on averaged metrics.

  • Problem

    Whether multi-teacher on-policy distillation can be applied to 3D geometry generation remains unexplored, amid challenges in training specialized teachers with 3D-specific rewards and stably consolidating capabilities.

    Source: Introduction · p. 2

  • Method

    Flow3D-OPD uses a two-stage pipeline to train dimension-specific teachers with verifier-based DPO, then distills them into one student using hard task routing and gradient accumulation.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Results

    Flow3D-OPD improves generation quality across all dimensions and exceeds all teachers on averaged metrics, with 61.0% B wins.

    Source: Main Results · p. 5, Conclusion · p. 7

  • Takeaways & Limitations

    These findings support multi-teacher on-policy distillation as an effective post-training paradigm for 3D generation.

    Source: Introduction · p. 2

Abstract

from arXiv · show

Recent image-to-3D generation models built on flow-matching diffusion Transformers (DiT) can produce high-fidelity meshes, yet their post-training strategy remains largely unexplored. There exist several critical bottlenecks in reinforcement learning: the inherent difficulty of defining comprehensive rewards for 3D geometric quality, and the gradient interference that arises when jointly optimizing heterogeneous objectives. Inspired by the practicability of on-policy distillation (OPD) in large language models and image generation, we propose \textbf{Flow3D-OPD}, a two-stage post-training framework that introduces multi-teacher distillation into 3D geometry generation. In the first stage, we utilize the semi-policy to enhance the foundational capability of the pretrained model and then design an agentic verifier for 3D geometric quality evaluation. Based on the verifier, we could cultivate domain-specialized teacher models via direct preference optimization (DPO). In the second stage, we consolidate heterogeneous expertise into a unified student model through on-policy distillation with hard task-routing sampling and gradient accumulation, which could mitigate the gradient interference in joint optimization. Without relying on elaborate modifications, our straightforward yet effective design achieves consistent improvements across all geometric quality dimensions and surpasses all teacher models in the average metric. Extensive experiments demonstrate that our approach provides an effective paradigm for reinforcement learning in 3D generation.

Introduction

Flow3D-OPD brings multi-teacher on-policy distillation to 3D geometry generation, addressing challenges in evaluating mesh quality and optimizing heterogeneous objectives. Its two-stage pipeline trains specialized teachers and consolidates their expertise into a unified student.

  • Reinforcement-learning post-training is absent from 3D generation, in part because global image-to-3D similarity misses fine-grained mesh quality and joint rewards can cause gradient interference.Source: Introduction · p. 1
  • Flow3D-OPD’s unified student surpasses all three specialized teachers on average win ratio, with experiments reporting consistent gains across quality dimensions and competitive or superior performance across multiple metrics.Source: Figure 1 · p. 1, Introduction · p. 2
  • DiffusionOPD (Li et al. 2026) and Flow-OPD (Fang et al. 2026) use dense velocity-field supervision to consolidate specialized capabilities, but applying multi-teacher distillation to 3D remains unexplored.The paper attributes this gap to difficulty training teachers with 3D-specific rewards and stably consolidating heterogeneous capabilities.Source: Introduction · pp. 1–2
  • Stage one enhances the base model with semi-policy preference optimization, then trains three dimension-specific teachers using DPO and an agentic VLM verifier for pairwise mesh comparison.The verifier uses multi-round structured reasoning, tool invocation, and case memory.Source: Introduction · p. 2
  • Stage two distills the three frozen teachers into a student that generates its own denoising trajectories, using hard task-routing and gradient accumulation for balanced multi-dimensional optimization.The routed teacher supplies step-level guidance, and gradients are accumulated over a mini-batch for a single backward pass.Source: Introduction · p. 2

Related Work

Prior work progresses from category-specific 3D generation to native flow-matching models, while reinforcement-learning post-training and on-policy distillation have advanced mainly in language and visual generation. Flow3D-OPD applies multi-teacher distillation to 3D geometry, where heterogeneous quality assessment makes it a natural fit.

3D Generation Models

Reinforcement Learning for Generative Models

On-Policy Distillation

  • On-policy distillation trains students on their own generated trajectories under teacher supervision; Flow-OPD (Fang et al. 2026) extends it to flow matching, and Flow3D-OPD adapts it to heterogeneous 3D quality assessment.Prior work includes GKD (Agarwal et al. 2024), MiniLLM, and DistiLLM (Ko et al. 2024) for language models, plus DiffusionOPD (Li et al. 2026) for diffusion; Flow-OPD uses SDE formulations and time-adaptive weighting, with multi-teacher distillation addressing reward sparsity and gradient interference.Source: Related Work · pp. 2–3

Preliminary

The preliminary section introduces DPO for pairwise preference learning and its diffusion adaptation, then explains on-policy distillation as a way to train students on their own generated trajectories under teacher supervision.

DPO in Diffusion Models

  • DPO learns from winning and losing responses conditioned on a prompt without requiring explicit reward modeling.Source: Preliminary · p. 3
  • DiffusionDPO derives a denoising-trajectory loss from the ELBO, with the flow-matching formulation using per-step noise-prediction differences and target velocities.The formulation defines noisy winner and loser intermediates at timestep t; vθ and vref are the policy and reference velocity predictions, weighted by ω(t).Source: Preliminary · p. 3

On-Policy Distillation

  • On-policy distillation minimizes reverse KL between a student and frozen teacher using trajectories sampled from the student’s current policy.Training on self-generated distributions addresses offline distribution shift and exposure bias by supervising states the student visits at inference.Source: Preliminary · p. 3

Our Method

Flow3D-OPD first fine-tunes the pretrained model and trains domain-specialized teachers, then consolidates their expertise in a student through on-policy distillation. Its method combines preference-based supervision, agentic mesh evaluation, and task-routed training with gradient accumulation.

Our Method

  • The flow-matching DiT generates structured latents x ∈ R4096×64 conditioned on image features, and the method uses a two-stage post-training strategy.The architecture follows Hunyuan3D-2.1 (Lai et al. 2025).Source: Our Method · p. 3

Stage 1: Domain-Specialized Teacher Training

  • Because the basic model produces inadequate high-quality samples for reliable preference learning, it is first fine-tuned with semi-policy preference optimization before teacher training.The teachers are trained on top of this fine-tuned model.Source: Stage 1: Domain-Specialized Teacher… · p. 3

Fine-Tuning with Semi-Policy Preference Optimization.

  • Semi-policy DPO treats ground-truth meshes as preferred and current-policy samples as rejected, and combines trajectory optimization with MSE supervision.It substitutes the ground-truth latent for the preferred latent in the DPO loss, enabling optimization along SDE or ODE denoising trajectories; the loss weights are λsemi = 0.2 and λmse = 0.8.Source: Fine-Tuning with Semi-Policy… · p. 4

Agentic Reward Design and Teacher Training.

  • A Gemini-3-Flash agent compares meshes across orientation, pose, and consistency, then supplies dimension-specific preference scores for DPO teacher training.It uses zoom-in rendering, viewpoint rotation, difference-map overlays, and case memory, with up to four tool-invocation rounds. Teacher training filters pairs with score margins below δ and uses iterative rollout-then-train because most sampled pairs fall below that threshold; the result is three domain-specialized teachers.Source: Agentic Reward Design and Teacher… · p. 4

Stage 2: Multi-Teacher On-Policy Distillation

On-Policy Distillation in Flow-Matching.

  • On-policy distillation minimizes reverse KL between student and teacher transition kernels at each step of the student’s trajectory, with ODE sampling reducing matching to L2 velocity alignment.Under SDE discretization, the one-step kernels are Gaussian, and the shared isotropic covariance gives a closed-form reverse KL objective.Source: On-Policy Distillation in Flow-Matching · p. 4

Hard Task-Routing and Mini-Batch Gradient Accumulation.

  • Hard task-routing directs each labeled sample to its matching teacher, while round-robin mini-batches and gradient accumulation combine all three teachers’ guidance per update to mitigate interference.The mini-batches cycle through orient, pose, and consistency before the optimizer step, yielding a balanced update-level consensus.Source: Hard Task-Routing and Mini-Batch… · pp. 4–5

Experiments

Flow3D-OPD achieves the best overall balance across orientation, pose, and consistency, outperforming its teachers on average and previous methods on the reported similarity benchmarks. Ablations and qualitative comparisons support the complementary roles of specialized teachers, task-routed gradient accumulation, and stage-specific sampling.

Experimental Setup

  • Evaluation uses pairwise win rates for orientation, pose, and consistency, plus ULIP and Uni3D scores for 3D-text and 3D-image similarity.Training builds on Hunyuan3D-2.1; the student is distilled from randomly sampled states along a 50-step denoising trajectory.Source: Experimental Setup · p. 5

Main Results

  • Specialized teachers improve their target dimensions but can degrade others; for example, the orientation teacher has 2.7% A wins on orientation and 30.8% A wins on consistency.The passage attributes this tradeoff to heterogeneity among the evaluated dimensions.Source: Main Results · p. 5
  • Flow3D-OPD reaches 18.8%, 93.5%, and 81% B wins on orientation, pose, and consistency, respectively, and averages 61.0% B wins, outperforming all teachers.Direct DPO on the initial model degrades performance, while OPD without the semi-policy stage gives moderate but weaker improvements.Source: Main Results · p. 5
  • Under GPT-5.5, Flow3D-OPD records 9.9% A wins and 51.4% B wins; blind human evaluators also give it the highest average ranking score.GPT-5.5 is independent of the Gemini-3-Flash model used for teacher-training rewards, and human annotators ranked methods without knowing their identities.Source: Main Results · pp. 5–6
  • Flow3D-OPD improves over Hunyuan3D-2.1 across all four ULIP and Uni3D metrics and outperforms all compared prior methods.The comparisons include Craftsman 1.5, Michelangelo, Trellis2/512, Direct3D-S2, Hi3DGen, and Hunyuan3D 2.0.Source: Main Results · p. 6

Ablation Studies

  • Gradient accumulation with hard task routing performs best across metrics and produces more stable, lower-loss convergence than single-step updates.Hard task routing also outperforms sequential sampling; the authors attribute the gains to reducing inter-task gradient interference.Source: Ablation Studies · p. 6, Table 3 · p. 6, Figure 3 · p. 6
  • SDE sampling is more effective during fine-tuning, while ODE sampling is more effective during OPD, with faster convergence and lower distillation loss.The authors link SDE's fine-tuning advantage to more diverse preference losers and higher reward margins, and ODE's OPD advantage to more precise teacher-velocity alignment.Source: Ablation Studies · p. 6, Table 4 · p. 6, Figure 4 · p. 6

Qualitative Visualization

  • The full model generates coherent, precise multi-view meshes, while the baseline can be misaligned or structurally inconsistent and OPD-only lacks comparable stability.Compared with individual teachers, the student preserves canonical orientation and articulated pose while producing richer surface texture.Source: Qualitative Visualization · pp. 6–7, Figure 5 · p. 7, Figure 6 · p. 7

Conclusion

Flow3D-OPD is a two-stage post-training framework for 3D geometry generation that combines specialized teachers with on-policy distillation. It improves all geometric dimensions and surpasses its teachers on the averaged metric.

Conclusion

  • Flow3D-OPD improves all geometric dimensions and surpasses its teachers on the averaged metric.The conclusion reports these results from extensive experiments.Source: Conclusion · p. 7
  • The first stage uses an agentic reward system for fine-grained geometric evaluation and trains domain-specialized teachers via DPO across three quality dimensions.Source: Conclusion · p. 7
  • The second stage distills multiple teachers into a student using hard task-routing, gradient accumulation, and velocity-field supervision along the student's denoising trajectory.Source: Conclusion · p. 7

Supplementary Material · Details of the Training Pipeline

The supplementary material details a two-stage training pipeline, including its shared training budget, teacher and student training procedures, and dimension-specific preference datasets. It also identifies Figure 7 as a diagram of the agentic verifier.

Hyperparameter Configuration

  • All stages use 5K training steps on 2 × 8 H20(90G) GPUs.Table 5 summarizes the hyperparameter configurations for the stages.Source: Hyperparameter Configuration · p. 10
  • Stage 1a fine-tunes the pretrained model on offline ground-truth winners and on-policy generated losers.This fine-tuned model initializes both the domain-specialized teachers and the final student.Source: Hyperparameter Configuration · p. 10
  • Each domain-specialized teacher is trained with on-policy DPO using Gemini-3-Flash pairwise rewards and iterative rollout-then-train to address the low valid-pair ratio.Source: Hyperparameter Configuration · p. 10
  • Stage 2 distills the three frozen teachers into the student using on-policy trajectory sampling, hard task-routing, and gradient accumulation across three data types.Source: Hyperparameter Configuration · p. 10

Dataset Statistics

  • Training uses three separate preference datasets: orientation for canonical axis alignment, pose for diverse articulated human configurations, and consistency for multi-view structural and textural coherence.Table 6 reports dataset sizes and descriptions for each quality dimension.Source: Dataset Statistics · p. 10
  • Evaluation uses three dedicated test sets that mirror the orientation, pose, and consistency training dimensions to isolate generation ability along each aspect.Source: Dataset Statistics · p. 10
  • Figure 7 presents the detailed pipeline of the agentic verifier.Source: Figure 7 · p. 10

Agentic Verifier Details

The agentic verifier uses Gemini-3-Flash and structured, tool-augmented reasoning to compare candidate meshes generated from the same reference image for DPO preference pairs.

Agentic Verifier Details

  • The Gemini-3-Flash verifier compares two meshes generated from the same reference image using structured, tool-augmented reasoning to construct DPO preference pairs.Its process includes a system prompt, tool calls, and case memory.Source: Agentic Verifier Details · p. 10

System Prompt

  • The system prompt evaluates orientation, human pose and body structure, and image-to-3D consistency against the reference image.It treats perspective baking or retained diagonal slant as a fatal orientation defect, prioritizes limb structure and pose accuracy, and assesses consistency without judging orientation or artistic style.Source: System Prompt · pp. 10–11

Case Memory Mechanism

  • Before judging a new pair, the agent retrieves similar past cases by object category and uses them as few-shot examples to anchor judgments to precedents.The case memory is intended to reduce variance in VLM-based evaluation.Source: Case Memory Mechanism · p. 11

Derivation of OPD in Flow-Matching

Flow3D-OPD derives its reverse-KL objective by decomposing the student’s Markovian denoising rollout into stepwise transition KLs. In the deterministic ODE regime used for OPD, this reduces to direct squared-L2 velocity matching through the student.

  • The Markovian student rollout decomposes reverse-KL OPD into stepwise transition KLs, with flow-matching SDE steps discretized by Euler–Maruyama as Gaussian transitions.The transition mean is determined by the velocity drift, while the noise term supplies the covariance.Source: Derivation of OPD in Flow-Matching · pp. 11–12
  • With shared isotropic covariance, the student and teacher kernels’ per-step KL reduces to a time-weighted squared velocity difference, with gradients stopped through the frozen teacher.The resulting OPD objective is a time-weighted velocity-matching loss.Source: Derivation of OPD in Flow-Matching · p. 12

Additional Experiments

CFG ablations find that the best OPD distillation target uses w = 1.0, while inference performs best at w = 3.0. The section also presents visualizations of Flow3D-OPD meshes refined with LATTICE.

Impact of the CFG Weights

More Quality Visualization

Loading 2609.07137v1…