PaperlayerView exact revision on arXiv

Source-linked AI summary

DragMesh: Interactive 3D Generation Made Easy

Tianshan Zhang, Zeyu Zhang, Hao Tang

arXiv:2512.06424v1cs.CV

TL;DR

Interactive-articulation methods tend to be either physically consistent but too slow for real-time use or generative but in violation of basic kinematic constraints. DragMesh decouples semantic joint-type reasoning from geometric regression and feeds the resulting priors into a lightweight, FiLM-conditioned Dual Quaternion VAE for motion generation. DragMesh’s core generalizes to new objects with low computational overhead, while existing generalizable methods require 5-10 times more parameters and GFlops.

  • Problem

    Interactive-articulation methods tend to be either physically consistent but too slow for real-time use or generative but in violation of basic kinematic constraints.

    Source: §1 · p. 2

  • Method

    DragMesh decouples semantic joint-type reasoning from geometric regression and feeds the resulting priors into a lightweight, FiLM-conditioned Dual Quaternion VAE for motion generation.

    Source: §1 · p. 2

  • Results

    DragMesh’s core generalizes to new objects with low computational overhead, while existing generalizable methods require 5-10 times more parameters and GFlops.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DragMesh uses a lightweight generation core that generalizes across diverse 3D meshes while enabling real-time interaction.

    Source: §1 · p. 2

    The authors identify noisy joint-parameter predictions as a limitation motivating future research.

    Source: §9.1 · pp. 12–13

Abstract

from arXiv · show

While generative models have excelled at creating static 3D content, the pursuit of systems that understand how objects move and respond to interactions remains a fundamental challenge. Current methods for articulated motion lie at a crossroads: they are either physically consistent but too slow for real-time use, or generative but violate basic kinematic constraints. We present DragMesh, a robust framework for real-time interactive 3D articulation built around a lightweight motion generation core. Our core contribution is a novel decoupled kinematic reasoning and motion generation framework. First, we infer the latent joint parameters by decoupling semantic intent reasoning (which determines the joint type) from geometric regression (which determines the axis and origin using our Kinematics Prediction Network (KPP-Net)). Second, to leverage the compact, continuous, and singularity-free properties of dual quaternions for representing rigid body motion, we develop a novel Dual Quaternion VAE (DQ-VAE). This DQ-VAE receives these predicted priors, along with the original user drag, to generate a complete, plausible motion trajectory. To ensure strict adherence to kinematics, we inject the joint priors at every layer of the DQ-VAE's non-autoregressive Transformer decoder using FiLM (Feature-wise Linear Modulation) conditioning. This persistent, multi-scale guidance is complemented by a numerically-stable cross-product loss to guarantee axis alignment. This decoupled design allows DragMesh to achieve real-time performance and enables plausible, generative articulation on novel objects without retraining, offering a practical step toward generative 3D intelligence. Code: https://github.com/AIGeeksGroup/DragMesh. Website: https://aigeeksgroup.github.io/DragMesh.

1. Introduction

Interactive articulation must balance physical consistency with real-time efficiency, but existing methods tend to be either slow and physically consistent or fast and kinematically inconsistent. DragMesh separates kinematic reasoning from motion generation to pursue generalizable, low-overhead interaction.

1. Introduction

  • Current interactive-articulation methods tend to be physically consistent but too slow for real time, or generative but in violation of basic kinematic constraints.The paper frames this as a trade-off among fidelity, physicality, and efficiency.Source: §1 · p. 2
  • Existing generalizable models combine one-time kinematic reasoning with real-time motion generation in a monolithic, heavyweight architecture.The paper identifies this combination as a source of the persistent cost and model-specificity dilemma.Source: §1 · p. 2
  • DragMesh decouples semantic joint-type reasoning from geometric regression of the joint axis and origin before motion synthesis.The geometric regression is performed by KPP-Net, and the predicted priors are passed to the motion generator.Source: §1 · p. 2
  • DragMesh’s Dual Quaternion VAE uses FiLM conditioning to generate articulated motion, while its core is designed to generalize to new objects with low computational overhead.The paper contrasts this with generalizable methods requiring 5-10 times more parameters and GFlops, and lightweight methods requiring per-object training.Source: §1 · p. 2
  • The authors position DragMesh as a practical step toward generative 3D intelligence that bridges generative modeling and real-time interactive articulation.The conclusion is framed as a practical step, rather than a complete solution to generative 3D intelligence.Source: §1 · p. 2

2. Related Work

Prior 3D-generation work has advanced static geometry and texture, while interactive manipulation raises additional requirements for explicit structure, physical constraints, and low latency. DragMesh uses decomposable meshes for real-time, physically consistent manipulation and reports lower computational cost than existing generalizable approaches.

2.1. 3D Generation

  • Explicit mesh structure exposes vertex-face topology for vertex editing, skeletal rigging, and integration with physics and rendering pipelines.The paper describes this structure as essential to animation, simulation, and interactive design.Source: §2.1 · p. 3
  • DragMesh builds on decomposable mesh representations to enable real-time, physically consistent manipulation through a lightweight dual-quaternion generative model.The related-work discussion contrasts explicit meshes with holistic geometries that lack part decomposition.Source: §2.1 · p. 3

2.2. Interactive Manipulation

  • Direct 2D point editing does not meet 3D tasks’ requirements for multi-view consistency and physical constraints, motivating workflows that lift objects into 3D for editing.The described 2D-3D-2D workflow reprojects the edited 3D representation back into the original view.Source: §2.2 · p. 3
  • Diffusion-based interactive editing can require several minutes of iterative sampling, disrupting the low-latency workflow expected in interactive design.The discussion contrasts this with near-instant but low-quality, non-generative methods.Source: §2.2 · p. 3
  • Figure 2 reports that DragMesh achieves generalization at significantly lower computational cost than existing generalizable approaches.Bubble size represents core-module parameter count, while color distinguishes generalizable methods from methods requiring per-object training.Source: Figure 2 · p. 3

3. Methodology

DragMesh separates kinematic reasoning from motion generation, then conditions a dual-quaternion motion model on geometry, joint parameters, and user interaction. Its architecture fuses multimodal features and uses a parallel Transformer decoder with physical correction to produce constrained motion sequences.

3.1. Overview

  • The overview separates kinematic reasoning from motion generation, using predicted joint parameters to drive a pretrained DQ-VAE on novel meshes.The kinematics pipeline predicts semantic interaction type with a VLM and regresses geometric parameters with KPP-Net.Source: §3.1 · p. 3

3.2. Motion Representation and Pre-processing

  • Dual quaternions provide an 8-parameter, continuous, singularity-free representation for rigid-body transformations.The method constructs 16-frame ground-truth motion sequences by sampling angles for revolute joints or distances for prismatic joints.Source: §3.2 · pp. 3–4

3.3. Network Architecture

  • The network learns a distribution over dual quaternions for rigid-body transformations, conditioned on geometric features and interaction cues.The VAE jointly encodes geometric and kinematic priors.Source: §3.3 · p. 4

3.3.1. Multi-Modal Encoders

  • Specialized encoders represent joint constraints, point-cloud geometry, and the user’s drag interaction as separate features.The point-cloud encoder uses 4096 normalized mesh points with XYZ coordinates and a binary moving-part mask.Source: §3.3.1 · pp. 4–5

3.3.2. Feature Fusion and Conditional VAE

  • Local point features are sampled around the joint origin and drag point, then fused with joint and motion features for conditional VAE encoding.A learned gate combines the features, and the encoder also receives the raw joint feature to retain joint conditioning.Source: §3.3.2 · p. 5

3.3.3. Conditioned Transformer Decoder and Physics Correction

  • A non-autoregressive Transformer processes all 16 frames in parallel, with joint features injected through FiLM at every layer to enforce kinematics.A Physics Correction module adds a predicted residual to the base sequence, using joint type, joint features, and axis as context.Source: §3.3.3 · p. 5

3.4. VAE Training Objectives

DragMesh combines explicit physical losses and latent regularization for motion training with a separate network for predicting joint geometry. At inference, semantic joint intent and predicted geometry condition motion generation from a user drag.

3.4. VAE Training Objectives

  • Explicit physical losses complement geometric and quaternion reconstruction losses to keep generated motion consistent with joint constraints.Axis alignment and zero-motion penalties differ for revolute and prismatic joints; reconstruction losses weight the first frame 2x.Source: §3.4 · p. 5
  • Free Bits encourages meaningful latent information before prior matching and resolves the training difficulties initially observed.The KL divergence is penalized only for samples below a minimum information threshold δ.Source: §3.4 · pp. 5–6

3.5. Kinematics Prediction Network (KPP-Net)

  • KPP-Net predicts joint axis and origin from mesh geometry, part segmentation, and a drag interaction using global and local point encoders.It is trained independently with ground-truth URDF parameters, using geodesic axis loss and Smooth L1 origin loss.Source: §3.5 · p. 6

3.6. Inference Pipeline

  • At inference, a VLM predicts revolute-versus-prismatic intent, KPP-Net predicts axis and origin, and the Dual Quaternion VAE generates motion from those constraints and the user drag.The pipeline operates on raw meshes without URDF annotations and applies generated transformations to movable vertices.Source: §3.6 · p. 6

4. Experiments

The experiments use articulated-object data and evaluate geometry, joint constraints, and latent quality, then compare methods and ablate architecture, losses, and KPP design.

4.1. Datasets and Evaluation Metrics

  • GAPartNet supplies part-level geometry, URDF joint structures, and physically valid mobility parameters; evaluation measures geometry, joint-parameter errors, invalid-motion penalties, and KL divergence.The annotation-free inference pipeline segments the movable part, predicts joint type, regresses axis and origin, then generates a dual-quaternion animation.Source: §4.1 · pp. 6–7, Figure 4 · p. 7

4.2. Comparative Study

  • The comparative study uses Objaverse and method-specific data subsets, assessing data-processing complexity, computational overhead, and qualitative fidelity.Subsets follow each competitor’s official implementation because methods differ in annotation, mesh, and category requirements.Source: §4.2 · p. 7

4.3. Ablation Study

  • The full architecture (M7) resolves the baseline’s minimal-motion failure and the instability introduced by encoder fusion and FiLM, achieving optimal reconstruction, physical plausibility, and expressiveness.Table 1 is an additive ablation in which each row adds one component to the previous configuration.Source: §4.3 · p. 7, Table 1 · p. 8
  • Combining physics losses with free-bits KL yields the best results across all metrics, whereas either component alone leaves reconstruction or physical correctness deficient.Physics losses alone reduce constraint violations but degrade reconstruction; free bits alone improve VAE stability but do not enforce physical correctness.Source: §4.3 · pp. 7–8, Table 2 · p. 8
  • KPP’s dual-stream attention encoder reduces error 2×, and decoupled prediction heads provide a further 50% improvement over the PointNet baseline.Rows 4–6 use the complete mesh, mask, and drag feature set, isolating architectural effects on kinematic regression accuracy.Source: §4.3 · p. 8, Table 3 · p. 8

5. Efficiency

DragMesh’s core generation module has lower computational cost than generalizable models, while lightweight alternatives require per-object training.

5. Efficiency

  • Generalizable models incur 5 to 10 times higher computational costs than DragMesh’s core generation module, while other lightweight methods require separate training for each object.The comparison uses parameter count and GFLOPs and excludes one-off upstream costs, including segmentation preprocessing and VLM calls.Source: §5 · p. 8

6. Conclusion

DragMesh separates semantic joint-type reasoning from geometric regression of joint axes and origin, then uses a FiLM-conditional Dual Quaternion VAE to generate animation. The conclusion describes the resulting animations as physically consistent and temporally smooth.

6. Conclusion

  • DragMesh uses an LLM to determine joint type, KPP-Net to regress joint axes and origin, and a FiLM-conditional Dual Quaternion VAE to generate animation from those parameters.The conclusion characterizes the generated animation as physically consistent and temporally smooth.Source: §6 · p. 8

7. Implementation Details

The implementation adapts data preparation to each baseline and constructs drag vectors from local motion. It also documents practical adjustments and constraints in preparing comparisons.

7. Implementation Details

  • The comparison used official public code for the baselines and custom data preparation to meet each method’s input requirements.The interaction-data pipeline parses kinematic data, samples movable-part surface points, and derives corresponding drag vectors.Source: §7 · p. 11
  • For large rotations exceeding 180°, the start-to-end displacement can be ambiguous, so drag direction is derived from the next timestep.The resulting vector is extended by accumulating near-linear motion or scaling the instantaneous vector.Source: §7 · p. 11
  • PartRM’s discrete-state input requires a custom pipeline that renders states 0–5, with 12 views each, from continuous trajectories.The pipeline also solves 2D drag vectors from 3D trajectories and generates the propagated-drags index file.Source: §7 · p. 11
  • ArtGS camera views moved farther to improve image capture sometimes had no visible Gaussians, prompting depth-loss skipping and leaving RGB-only views with poor geometric detail.The training pipeline uses RGB loss alone for views whose depth loss cannot be backpropagated effectively.Source: §7 · p. 11
  • DragAPart’s 3D-to-2D drag projection introduced large errors, so the comparison used manually selected start and end points on each 2D image.The rendered RGB image served as the model’s conditional input.Source: §7 · p. 12
  • DragMesh training data was restricted to single-joint interactions, with motion types assigned through heuristic translation/rotation trials and manual labels instead of VLM classification.The selected label was intended to be the unique, correct, and reasonable interaction type for each object.Source: §7 · p. 12

8. Visualization Results

The qualitative comparison reports plausible interactions from DragMesh across categories and marks some baseline outputs blank when generation fails.

8. Visualization Results

  • Blank baseline results indicate generation failures with unrecognizable outlines; DragMesh generates plausible interactions across all categories.Source: §8 · p. 12

9. Limitations and Future Work

DragMesh’s stated limitations concern its motion scope, sensitivity to geometric inputs, and reliance on an external VLM. Future work targets broader kinematics, greater robustness, and a balance between physical understanding and lightweight generation.

9.1. Limitations

  • The current kinematic scope is single-joint translation or rotation around a fixed axis, excluding screw motion and multi-joint chains.Source: §9.1 · pp. 12–13
  • Kinematic prediction is sensitive to initial geometric input; incorrect joint-axis predictions can produce physically implausible mesh transformations.Source: §9.1 · p. 13
  • Motion-type classification relies on an external VLM, whose error rate and external dependency affect prediction-pipeline robustness.Source: §9.1 · p. 13
  • The broader aim is robust physical common sense in a compact, efficient architecture that supports real-time, physically plausible intelligent agents.Source: §9.1 · p. 13

9.2. Future work

  • Future work identifies a need to balance physical understanding with lightweight generation, especially for low-latency HOI and sim2real robotics applications.Source: §9.2 · p. 13
  • Planned directions include internalizing semantic reasoning, extending to screw motion and multi-joint chains, and reducing sensitivity to upstream prediction errors.Source: §9.2 · p. 13
Loading 2512.06424v1…