PaperlayerView exact revision on arXiv

Source-linked AI summary

ArtiMo: Agent-Driven Articulated Mesh Animation

Chunyu Zou, Peng Dai, Yi-Hua Huang, Ze Yuan, Jingwei Huang, Yeming Yao, Xiaojuan Qi

arXiv:2608.20699v1cs.CV

TL;DR

Text-driven articulated mesh animation requires both strict kinematic validity and causal coordination among object parts, but existing generic mesh animation methods generally lack explicit articulation supervision. ArtiMo combines URDF constraints with VLM/LLM-based semantic reasoning, motion planning, and visual refinement in a zero-shot framework, and experiments show strong performance across articulated motions.

  • Problem

    Text-driven articulated mesh animation must satisfy kinematic constraints and causal part interactions, while existing data-driven methods typically lack explicit articulation structures and task-specific supervision.

  • Method

    ArtiMo uses a zero-shot agentic pipeline that combines URDF articulation, VLM-based link and causal reasoning, LLM motion planning, and critic-guided visual refinement.

  • Results

    ArtiMo consistently outperforms 2D and 4D animation baselines, particularly on kinematically coupled and causally dependent motions, achieving P_gIoU 0.450 versus 0.019 for AnimateAnyMesh and −0.130 for Animate3D with predicted articulation.

  • Takeaways & Limitations

    The framework supports action-conditioned articulated animation by combining explicit feasible-motion constraints with semantic and causal reasoning, without relying solely on ground-truth articulation.

  • Takeaways & Limitations

    Performance depends on accurate articulation structures and reliable visual grounding, while current plans approximate physical effects and the benchmark evaluates ordered key states rather than exact frame-level trajectories.

Abstract

from arXiv · show

Animating articulated 3D meshes via text requires satisfying strict kinematic constraints, modeling causal interactions between parts, and achieving instruction fidelity. Due to the absence of task-specific training data and explicit articulation supervision, existing data-driven mesh animation methods are largely inapplicable to this setting. To address this, we propose ArtiMo, a novel agent-driven framework for text-guided articulated mesh animation. Operating in a zero-shot manner, ArtiMo develops an agentic pipeline powered by Large Language and Vision-Language Models (LLMs/VLMs) to orchestrate motion generation. By synergizing the explicit kinematic constraints of URDF with the agent's reasoning and planning capabilities, it effectively produces causally coherent part motions and interactions without requiring model fine-tuning. To ensure motion correctness, the agent additionally utilizes a visual self-improvement mechanism: generated animations are rendered into compact keyframes and motion cues, enabling the VLM to iteratively diagnose and correct errors. Furthermore, we contribute a new benchmark dataset spanning 21 articulated object categories, featuring high-quality motion annotations enriched with causal relationships. Extensive experiments demonstrate that ArtiMo significantly outperforms baselines, particularly on complex, causally driven motions. The project page is available at https://zou-2004.github.io/ArtiMo/.

1. Introduction

Text-driven articulated mesh animation must satisfy both part-level kinematic constraints and causal interactions, which existing generic mesh methods typically do not model. ArtiMo addresses this gap with a zero-shot agentic pipeline that combines URDF constraints, VLM/LLM reasoning, executable motion planning, and visual self-improvement, supported by a new benchmark.

  • Motivation: Articulated animation requires kinematically valid motion and causally correct interactions among parts, not merely plausible mesh movement.Examples include drawers sliding along rails, wheels rotating with body motion, and pedals triggering lids.
  • Research gap: Existing text-driven 4D methods typically ignore explicit articulation structures, limiting their ability to handle joint axes, limits, dependencies, and causal ordering.URDF encodes feasible part motions and relations but does not determine the action-conditioned trajectory.
  • Approach: ArtiMo is a zero-shot agentic framework that combines URDF-defined kinematic constraints with VLM/LLM semantic, causal, and motion-planning capabilities.The VLM grounds part semantics and causal relationships, while the LLM translates them into executable joint-level trajectories with timing and state transitions.
  • Approach: ArtiMo adds a visual self-improvement loop that renders animations as compact keyframes and motion cues for iterative VLM diagnosis and correction.This replaces single-pass planning with critic-guided refinement of generated motion.
  • Evaluation: The benchmark contains 225 annotated motion sequences across 21 object categories, covering causal and non-causal actions.Experiments report substantial gains over existing 2D/4D animation baselines, with ablations supporting structured planning and visual criticism.

2. Related Work

Prior work animates generic meshes, images, or parts using learned motion and video priors, interaction prediction, procedural rules, or agentic workflows. ArtiMo differs by explicitly using the target object's URDF while combining structured planning with critic-guided refinement.

  • Object animation and interaction: Animate3D, MotionDreamer, AnimateAnyMesh, and Puppet-Master generate mesh or image motion using video, semantic, text-conditioned, or sparse-control priors.These approaches primarily target visually plausible motion rather than explicit articulated structure.
  • Object animation and interaction: Interaction methods predict actionable parts, contact locations, motion directions, or manipulation trajectories, while procedural and robotic methods realize motion through rules, simulation, or external manipulation.These lines of work provide complementary interaction or execution mechanisms.
  • ArtiMo's distinction: Unlike previous methods, ArtiMo explicitly takes the target object's URDF as input for articulated mesh animation.This representation supplies structured articulation information to the agentic workflow.
  • Agentic graphics workflows and self-refinement: Agentic graphics workflows organize language models into planning, execution, evaluation, and revision stages, with self-refinement improving outputs through repeated critique and correction.Related graphics systems apply language models to spatial planning, environment generation, tools, and critic guidance.

3. Method

ArtiMo generates action-conditioned articulated mesh animations from URDF, meshes, and text prompts by combining visual grounding, causal planning, constrained execution, and iterative visual criticism. Its benchmark and metrics evaluate whether generated motions preserve articulation, causal behavior, and part-aware geometry.

  • System formulation: ArtiMo takes an articulated mesh, its URDF, and an action prompt to generate a GLTF 4D animation satisfying kinematic and causal requirements.The system produces executable motion over joint configurations while respecting URDF-defined articulation.
  • Pipeline: The agentic pipeline preprocesses meshes and URDFs, then uses perceiver, planner, and executor stages to analyze parts, plan motion, and render animation.Preprocessing includes rendered viewpoints, colored link overlays, joint summaries, and scale context.
  • Pipeline: The VLM identifies manipulated links and causally affected joints, while the LLM converts these relations into timed joint configurations and state transitions.The planner also infers temporal offsets and latencies to order dependent motions.
  • Pipeline: URDF-constrained execution interpolates planned key states and propagates transforms through the parent-child hierarchy before exporting the animation as GLTF.This execution stage translates the planned configurations into continuous joint trajectories.
  • Self-improvement: The critic loop renders compact keyframes with trajectories and translational or rotational cues because sparse RGB frames can obscure motion direction and extent.The visual critic is designed to diagnose errors such as incorrect direction, magnitude, or return behavior.
  • Benchmark and evaluation: The benchmark addresses missing text-conditioned motion ground truth, while part-aware metrics separately weight dynamic and static geometry using λ = 0.8.The metrics include part-wise generalized IoU and part-wise point consistency, with dynamic parts emphasized for action relevance.

4. Experiments

ArtiMo outperforms generic animation baselines on articulated action execution, preserving geometry while modeling joint-level and causal behavior. Ablations show that structured planning and iterative visual refinement each improve results.

  • Results: ArtiMo improves action following, causal correctness, and geometry consistency for generic articulated objects.The method produces mesh sequences that better execute intended articulated actions than the evaluated baselines.
  • Qualitative comparison: ArtiMo better preserves object geometry while executing hinge-axis motion, control-part movements, causal responses, and release–return behavior.The comparison contrasts ArtiMo with methods that often produce static motion, global deformation, or ambiguous image-space movement.
  • Quantitative comparison: Across all metrics, ArtiMo achieves the best performance on causal and non-causal articulated motions.With predicted articulation, ArtiMo reaches a P_gIoU of 0.450 versus 0.019 for AnimateAnyMesh and −0.130 for Animate3D.
  • Ablation study: 0.962 P_gIoU and 0.899 P_OccF1 result from single-pass agentic planning plus iterative critic–actor refinement.The single-pass pipeline raises P_gIoU from 0.703 to 0.962 and P_OccF1 from 0.636 to an intermediate value, while the refinement loop further increases them to 0.985 and 0.899.
  • Ablation study: The full agent corrects subtle button release–return errors that remain after direct or single-pass planning.Rendered intermediate results are evaluated by a critic–actor loop, which refines the motion plan and improves causal consistency.

5. Application to Robotic Manipulation

ArtiMo’s structured motion plans can be reused as object-centric targets for robotic manipulation. They represent not only trajectories, but also the interactions and causal ordering needed for different execution patterns.

  • Application to Robotic Manipulation: ArtiMo reuses generated motion plans directly for robotic execution because they encode manipulated parts, effects, and causal dependencies.These requirements match the information inferred during articulated motion generation.
  • Application to Robotic Manipulation: A release-button plan makes the robot press the button rather than directly manipulate the lid.The plan models lid opening as the downstream effect of pressing the control.
  • Application to Robotic Manipulation: ArtiMo supports both direct grasp-and-follow manipulation and contact-triggered articulated motion.The dishwasher uses a generated door trajectory after grasping, while the kettle uses button contact to trigger lid opening.

6. Conclusion

ArtiMo combines URDF-based articulation with multimodal reasoning and critic–actor refinement for action-conditioned articulated mesh animation. It outperforms generic baselines and extends its object-centric motion plans to robotic execution.

  • Conclusion: ArtiMo infers action-relevant links, causal dependencies, and executable motion plans from URDF files, mesh geometry, and natural-language prompts.Its critic–actor loop diagnoses rendered motion errors and updates the animation plan.
  • Conclusion: ArtiMo outperforms generic 3D and 2D animation baselines in quantitative and qualitative comparisons.A benchmark of 225 articulated animations covers causal and non-causal actions with part-normalized 3D and 2D evaluation protocols.
  • Conclusion: Object-centric motion plans can be retargeted to robotic execution.The plans explicitly represent manipulated parts and interaction-induced effects.

1. Preliminaries and Input Representation

URDF supplies executable kinematic structure, but not the action-conditioned semantics, causal dependencies, or temporal ordering required for natural-language articulated animation. ArtiMo therefore combines URDF with multimodal reasoning and visual representations of object parts.

  • URDF Representation: URDF represents articulated objects as links and joints with types, axes, parent–child relations, origins, and motion limits.Common joint types include revolute, continuous, prismatic, and fixed joints.
  • URDF Representation: URDF converts motion plans into time-varying link poses through joint axes, origins, limits, and mesh geometry.These fields provide the low-level executable structure for animation.
  • URDF Representation and Gap: URDF does not specify which links respond to a natural-language action, how joints are causally coupled, or how motion phases are ordered.It defines feasible motion space rather than an action-conditioned trajectory.
  • ArtiMo Input Representation: ArtiMo uses multimodal agentic reasoning to convert action prompts into time-parameterized animation plans.URDF provides the low-level articulation prior while semantic grounding and causal reasoning determine action-relevant motion.
  • Visual and Joint Representations: Reference and colored overlay views expose assembled appearance, visible and internal links, labels, and spatial extents for VLM grounding.The VLM also receives joint summaries containing joint types, parent–child relations, axes, limits, and origins.
  • Scale Context: Scale context from URDF and GLB bounding boxes calibrates motion magnitude, camera distance, and mesh-to-kinematic conversion.These quantities are numerical inputs to planning and rendering rather than learned labels.

2. Agentic Animation Pipeline Details

ArtiMo uses structured VLM/LLM stages to convert actions into URDF-constrained, causally ordered joint timelines. Coverage selection and visual criticism support iterative motion correction through compact rendered cues.

  • Perceiver: The Perceiver grounds actions to canonical URDF links, separating directly manipulated controls from downstream causal effects.It uses visual labels, masks, affordances, and reference renders for link identity and localization.
  • Planner: The Planner converts causal JSON and URDF limits into executable joint-level timelines with explicit phases, delays, and timing constraints.Control-triggered effects use distinct actuation, latency or release, and effect-motion phases unless rigid coupling is explicit.
  • Planner: Motion plans preserve target identities, causal order, world directions, and explicit joint targets without imposing unsupported symmetry or limit transformations.Continuous rotations use joint_velocity, while spring returns require a real restoring mechanism.
  • Coverage-view selection: The critic selects diagnostic views for required moving links and requests additional views when current coverage is insufficient.A usable view requires both a visible bounding box and readable label text; the module cannot edit the action or motion plan.
  • Visual self-improvement: Rendered keyframes and motion cues let the critic emit bounded repair hints that update motion magnitude or return timing.For a button that fails to return, the critic can request NO_RELEASE_RETURN with timing adjustment and extra-large magnitude.

3. Benchmark Construction and Evaluation Protocol

The benchmark represents articulated actions as structured phases, constraints, and temporal relations rather than fixed frame-level trajectories. Its evaluation distinguishes required sequential states from valid overlapping motions.

  • Dataset: The benchmark contains 225 action instances from 180 unique source assets across 21 object categories.Sources include ArtVIP, LightWheel, and PartNet-Mobility.
  • Annotation format: Each annotation specifies phases, per-phase motion constraints, and temporal relations between phases.Constraints cover targets, directions, joint angles, displacement, velocity, or return mode.
  • Temporal relations: Semantic phase order does not by itself require non-overlap, allowing annotations to represent concurrent motion when relations permit it.This avoids treating every ordered phase as strictly sequential.
  • Sequential motion: For basket lifting, handle rotation precedes whole-basket translation because the relation sets overlap_allowed=false.The evaluator matches these phases as sequential key states.
  • Overlapping motion: For door opening, knob return may overlap with door motion after the door-opening phase reaches a permitted progress point.The benchmark therefore accepts physically plausible timing variation while preserving causal order.

4. Additional Experiments

Additional experiments examine motion-type coverage, critic diagnosis, runtime, and repeatability. They test the refinement loop under controlled failures and assess whether results remain stable across repeated runs.

  • Motion-type coverage: The 225 benchmark cases comprise 71 independent, 58 mechanically coupled, 67 state-dependent, and 29 environment-mediated motions.The four types are assigned using the ground-truth plan and annotation structure.
  • Critic diagnosis: The critic is tested on 168 injected failures across 44 cases, four motion types, 13 object classes, and five failure categories.Each test starts from a correct plan and corrupts execution without exposing the ground-truth animation.
  • Runtime: 282.2 ± 133.6 seconds is the mean runtime per asset–action run, with 3.87 VLM calls and one LLM call.About 44% of tracked wall-clock time is attributed to VLM API latency and 56% to rendering, geometry, trajectory processing, export, and the LLM call.
  • Repeatability: On a balanced 32-case subset, run-level standard deviations are 0.011 for P_gIoU, 0.009 for P_PC, and 0.013 for P_OccF1.The mean per-case P_gIoU standard deviation is 0.024 across three end-to-end runs.

5. Limitations

ArtiMo depends on accurate input articulation structures and can struggle with visually difficult parts. Its plans prioritize kinematic and causal correctness over full physical simulation, while the benchmark evaluates ordered key states rather than exact frame trajectories.

  • Scope and assumptions: Incorrect joint axes, origins, or link bindings can produce implausible motion even when the causal plan is correct.The system may also struggle with small, textureless, or occluded parts.
  • Physical modeling: Spring return, damping, and rolling parameters are only approximate because current plans emphasize kinematic and causal correctness rather than full physical simulation.The limitation concerns physical fidelity, not the stated kinematic execution constraints.
  • Evaluation scope: The benchmark focuses on ordered key states rather than exact frame-level trajectories because many prompts admit multiple valid timings.Future work may add stronger articulation reconstruction, physics validation, and learned motion critics.
Loading 2608.20699v1…