PAMI: Part Anchored Motion for Text to Human-Object Interaction Generation

• Full-body HOI generation must match text while coordinating human motion and object trajectories; small relative errors can yield drift, missed contact, or penetration.Source: §1 · p. 2
• Prior methods commonly represent human and object trajectories separately, while PAMI targets a latent representation that captures their intricate relations and contacts.Source: §1 · p. 2
• PamiGen generates a coarse text-conditioned interaction in the part-anchored latent space, and PamiRefiner then resolves fine-grained contact geometry through hybrid long- and short-range sensing.Source: §1 · p. 2
• 14.5% higher contact recall than the previous state-of-the-art method, with more faithful interactions and more accurate human-relative object motion on InterAct.Source: §1 · p. 2
• PAMI introduces a part-based latent generative framework for text-conditioned full-body HOI, using body-part anchors and learned weights to vote for object motion.Source: §1 · pp. 2–3
• The RefineNet uses mixed generated and ground-truth data and improves both PAMI latent-generation results and baseline generations zero-shot.Source: §1 · p. 3

Text-to-HOI and latent motion generation

• Full-body HOI methods commonly generate per-frame states and couple human and object streams through feature exchange, relation or contact prediction, or sampling guidance; latent motion generation is common, but prior latent HOI work cited here does not model full-body interaction.Source: §2 · p. 3

Local and relative object representations

• Prior relative-object methods use marker-to-surface vectors, contact-distance weights, or wrist-relative trajectory mixtures; PAMI extends learned composition to full-body anchors in latent space.Source: §2 · p. 3

Guidance and refinement for HOI

• HOI correction spans contact-aware training, diffusion guidance, iterative correction, optimization, learned or physics-based correction, and local-geometry methods; PAMI uses latent generation followed by local sensing.Source: §2 · p. 3

Overview

• Figure 2 depicts the pipeline from anchor-voted latent interaction and text-conditioned coarse generation to explicit-space refinement with hybrid probes.Source: Figure 2 · p. 4

3 METHOD

• PamiVAE encodes coupled human–object motion, PamiGen generates a coarse interaction from text, and PamiRefiner recursively refines contact geometry.The stages operate first in latent space and then in explicit motion space.Source: §3 · p. 4, §3.1 · p. 4, §3.2 · p. 5

3.1 PART-ANCHORED INTERACTION VAE

• Separate part and object encoders feed spatiotemporal attention, producing a compact latent representation that captures human–object coupling.The encoders downsample motion so each latent token represents four frames.Source: §3.1 · pp. 4–5
• Body parts vote for object motion through learned per-frame weights, while a free anchor handles motion not driven by a body part.At decoding, anchor frames are computed from the decoded human motion, dynamically coupling the votes to the body.Source: §3.1 · pp. 4–5

3.2 TEXT-CONDITIONED LATENT INTERACTION GENERATION

• PamiGen uses flow matching to generate interaction latents conditioned on text and a canonical object mesh.Its training adds explicit-space losses because latent velocity supervision alone does not constrain recovered object trajectories.Source: §3.2 · pp. 5–6

3.3 RECURSIVE GEOMETRIC REFINEMENT

• PamiRefiner combines long-range probes around body joints with short-range probes near hand joints and fingertips to represent overall influence and detailed surface geometry.The short-range probes include nearby object-surface points, while long-range probes capture direction, distance, and surface normals.Source: §3.3 · p. 6
• PamiRefiner predicts motion updates and is trained on both corrupted ground-truth and generated motions, using geometric objectives for generated samples.Those objectives address contact, penetration, support, smoothness, and preservation of non-interacting regions; refinement is applied recursively.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

• Experiments use the standard InterAct benchmark and evaluate semantic alignment, motion quality, and interaction geometry for hands and all body joints.LIGHT reports the three interaction-geometry metrics only on hand joints.Source: §4 · p. 7

4.1 BASELINE COMPARISON

• PAMI achieves the best performance on almost all InterAct metrics, with large improvements in semantic alignment and interaction geometry.The reported geometry gains particularly concern penetration and contact.Source: Table 1 · p. 7, §4.1 · p. 7, §4 · p. 7

4.2.1 PART ANCHORED VOTING REPRESENTATION

• Part factorization lowers autoencoder reconstruction error by 60.8%, and replacing part encoders with a full-body encoder degrades all metrics.Representation variants are paired with refiners trained on their own outputs for the refined comparison.Source: §4.2.1 · p. 8
• Removing non-wrist anchors, all anchors, learned per-frame voting weights, or absolute root poses degrades generation performance.Removing absolute poses substantially degrades all metrics; uniform averaging models local anchor influence less effectively.Source: §4.2.1 · p. 8

4.2.2 FINE-GRAINED REFINEMENT

• Four PamiRefiner steps nearly halve penetration, improve full-body and hand interaction F1, bring Contact closer to ground truth, and improve FID while R-1 remains stable.The comparison applies refinement to the same coarse generations.Source: §4.2.2 · p. 8
• Training PamiRefiner on both controlled corruptions and generator outputs performs better than omitting either stream, while long- and short-range probes support overall and hand-contact quality, respectively.Removing long-range probes hurts overall interaction quality; removing short-range probes mainly degrades hand contacts.Source: §4.2.2 · p. 9
• A frozen PamiRefiner improves all baseline generations from InterAct and LIGHT after their outputs are converted with inverse kinematics.The passage describes this as transferable fine-grained interaction-geometry reasoning.Source: §4.2.2 · p. 9, Table 4 · p. 9

5 CONCLUSION AND LIMITATIONS

• On InterAct, PAMI achieves 14.5% higher contact recall than the previous art and best performance in almost all metrics.The conclusion says ablations confirm contributions from the part-based latent, anchor voting, learned weights, and PamiRefiner.Source: §5 · p. 9
• PAMI uses per-frame weights to aggregate body-part votes for object motion, then refines a coarse latent-space interaction with hybrid surface sensing.The conclusion describes the approach as following the Hough-Transform idea.Source: §5 · p. 9
• PamiRefiner improves LIGHT and InterAct outputs without retraining.The conclusion states that the method will be evaluated in future work on interactions involving multiple objects only indirectly?Source: §5 · p. 9

A ADDITIONAL EXPERIMENTS

• Table 5 reports top-2/3 R-precision and diversity metrics omitted from the main table.These metrics complement Table 1's InterAct results.Source: Table 5 · p. 14, A ADDITIONAL EXPERIMENTS · p. 14

A.1 NUMBER OF REFINEMENT STEPS

• Most contact improvement occurs within four refinement updates; further updates add little contact gain and eventually worsen FID, so the experiments use K=4.Figure 5 plots full-body contact F1, FID, and hand-contact F1 across recursion depth, with means over three draws and 95% confidence intervals.Source: §A.1 · p. 14, Figure 5 · p. 15

A.2 RUNTIME AND MEMORY

• For a 231-frame sequence at batch size 1, generation takes 0.13 s and 5.7 GB; four refinement steps add 0.66 s and use 2.8 GB.The reported model sizes are 755M parameters for PamiVAE, 55M for PamiGen, and 4.4M for PamiRefiner.Source: §A.2 · p. 14

B.1 MOTION REPRESENTATION AND COORDINATES

• Motion is processed at 30 fps in windows of at most 300 frames, with global human and object motion expressed relative to the first-frame human root.The body representation uses global root orientation and translation, root-relative joint positions and velocities, parent-relative 6D rotations, and separately represented hands.Source: §B.1 · p. 15
• PamiVAE uses eight causal body-part encoders and an object encoder, producing 16-dimensional tokens after fourfold temporal downsampling.Six spatiotemporal-attention layers exchange information across tokens and time; a joint human decoder and two upsampling stages restore frame rate.Source: §B.1 · p. 15

B.2 ANCHOR VOTING IMPLEMENTATION

• Five body-part anchors and one free anchor vote for object translation, with body-part votes transforming decoded relative offsets and the free vote predicting translation directly.The target weights favor parts within 1 cm of the object; if no body-part contact is detected, the free branch receives the full vote.Source: §B.2 · p. 15
• Table 7 isolates representation choices in coarse outputs before refinement, while Table 8 gives additional metrics for refined representation ablations.Both captions report means and 95% confidence intervals over three draws or runs, respectively.Source: Table 7 · p. 16, Table 8 · p. 16

B.3 PAMIGEN: TEXT-CONDITIONED LATENT INTERACTION GENERATION

• PamiGen uses eight human-part tokens and one object token per latent step, with a 10-layer transformer attending to the full token grid and frozen caption features.Rotary embeddings encode time, token embeddings distinguish body parts and the object, and the object token also uses canonical-mesh basis-point-set features.Source: §B.3 · p. 16
• PamiGen uses classifier-free guidance at scale 2.5 for 50 integration steps, with decoded losses applied only for 0.1 < τ < 0.5.Its free-position, free-velocity, and composed-translation terms each have weight 5.0, while alignment has weight 2.5 and is evaluated only when a body-part vote is active.Source: §B.3 · p. 16

B.4 PAMIREFINER ARCHITECTURE AND ROLLOUT

• PamiRefiner uses 54 tokens per frame: 22 body joints, 30 finger joints, one object token, and one global token.Joint tokens contain 6D rotation and long-range probes; the global token contains root pose and a 16D body-shape feature.Source: §B.4 · pp. 16–17
• Table 9 reports full metrics for zero-shot refinement of LIGHT and InterAct outputs with K=4, averaged over two sampling campaigns.Bold marks the best result in each generator block.Source: Table 9 · p. 17
• Long-range probes use 512 surface points, while short-range sensing encodes up to 64 nearby points within 4 cm for each finger joint and fingertip.Short-range points come from a separate 16,384-point area-weighted sample; both probe types are recomputed after every refinement step.Source: §B.4 · p. 17
• Four spatial-temporal transformer blocks predict bounded updates to joint rotations, root motion, and object pose, trained through a three-step rollout.Object updates are capped at 5 cm and 10°, and root updates at 10 cm and 5° in yaw.Source: §B.4 · p. 17

B.5 PAMIREFINER TRAINING LOSSES

• Recovery and generation batches alternate 1:1 while the VAE and generator remain frozen, and recovery learns from corrupted VAE round trips supervised by the original interaction.Corruptions are sampled across clean, root, torso, upper-body, lower-body, object, and mixed groups.Source: §B.5 · p. 17
• Temporal noise follows an Ornstein–Uhlenbeck process with α ≈0.895, alongside specified root, body-part, and object perturbations.The corruption distribution stays fixed across the three training steps, and generated samples are precomputed.Source: §B.5 · p. 17
• The objective groups terms for contact, penetration, support, smoothness, and preservation, with all five group weights set to 1.Contact masks and segment anchors are computed from coarse generation and kept fixed during recursion.Source: §B.5 · p. 17
• Table 11 lists hyperparameters used by the final checkpoints, including parameter counts, optimizer schedules, and scalar loss weights.The values are read from the checkpoints’ recorded configurations.Source: §B.5 · p. 17

C LIMITATIONS AND FUTURE WORK

• Experiments cover the single-human, single-rigid-object setting of InterAct and assume a known canonical object mesh for shape conditioning and surface sensing.The authors note that this assumption may limit applicability when accurate object geometry is unavailable.Source: C LIMITATIONS AND FUTURE WORK · p. 17
• PamiRefiner is a bounded local corrector that can improve contact when the coarse interaction is already reasonable, but cannot recover from an incorrect action, severe temporal misalignment, or a large trajectory error.Source: C LIMITATIONS AND FUTURE WORK · p. 17
• Future work will extend the framework to multiple objects and improve generalization to novel object shapes beyond those observed during training.Source: C LIMITATIONS AND FUTURE WORK · p. 17