MAMHOI: Factorizing Scene-Aware Human-Object Interaction through Affordances

• Scene-aware HOI needs both environmental feasibility reasoning and realistic interaction dynamics, but these capabilities rarely have joint large-scale supervision.Human-scene data capture environment-aware behavior, while human-object data capture synchronized interaction dynamics; prior approaches leave open how to connect them across a distribution gap.Source: §1 · pp. 1–2
• MAMHOI separates scene understanding from human–object motion synthesis, using motion affordance as an explicit interface between them.Source: §1 · p. 2
• The model predicts affordance from scene geometry, object-motion cues, and language, then uses local occupancy representations to condition a diffusion-based motion generator.Planning provides coarse object-level structural cues rather than determining the resulting human motion.Source: §1 · p. 2
• The motion affordance conveys task-relevant environmental constraints without restricting generation to one prescribed trajectory.Source: §1 · p. 3
• Across complex indoor scenes, MAMHOI reduces object–scene penetration while preserving high-quality human–object interaction dynamics.Source: §1 · pp. 2–3

2.1 Human-Centric Motion Generation

• Human-centric motion work spans HOI methods for coordinated manipulation and HSI methods conditioned on static scene geometry; Afford-Motion uses a scene-affordance field but targets body–scene proximity.MAMHOI's motion affordance instead represents feasible spatial support for joint human–object interaction and transfers scene feasibility to a scene-free HOI generator.Source: §2.1 · p. 3

2.2 Scene-Aware Human-Object Interaction Generation

• Scene-aware HOI methods use planned waypoints or grasp anchors, while synthetic-pair and hybrid-training approaches address scarce paired data but retain trajectory-adherence or feasibility–interaction trade-offs.Source: §2.2 · p. 3
• MAMHOI represents scene constraints through motion affordance and learns scene feasibility separately from human–object motion, preserving the interaction distribution learned from dedicated HOI data.Its scene-to-affordance factor uses indoor HSI data, while affordance-to-motion uses scene-free HOI data.Source: §2.2 · pp. 3–4, Figure 2 · p. 4

3.1 Problem Formulation

• The formulation factors scene-aware motion through an affordance, treating detailed scene conditions as approximately redundant for motion synthesis once that affordance is given.This separation allows scene reasoning and interaction synthesis to be learned independently from human-scene and human-object data.Source: §3.1 · pp. 4–5

3.2 Data Representation

• Each interaction segment represents human and object motion, object geometry, sparse motion conditions, and a scene-aware affordance condition.Human and object motion share a canonical human-local frame; object geometry uses a Basis Point Set descriptor.Source: §3.2 · p. 5

3.3.1 Scene Understanding Model

• The predicted affordance is lifted into 3D and queried around the pelvis, object, and goal to form three 32 × 32 × 32 local voxel grids.The concatenated voxels condition the motion diffusion backbone.Source: §3.2 · p. 5, §3.3.1 · p. 6
• The scene model predicts a binary 2D interaction affordance from scene depth, a planner-provided object path, and language, with supervision from scene-aware motion trajectories.A residual 2D U-Net and cross-attention condition spatial scene features on path and text; auxiliary losses emphasize path regions and coverage.Source: §3.3.1 · pp. 5–6

3.3.2 Affordance-Conditioned Motion Diffusion Backbone

• The motion denoiser uses noisy motion and motion, text, geometry, and affordance features; ViT-encoded local voxels enter the backbone through cross-attention.Training on scene-free HOI data uses affordances derived from observed motion, while inference uses the scene model’s predicted affordance.Source: §3.3.2 · p. 6
• Generation proceeds autoregressively, re-querying local affordance voxels as the human pelvis and object positions update between segments.The goal condition determines the goal-centered query.Source: §3.3.2 · p. 6
• A positive SDF anchor penalizes joints outside or near the affordance boundary, while hard-negative ranking distinguishes feasible motion from alternative modes.The ranking compares trajectories with similar start and goal configurations but different intermediate motion; it supplies relative supervision when modes are insufficiently separated.Source: §3.3.2 · pp. 6–7
• Training adds clean-motion, semantic-contact, and forward-kinematics losses; optional test-time scene-SDF guidance penalizes penetration using proxy body joints.Weak classifier-free guidance strengthens scene conditioning while limiting deviation from the learned human–object motion distribution.Source: §3.3.2 · p. 7

4.1 Evaluation Settings

• Evaluation uses LINGO for scene understanding and OMOMO for the motion backbone; the constructed scene-aware test set comprises about 820 trials across 20 scenes.The trials use 164 planned object paths and five movable objects per path; CHOIS is retrained with 30-frame windows for comparison.Source: §4.1 · pp. 7–8

4.2 Evaluation Metrics

• Interaction-only metrics cover distributional similarity, text-motion alignment, feature distance, and diversity; scene-aware metrics assess penetration, human-motion jerk, path adherence, and contact.Lower is better for all reported physical metrics except contact rate, Rcon.Source: §4.2 · p. 8

4.3 Results on OMOMO Validation

• Without scene-specific guidance, MAMHOI improves FID, R-precision, MMD, and diversity over CHOIS, while InfBaGel has strong retrieval performance and the highest diversity under its pelvis-constrained setting.The OMOMO validation results assess interaction and motion generation quality rather than scene-aware physical plausibility.Source: §4.3 · p. 8, Table 1 · p. 9

4.4 Results on Scene-Aware Test Set

• On the constructed scene-aware test set, MAMHOI reduces human–scene and object–scene penetration while improving object-path adherence and human–object contact over CHOIS and InfBaGel.Unguided MAMHOI also outperforms CHOIS + Gsdf on all reported human–scene and object–scene measures, motion jerk, object-path error, and contact.Source: Table 2 · p. 9, §4.4 · p. 9
• Applying Gsdf reduces human–scene penetration for both CHOIS and MAMHOI; for MAMHOI it yields the lowest overall human–scene penetration, with small trade-offs in object–scene penetration and path adherence.InfBaGel uses its native Bump-aware Guidance rather than additional Gsdf refinement.Source: §4.4 · p. 9, Figure 3 · p. 10
• In the perceptual study, participants preferred MAMHOI over CHOIS in 86.7% of votes and over CHOIS+Gsdf in 92.3%.MAMHOI+Gsdf was preferred in 81.1% and 84.3% of votes, respectively.Source: §4.4 · p. 10

5 Conclusion

• MAMHOI improves scene feasibility, particularly by reducing object–scene penetration, while preserving high-quality human–object interaction dynamics in complex indoor environments.Its affordance-mediated factorization separates scene-level interaction feasibility from detailed human-object motion synthesis without requiring paired human-object-scene data.Source: §5 · p. 10

A Derivation of the Factored Generation Objective

• The objective factors generation by first sampling an affordance and then generating motion conditioned on that affordance and the task specification.The derivation introduces sequence-level A and marginalizes over it to obtain the factored objective.Source: A Derivation of the Factored… · p. 13
• The derivation assumes that affordance summarizes scene-dependent spatial constraints, making the remaining scene condition approximately redundant once A is given.It also introduces a coarse task condition containing only components needed for affordance prediction.Source: A Derivation of the Factored… · p. 13
• Substituting these approximations into the marginalization yields the factored generation objective.The resulting expression combines motion generation conditioned on A and the task with affordance prediction conditioned on the coarse task.Source: A Derivation of the Factored… · p. 13

B.1.1 Affordance Training Objective

• The denoising network predicts the clean affordance from noisy affordance conditioned on scene depth, planner-provided path, and text embedding.The complete training objective sets λBCE = 0.1 and λpath = 0.2 in all experiments.Source: §B.1.1 · p. 14
• Path-weighted binary cross-entropy gives stronger supervision near the planner path while retaining pixel-wise supervision over the full affordance map.The distance-decaying weights use τ = 3 and maximum weight 6.Source: §B.1.1 · p. 14
• A path-coverage term encourages planner-provided path points to lie inside predicted affordance, emphasizing the start and goal regions.Start and end points use ωj = 4; remaining path points use ωj = 1.Source: §B.1.1 · p. 14

B.2.1 Additional Training Objectives and Test-Time Guidance

• Motion supervision combines clean-motion reconstruction and semantic-contact prediction with forward-kinematics consistency for human end effectors and object keypoints.Hand–object contact and feet–floor guidance are retained during diffusion sampling; optional scene-SDF refinement is disabled for default results.Source: §B.2.1 · pp. 14–15
• Test-time hand–object and feet–floor guidance is applied during diffusion, while optional scene-SDF guidance refines pelvis and foot positions against scene geometry.Scene-SDF guidance is disabled for the default model.Source: §B.2.1 · p. 15

C.1 Data and Preprocessing

• Scene understanding uses LINGO data, while the affordance-conditioned motion generator uses synchronized human–object sequences from OMOMO, divided into 30-frame windows.Object geometry is represented with a 1,024-vector BPS descriptor.Source: §C.1 · pp. 15–16

C.2 Scene Understanding Model

• The scene model predicts a 256 × 256 binary affordance from scene geometry, an object path, and language, using cross-modal conditioning and path-weighted training losses.At inference, its probability map is thresholded at 0.5.Source: §C.2 · p. 16

C.3 Affordance Interface and Motion Generator

• Three local voxel volumes around the pelvis, object, and goal form the affordance condition for a diffusion generator that synthesizes human–object motion.The affordance feature enters the motion network through a cross-attention adapter.Source: §C.3 · p. 17

C.4 Motion Training Strategies

• Training balances relative human–object orientations, upweights frames around interaction onset, and ranks alternative intermediate trajectories by affordance violation.The ranking pairs have similar relative start and goal configurations but substantially different intermediate trajectories.Source: §C.4 · pp. 17–18

C.5 Training and Inference

• The LINGO scene-to-affordance model and OMOMO affordance-to-HOI generator are trained independently, avoiding paired human–object–scene manipulation data.The scene model learns the scene-to-affordance mapping, while the generator learns the affordance-to-HOI mapping.Source: §C.5 · p. 18
• Long interactions use overlapping 30-frame windows with updated local affordance queries; blending preserves temporal continuity, while weak scene guidance limits deviation from learned interaction motion.The guidance strength is s ∈ [1.0, 1.3].Source: §C.5 · p. 18

D.1 Ablation of Voxel Coordinate Frame

• Human-aligned voxel canonicalization improves all reported metrics without test-time guidance, especially human–scene penetration and motion jerk; SDF guidance further reduces penetration and improves contact.The guided human-aligned model achieves the strongest human–scene results, while the unguided variant retains slightly better results on several object–scene and human–object penetration metrics.Source: §D.1 · p. 19, Table 3 · p. 19

D.2 Ablation of Affordance-Aware Motion Ranking

• Ranking compares motions with similar start and goal configurations but different intermediate trajectories, encouraging lower affordance-violation cost for the positive motion.The objective uses the positive ground-truth affordance’s signed distance field.Source: §D.2 · p. 19
• Lrank reduces human–scene and object–scene penetration, lowers motion jerk, and raises contact ratio, with trade-offs in object-path adherence and human–object penetration.The ranking-loss ablation is evaluated both with and without test-time SDF guidance.Source: §D.2 · pp. 19–20, Table 4 · p. 20
• Test-time SDF guidance is complementary to Lrank and further reduces human–scene penetration.Source: §D.2 · p. 20

D.3 Results on the TRUMANS-Based Test Set

• TRUMANS contains paired scene and motion information, so this auxiliary evaluation does not establish the advantage of unpaired training.The experiment instead tests whether factorization preserves scene-aware and interaction-aware information when the factors are entangled.Source: §D.3 · pp. 20–21
• The controlled comparison uses 30 planned motion sequences across 20 scenes, a smaller test set retained because HOSIG inference makes larger-scale evaluation impractical.Source: §D.3 · p. 20
• MAMHOI achieves the best frame-averaged human–scene penetration, penetration rate, and motion jerk, while remaining close to Lingo in sequence-level vertex penetration.Source: §D.3 · p. 20, Table 5 · p. 21
• MAMHOI has the lowest available object–scene penetration and the smallest object path deviation in the TRUMANS comparison.Source: §D.3 · p. 20, Table 5 · p. 21
• HOSIG has the highest human–object contact ratio, while CHOIS has the lowest human–object penetration; HOSIG uses sparse object keyframes followed by post-processing.Source: §D.3 · pp. 20–21

E User Study Details

• The study evaluates whether generated human–object interactions appear physically plausible and natural in complex indoor scenes, complementing geometry-based evaluation.Participants compare generated sequences side by side and select the more plausible and natural interaction.Source: E User Study Details · p. 22
• 136 pairwise comparisons cover four pairings of Ours and CHOIS, with or without SDF guidance, split into eight tracks of 17 trials each.Both sequences in each comparison use the same task and scene configuration.Source: E User Study Details · p. 22
• Approximately 48 participants produced 52 completed track-level responses, yielding 884 valid pairwise judgments.Some participants completed more than one track.Source: E User Study Details · p. 22
• Preferences are aggregated separately for each method pairing as the percentage of judgments favoring each method, with results reported in Table 6.Method A and Method B are table labels, not fixed left–right positions; placement was randomized for each trial.Source: E User Study Details · p. 22, Table 6 · p. 22