Unified Human-Scene Interaction via Prompted Chain-of-Contacts

1 INTRODUCTION

• UniHSI targets versatile language-controlled interaction and an accessible interface, gaps that remain despite progress in motion quality and physical plausibility.It defines interactions as ordered human-joint/object-part contact pairs and uses an LLM Planner and Unified Controller to plan and execute them.Source: §1 · p. 2
• LLM-generated interaction plans reduce annotation requirements, and ScenePlan contains thousands of plans based on PartNet and ScanNet scenarios.Experiments report effectiveness in versatile interaction control and generalization to real scanned scenarios.Source: §1 · p. 2

2 RELATED WORKS

• Prior physics-based HSI methods are described as relying on task-specific policies, coarse action control, or annotated motion; UniHSI uses a shared representation and LLM planning.Language-based alternatives also face challenges grounding target objects and contact areas or require predefined tasks and separate low-level policies.Source: §2 · p. 3

3.1 CHAIN OF CONTACTS

• Chain of Contacts (CoC) represents interactions as ordered steps containing human-joint/object-part contact pairs, including contact type and relative direction.Each pair specifies an object, object part, humanoid joint, contact state, and direction; a step may include several pairs.Source: §3.1 · pp. 3–4

3.2 LARGE LANGUAGE MODEL PLANNER

• The LLM Planner combines language commands with scene, object-part, and joint information plus prompted instructions to generate CoC plans.Prompt changes can request different numbers or types of plans, including plans generated for the given scenes.Source: §3.2 · p. 4

3.3 UNIFIED CONTROLLER

• The Unified Controller executes CoC as realistic, scene-coherent movements, while the TaskParser converts contact pairs into shared observations and objectives and fetches steps sequentially.It uses adaptive contact weights and an ego-centric heightmap for scene awareness; the controller builds on AMP's motion prior.Source: §3.3 · pp. 4–6

4.1 DATASETS AND METRICS

• ScenePlan combines PartNet objects and ScanNet rooms into generated plans, with 1,040 PartNet and 100 ScanNet evaluation plans spanning varied interactions, horizons, and object counts.Training uses 40 PartNet objects, each with 5–20 plausible interaction steps; each scenario randomly includes 1–4 objects. Success Rate measures completion of every plan step within the time limit, alongside contact error and Success Steps.Source: §4.1 · p. 7

4.2 PERFORMANCE ON SCENEPLAN

• ScenePlan performance declines as plans become more diverse and complex, although Success Steps rises; test scenarios are unseen, and ScanNet adds a modality gap.Plans are grouped as simple (up to 3 steps), medium (more than 3 steps with one object), and hard (multiple objects).Source: §4.2 · pp. 7–8

4.3.1 KEY COMPONENTS ABLATION

• GPT-4 outperforms GPT-3.5 on planning, but both lag behind human performance and can omit transitions or generate out-of-distribution actions.The comparison uses 100 language-command sets, with 10 tests per plan; additional prompt rules and GPT-4 can mitigate, but not eliminate, errors.Source: §4.3.1 · p. 8
• Adaptive Weights improve performance across task levels by downweighting unused or easily learned contact pairs and increasing weights for harder pairs.The weighting strategy becomes especially important as tasks grow more complex.Source: §4.3.1 · pp. 8–9
• Removing the Ego-centric Heightmap degrades performance, especially on difficult tasks, and weakens navigation and generalization to scanned scenes.The heightmap supports perception of surroundings and helps prevent collisions in complex scenes with many objects.Source: §4.3.1 · p. 9

4.3.2 DESIGN COMPARISON WITH PREVIOUS METHODS

• UniHSI outperforms or matches baseline implementations across metrics, with the clearest advantage on “Lie Down”; vanilla AMP combination drops across tasks.Figure 6(b) shows higher success rates and faster convergence than baseline implementations, while Figure 6(a) illustrates more natural and accurate “Sit” and “Lie Down” motions.Source: Table 3 · p. 8, Figure 6 · p. 8, §4.3.2 · p. 9
• Baseline objectives combine walking toward an object and task performance simultaneously, whereas UniHSI’s language-planned decomposition produces more natural task progression.The comparison accounts for differences in training data and unavailable code by reporting prior results and implementing a baseline with design elements from Hassan et al. (2023).Source: §4.3.2 · p. 9

5 CONCLUSION

• UniHSI represents interactions as sequences of human joint-object part contacts, translated from language commands by an LLM Planner and executed by a Unified Controller.The conclusion characterizes the system as supporting diverse interactions and language commands.Source: §5 · p. 9

A LIMITATIONS AND FUTURE WORK

• UniHSI controls interactions with fixed objects only; movable-object interaction and integrating the LLM into training remain future directions.The current design uses pre-generated plans.Source: A LIMITATIONS AND FUTURE WORK · p. 10

B IMPLEMENTATION DETAILS

• The policy uses CNN and MLP layers, the discriminator uses MLP layers, and PPO training uses Adam at a 2e-5 learning rate.Experiments run in IsaacGym with one Nvidia A100 GPU and 8,192 parallel environments.Source: B IMPLEMENTATION DETAILS · p. 10

C DETAILED PROMPTING EXAMPLE OF THE LLM PLANNER

• UniHSI models joint-to-joint interactions like interactions with objects, replacing an object's part point cloud with a joint position.The passage identifies this as a design choice.Source: C DETAILED PROMPTING EXAMPLE OF THE… · p. 10
• Contact-free walking is represented as a pseudo contact between the pelvis (root) and the target place point.The plan representation uses {object, none, none, none, direction} for these cases.Source: C DETAILED PROMPTING EXAMPLE OF THE… · p. 10
• The authors propose collecting language commands and integrating ChatGPT and GPT into the loop to evaluate the full UniHSI framework in future work.Source: C DETAILED PROMPTING EXAMPLE OF THE… · p. 10

D DETAILS OF THE SCENEPLAN

• Hard-level ScenePlan plans contain more than three steps and more than one object, with plans recording object information and contact-pair steps.Simple plans have up to three steps with one object; medium plans have more than three steps with one object.Source: D DETAILS OF THE SCENEPLAN · p. 11
• ScenePlan's interaction-type lists are provided in Tables 11 and 12, and multi-agent interaction is currently limited to the commands level.Source: D DETAILS OF THE SCENEPLAN · p. 11, Figure 9 · p. 12

E MORE VISUALIZATIONS

• Additional quantitative results are presented in Figures 7, 8, and 9.Source: E MORE VISUALIZATIONS · p. 12

F DEMONSTRATION OF FAILURE PLANNING

• Planning failures include errors in spatial relations, attributed mainly to suboptimal spatial-context understanding, and plans for object manipulation that are difficult to execute at this stage.Source: F DEMONSTRATION OF FAILURE PLANNING · p. 12, Table 5 · p. 13

G USER STUDY ON MOTION REALITY.

• The user study rates motion naturalness and semantic faithfulness on 0-to-5 scales, with higher semantic-faithfulness scores indicating closer alignment with the input.Results are presented in Table 6.Source: G USER STUDY ON MOTION REALITY · p. 13, Table 6 · p. 13
• Quantitative evaluation remains challenging at this stage and requires further exploration.Source: G USER STUDY ON MOTION REALITY · p. 13

• Figure 1 illustrates a multi-object interaction scenario.Source: Figure 1 · p. 22

A LIMITATIONS AND FUTURE WORK.

• Current control is limited to fixed objects; movable-object interaction and integrating the LLM into training remain future directions.The current design uses pre-generated plans.Source: A LIMITATIONS AND FUTURE WORK · p. 22

B IMPLEMENTATION DETAILS

• The controller uses policy and discriminator networks, with PPO for policy training and Adam at a 2e-5 learning rate.Experiments use IsaacGym, one Nvidia A100 GPU, and 8192 parallel environments.Source: B IMPLEMENTATION DETAILS · p. 22

C DETAILED PROMPTING EXAMPLE OF THE LLM PLANNER

• The prompting example treats joint-joint interaction like object interaction and represents contact-free walking as a pseudo contact from the pelvis to a target place.Figure 2 illustrates a multi-step interaction involving the same object.Source: C DETAILED PROMPTING EXAMPLE OF THE… · p. 22, Figure 2 · p. 23

D DETAILS OF THE SCENEPLAN

• ScenePlan examples distinguish simple plans of up to three steps with one object, medium plans exceeding three steps with one object, and hard plans exceeding three steps with multiple objects.The appendix also lists interaction types in Tables 7 and 8.Source: D DETAILS OF THE SCENEPLAN · p. 23

E MORE VISUALIZATIONS

• The appendix points to Figures 1, 2, and 3 for additional quantitative results.Source: E MORE VISUALIZATIONS · p. 23

F DEMONSTRATION OF FAILURE PLANNING

• Table 1 demonstrates cases where LLM-generated plans are inaccurate.Source: F DEMONSTRATION OF FAILURE PLANNING · p. 23
• Planning failures can involve incorrect spatial relations, which the paper attributes mainly to suboptimal grasp of spatial contexts.Source: F DEMONSTRATION OF FAILURE PLANNING · p. 23
• LLMs may also devise plans involving object manipulation, which are difficult to execute successfully at the current stage.Source: F DEMONSTRATION OF FAILURE PLANNING · p. 23

G USER STUDY ON MOTION REALITY.

• The user study rates motion naturalness and semantic faithfulness on 0-to-5 scales, with higher semantic-faithfulness scores indicating closer alignment with the input.Source: G USER STUDY ON MOTION REALITY · p. 23, Table 2 · p. 25
• The paper states that quantitative evaluation is challenging at this stage and requires further exploration.Source: G USER STUDY ON MOTION REALITY · p. 23
• The illustrated multi-agent interaction is currently limited to the commands level.Source: Figure 3 · p. 24