Source-linked AI summary
COUCH: Towards Controllable Human-Chair Interactions
Xiaohan Zhang, Bharat Lal Bhatnagar, Vladimir Guzov, Sebastian Starke, Gerard Pons-Moll
TL;DR
Existing human-scene interaction methods provide limited fine-grained control over where and how people contact objects. COUCH predicts contact-aware hand controls to synthesize contact-conditioned human-chair motion, and its experiments report improved contact accuracy alongside diverse, natural interactions.
Problem
Existing human-scene synthesis methods do not provide fine-grained control over intentional object contacts, although such contacts are important for varied human-chair interactions.
Method
COUCH plans hand motion with contact-aware control signals and uses them to synthesize full-body human-chair motion, supporting user-specified or automatically sampled contacts.
Results
COUCH improves average contact accuracy by approximately 55% over the state of the art while generating natural and diverse motions satisfying specified contacts.
Takeaways & Limitations
COUCH enables explicit control over how a person contacts a chair through interactive or generative contact selection.
Takeaways & Limitations
Synthesized motion can slightly intersect the chair, and generalization to more chair shapes requires better scene-geometry encoding without overfitting.
Abstract
from arXiv · showhide
Humans interact with an object in many different ways by making contact at different locations, creating a highly complex motion space that can be difficult to learn, particularly when synthesizing such human interactions in a controllable manner. Existing works on synthesizing human scene interaction focus on the high-level control of action but do not consider the fine-grained control of motion. In this work, we study the problem of synthesizing scene interactions conditioned on different contact positions on the object. As a testbed to investigate this new problem, we focus on human-chair interaction as one of the most common actions which exhibit large variability in terms of contacts. We propose a novel synthesis framework COUCH that plans ahead the motion by predicting contact-aware control signals of the hands, which are then used to synthesize contact-conditioned interactions. Furthermore, we contribute a large human-chair interaction dataset with clean annotations, the COUCH Dataset. Our method shows significant quantitative and qualitative improvements over existing methods for human-object interactions. More importantly, our method enables control of the motion through user-specified or automatically predicted contacts.
1 Introduction
COUCH addresses controllable human-chair motion by conditioning synthesis on intentional hand contacts and planning contact-aware movement. It introduces a contact-focused dataset and supports both user-specified and automatically sampled interactions.
- Motivation: Existing scene-interaction methods do not reason about intentional contacts or support user-provided contact control.
- Problem: COUCH synthesizes human-chair motion conditioned on contact positions, enabling variations such as different hand supports and interaction styles.
- Method: ControlNet predicts future limb control signals, while PoseNet uses them to synthesize motion that reaches the intended contacts.
- Runtime modes: COUCH operates interactively with user-specified contacts or generatively by automatically sampling intentional contacts on the chair.
- Dataset: The COUCH dataset contains 3 hours and over 500 sequences of human-chair motion with multiple subjects, registered chair geometry, RGB-D data, and annotated hand contacts.
- Contributions: The method is presented as the first controllable contact-based human-chair interaction approach and outperforms the state of the art qualitatively and quantitatively.
2 Related Work
Prior work covers scene-agnostic motion prediction, static affordances, and dynamic scene interactions, but lacks fine-grained controllable contact interaction. COUCH targets this gap for human-chair motion.
- Scene-agnostic motion prediction: Scene-agnostic motion predictors aim for realistic motion but are generally not conditioned on scene geometry.
- Static scene interactions: Affordance and hand-object methods model static poses or contacts but do not generate full human motion sequences with scene interaction.
- Dynamic scene interactions: Dynamic interaction methods address scene-aware motion, but NSM lacks motion variations for the same task and geometry, while SAMP does not provide interaction-style control.
- Research gap: COUCH focuses on controllable, fine-grained interactions specified by contacts on the object, a problem prior work had not addressed for human-chair interactions.
3 The COUCH Dataset
The COUCH Dataset provides multi-subject, motion-rich human-chair interactions with real chair geometry, RGB-D data, and annotated contacts. Its capture and annotation design supports studying varied, contact-conditioned interactions.
- The dataset contains 6 subjects, over 500 motion sequences, and 3 hours of human-chair interaction capture.
- It combines 17 wearable IMUs with Kinect-based capture to balance scalable motion capture and accuracy during human-object interactions.
- The registered human-model and chair-scan fitting errors are 3.12 cm and 1.70 cm, respectively, in Chamfer distance.
- The dataset includes multiple starting points and interaction styles, including armrest contact, sitting-plane contact, and no hand contacts.
- It contains three chair models and a sofa with varied shapes, plus augmentation intended to generalize synthesis to unseen objects.
- Contacts are annotated for five key joints—the pelvis, hands, and feet—from synchronized body-motion and object-pose capture.
4 Method
COUCH synthesizes chair interactions by separating motion planning from pose prediction and conditioning future motion on desired hand contacts. It supports interactive user-specified contacts and generative contact sampling while activating detailed hand planning near the chair.
- COUCH takes the current pose, chair geometry, and target hand contacts, then autoregressively predicts future poses satisfying those contacts.
- The method separates motion planning and motion prediction into ControlNet and PoseNet, using both networks for contact-constrained sitting.
- COUCH decomposes synthesis into approaching and sitting, activating ControlNet only during sitting when detailed hand planning is needed.
- ControlNet predicts future hand trajectories and local phases, where the phase progresses from 0 at movement onset to 1 near contact.
- The control signal predicts two-hand positions relative to desired contacts at 7 uniformly sampled timestamps within a 1-second future window.
- PoseNet combines ControlNet signals with past motion and scene information to predict full-body motion, including future poses, root trajectories, goals, and contacts.
- ContactNet uses a conditional variational auto-encoder to generate plausible hand contacts for a chair from sampled latent vectors during inference.
5 Evaluation
COUCH is evaluated for contact accuracy, control, motion diversity, and automatic contact generation. It outperforms contact-conditioned baselines, supports sequential contact goals, and generates diverse contacts on unseen chair shapes.
- 5.2 Evaluation on Control: COUCH evaluates contact accuracy on 120 testing sequences across six subject-specific models without post-processing.The evaluation measures whether synthesized motions reach the specified contacts and assesses contact-constrained synthesis on the COUCH dataset.
- 5.1 Baselines: Naively conditioning existing motion synthesis methods on contacts does not ensure that the generated motion satisfies those contacts.The paper attributes this limitation to contacts being ignored during autoregressive prediction, motivating explicit motion planning with trajectory predictors.
- 5.1 Baselines: COUCH generates motions that satisfy user-specified contacts better than NSM+Control and SAMP+Control.The evaluation compares contact accuracy using average contact error, average contact precision, and related control measures.
- 5.3 Evaluation on Motion Diversity: Explicit contacts enable COUCH to generate more varied approaching and sitting motions than the baselines under APD evaluation.Diversity is measured on synthesized pose features after randomized initialization, chair selection, and contact sampling.
- 5.4 Sequential Contacts: COUCH supports complex interactions by iteratively supplying new hand-goal locations after the current contact locations are reached.This enables sequences such as sitting with hands on the armrests, lifting them to support the head, and returning them to the armrests.
- 5.5 Contact Prediction on Novel Shapes: ContactNet predicts diverse hand-contact positions for unseen chairs, achieving an APD of 11.82 cm versus 14.07 cm for the ground-truth dataset.It predicts 10 contact positions for each hand on 200 unseen chairs, and the resulting contacts generalize qualitatively to unseen shapes.
6 Conclusion
COUCH synthesizes controllable human-chair interactions by planning hand motion toward specified contacts and using it to drive full-body poses. The paper contributes a three-hour, six-subject dataset and reports improved contact accuracy, naturalness, and control over prior methods.
- 6 Conclusion: COUCH plans hand motion toward chair contacts, which drives full-body poses to satisfy those contacts.The method is presented as the first approach for controllable contact-driven human-chair interaction.
- 6 Conclusion: The COUCH dataset contains three hours of motion capture from six subjects interacting with registered 3D chair models.It includes varied sitting motions involving different approaches and contacts.
- 6 Conclusion: COUCH improves average contact accuracy by approximately 55% over the state of the art and produces more natural motion than baseline methods.The conclusion also reports better control through contact-driven synthesis.
APPENDIX
The appendix provides additional dataset, implementation, and post-processing information while discussing current limitations as directions for future research.
- APPENDIX: The appendix covers dataset details, implementation details, post-processing techniques, and current limitations.It frames the limitations discussion as part of future research perspectives.
1 Dataset
The COUCH dataset combines motion capture, human-model fitting, temporal smoothing, foot-contact processing, and registered chair geometry. It includes three hours of motion capture and more than 500 motion sequences spanning different interaction types.
- Dataset: The dataset contains three hours of motion capture with more than 500 motion sequences across different interaction types.Table 4 breaks down the dataset by interaction type.
- SMPL Fitting: Human body models are fitted by combining RGB segmentation, multi-view depth, FrankMocap initialization, and instance-specific optimization.The pipeline lifts segmented human observations into 3D before fitting the SMPL model.
- Temporal Smoothing: Joint rotations are optimized for temporal smoothness using a loss involving successive rotation differences and joint acceleration.The acceleration term is approximated by central differences.
- Foot Contact Processing: Binary toe and heel contact labels are used to remove foot sliding by averaging joint positions during positive contact intervals.The resulting target positions are applied to all four foot joints.
- Object Processing: Chair meshes are fitted from pre-scanned objects using manually marked multi-view keypoints, after which segmentation masks are obtained by projection.This registers chair geometry with the captured frames.
2 Training Details
COUCH combines ControlNet, PoseNet, and ContactNet-related components to model controllable human-chair interactions. The training setup specifies recurrent control processing, mixture-of-experts pose prediction, and scene-conditioned contact generation.
- Architecture overview: Figure 7 depicts the method’s combination of ControlNet and PoseNet.The PoseNet architecture details are summarized in Table 5.
- ControlNet: ControlNet uses a two-layer LSTM to process pose and control signals, including hand trajectories and local phases.Each encoded input passes through two-layer fully connected networks with shape {128, 128} before the LSTM.
- PoseNet: PoseNet uses a mixture-of-experts structure with 10 experts and three-layer gating and prediction networks.The gating and prediction networks have hidden dimensions of 128 and 512, respectively.
- ContactNet: ContactNet encodes the scene with a three-layer network and uses a size-6 VAE latent vector for contact prediction.Its scene encoder has shape {512, 512, 64}, and the Kullback-Leibler divergence weight is β = 0.1.
3 Contact Projection and Trajectory Fitting
COUCH projects sufficiently close predicted contacts onto the chair surface while discarding contacts that fall too far away. Its ControlNet also predicts future hand trajectories for subsequent pose fitting.
- Contact projection: Predicted contacts within 10 cm of the chair surface are projected onto the nearest surface point.This post-processing step ensures accepted contacts land exactly on the object surface.
- Contact projection: Predicted contacts more than 10 cm from the surface are discarded.The threshold separates contacts eligible for projection from neglected predictions.
- Trajectory fitting: ControlNet predicts future hand trajectories that can be used to fit the predicted pose.The passage describes trajectory prediction as part of the contact and motion-processing pipeline.
4 Limitations and Future Direction
The authors identify collision artifacts, limited generalization across chair shapes, and unresolved subject-variant synthesis as limitations. They propose broader object coverage, including dynamic objects, as future work.
- Limitations: Synthesized motion can slightly intersect with the chair, motivating post-processing to avoid collisions.The authors also connect broader chair-shape generalization with better scene-geometry encoding and reduced over-fitting.
- Limitations: Different body shapes can intersect with the same object differently during the same motion, leaving subject-variant synthesis unresolved.The COUCH dataset captures different body shapes and supports studying body-shape conditioning.
- Future directions: Future work should extend contact-based control beyond static chairs to dynamic objects such as boxes and doors.The authors specifically mention lifting a box and opening a door as target interactions.