PaperlayerView exact revision on arXiv

Source-linked AI summary

Affordance Diffusion: Synthesizing Hand-Object Interactions

Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shubham Tulsiani, Sifei Liu

arXiv:2303.12538v3cs.CVcs.RO

TL;DR

Generating a plausible hand interaction from a particular object image is challenging because it calls for reasoning about affordances and visual interaction details. The paper uses LayoutNet and ContentNet to synthesize interactions in two steps. It reports favorable comparisons with baselines and generalization to new datasets, while noting limits tied to training-set coverage and scene conditions.

  • Problem

    Generating realistic hand–object interactions from a particular object image remains challenging because it requires modeling affordances, physical constraints, and visual interaction details.

    Source: §1 · p. 2

  • Method

    LayoutNet samples hand–object layouts from the object image, and ContentNet uses the predicted layout and object image to synthesize interaction images.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    The method outperforms generic image-generation baselines, its reconstructed hand poses are favored over directly predicted poses in user studies, and it generalizes to EPIC-KITCHEN without finetuning.

    Source: §1 · p. 2, §4.1 · p. 6

  • Takeaways & Limitations

    Synthesized interaction images provide a route to more plausible 3D affordance by reconstructing hand poses from the images.

    Source: §5 · p. 8

    The method can fail on cluttered scenes, extreme lighting, very large or very small objects, and hand viewpoints underrepresented in its training data.

    Source: §A.6 · p. 13

Abstract

from arXiv · show

Recent successes in image synthesis are powered by large-scale diffusion models. However, most methods are currently limited to either text- or image-conditioned generation for synthesizing an entire image, texture transfer or inserting objects into a user-specified region. In contrast, in this work we focus on synthesizing complex interactions (ie, an articulated hand) with a given object. Given an RGB image of an object, we aim to hallucinate plausible images of a human hand interacting with it. We propose a two-step generative approach: a LayoutNet that samples an articulation-agnostic hand-object-interaction layout, and a ContentNet that synthesizes images of a hand grasping the object given the predicted layout. Both are built on top of a large-scale pretrained diffusion model to make use of its latent representation. Compared to baselines, the proposed method is shown to generalize better to novel objects and perform surprisingly well on out-of-distribution in-the-wild scenes of portable-sized objects. The resulting system allows us to predict descriptive affordance information, such as hand articulation and approaching orientation. Project page: https://judyye.github.io/affordiffusion-www

1. Introduction

Generating plausible hand–object interactions from a particular object image remains challenging because it requires reasoning about affordances and visual interaction details. The paper separates interaction layout from hand articulation and reports stronger performance than generic generation baselines, alongside broader applications.

1. Introduction

  • Hand–object interaction synthesis is presented as useful for functional scene understanding, visual imitation, and manipulation.The paper lists synthesizing realistic interactions and plausible 3D poses, generalizing to out-of-distribution scenes, and using inpainting supervision with data augmentation among its contributions.Source: §1 · pp. 1–2
  • Image-generation models can produce realistic hand–object images from text, but conditioning generation on a particular object instance remains challenging.The paper distinguishes this task from the recent progress in high-resolution image generation and text-driven diffusion models.Source: §1 · p. 1
  • Generating a realistic interaction from an object image requires modeling affordances, physical constraints, hand–object occlusion, scale, lighting, and texture.For example, a hand should favor a kettle handle over a knife blade.Source: §1 · p. 2
  • The approach separates where to interact from how to interact: LayoutNet generates hand–object layouts, then ContentNet synthesizes interaction images conditioned on the object image and layout.The authors argue that interaction diversity largely arises from layout, while finger articulation depends on local object geometry.Source: §1 · p. 2
  • The method outperforms generic image-generation baselines, its reconstructed hand poses are favored over directly predicted poses in user studies, and it generalizes across datasets.The authors also report adaptation to new interactions with few examples and editing or guided generation from partially specified layouts.Source: §1 · p. 2

2. Related Work

Related work covers hand–object understanding, visual and 3D affordance prediction, and diffusion-based image editing. This paper instead synthesizes interactions from object images, reasoning about both layout and appearance.

2. Related Work

  • Prior hand–object work detects contact regions, reconstructs hand and object shapes, or locates key frames, whereas this work hallucinates possible hand interactions with an object.The review describes detection by bounding boxes [76] or segmentation, and shape reconstruction by template-based [6] [19] [83] [85] or template-free methods [10] [29] [39] [91].Source: §2 · p. 2
  • Visual affordance methods predict action categories or interaction heatmaps, while other work predicts richer properties; this paper directly synthesizes interaction images to show where and how to interact.The review characterizes affordance as functions that environments could offer.Source: §2 · p. 2
  • Whereas prior 3D hand-affordance methods typically use known object shapes to predict stable grasps, this work predicts coarse 3D grasps from general object RGB images.The paper reconstructs hand poses from synthesized interaction images rather than directly regressing poses from images [39].Source: §2 · p. 2
  • The method’s two diffusion models use LayoutNet to predict an interaction layout and ContentNet to synthesize the hand–object image from that layout and the object image.The modules are connected by an articulation-agnostic hand proxy.Source: Figure 2 · p. 3
  • Unlike style-transfer edits that mostly modify textures and style while preserving structures [47] [62], or content insertion into user-specified regions [1] [66] [72], affordance synthesis automatically reasons about both layout (structure) and appearance.Source: §2 · p. 3

3. Method

The method synthesizes plausible hand-object interactions through a coarse-to-fine process: LayoutNet predicts where to interact, then ContentNet generates how the hand appears. Both use conditional diffusion, and paired object-only and interaction images provide training supervision.

3.1. Preliminary: Diffusion models

  • Diffusion models learn generation through forward noising and a learned backward denoising process; here the DDPM loss reconstructs noise while conditioning on undiffused input.The method uses this conditional diffusion framework for both layout and visual-content generation.Source: §3.1 · p. 3, §3 · p. 3

3.2. LayoutNet: predicting where to grasp

  • LayoutNet predicts an articulation-agnostic hand proxy specifying grasp location, palm size, and approach direction, separating interaction layout from hand appearance.The proxy is splatted into image space so the model can reason about spatial relations between hand and object.Source: §3.2 · p. 4

3.3. ContentNet: predicting how to grasp

  • ContentNet conditions on the object image and predicted layout to synthesize a stochastic hand appearance, including finger articulation, wrist orientation, and occlusion.It is finetuned from a pretrained inpainting diffusion model in latent space, learning to preserve unoccluded object appearance while synthesizing the hand.Source: §3.3 · p. 4

3.4. Constructing Paired Training Data

  • Training pairs require pixel-aligned object-only and hand-object images; the method creates them by segmenting hands from egocentric crops and inpainting the hidden objects.SDEdit reduces inpainting artifacts but can blur or modify unmasked objects, so training mixes object images with and without SDEdit.Source: §3.4 · pp. 4–5, Figure 3 · p. 5
  • The authors collect 364k object-only and hand-object image pairs from HOI4D and call the resulting dataset HO3Pairs.Pairs undergo automatic sanity filtering, including checks that hands have been removed.Source: §3.4 · p. 5

4. Experiments

The experiments evaluate HOI image synthesis, extracted 3D hand poses, and applications including adaptation and guided generation. The method produces more plausible hand-object interactions, transfers to EPIC-KITCHEN without fine-tuning, and supports several forms of interaction control.

4. Experiments

  • Experiments train on HO3Pairs, evaluate on HOI4D and zero-shot EPIC-KITCHEN, and assess image synthesis with FID, contact recall, and user studies.HOI4D testing holds out object instances; EPIC-KITCHEN testing uses 500 object-only images.Source: §4 · p. 5, §4.1 · p. 5

4.1. Evaluating Image Synthesis

  • The method generates more plausible hand articulation and contact regions, achieves the best contact recall, and is favored most by users despite comparable FID.The same trends are reported for zero-shot EPIC-KITCHEN, including unseen object categories.Source: §4.1 · p. 6, Figure 4 · p. 6, Table 1 · p. 6

4.2. Evaluating Extracted 3D Hand Poses

  • Extracted poses are compatible with objects; unlike the baselines’ failure cases, the model can generate both left and right hands.GANHand can fail when grasp type is misclassified, while direct pose diffusion sometimes produces infeasible acute joint angles.Source: §4.2 · p. 7, Figure 5 · p. 7, Table 3 · p. 7

4.3. Application

  • With 32 samples, the model adapts to new HOI categories and performs better than a pretrained inpainting baseline.The comparison fine-tunes both models on three novel HOI4D categories and tests on novel instances.Source: §4.3 · pp. 7–8, Table 4 · p. 7
  • Users can edit hand location and orientation, guide layouts with sampled heatmap points without retraining, and transfer object-wise interactions into a scene with consistent hand scale.Changing approach direction can change the grasp from pinching a mug handle to a wider grip on its body.Source: §4.3 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8

5. Conslusion

The paper synthesizes hand-object interactions from an object image using LayoutNet and ContentNet, with generated images supporting reconstruction of more plausible 3D affordances. The authors note that generation quality and consistency between images and extracted 3D poses can be improved.

5. Conslusion

  • LayoutNet and ContentNet synthesize hand-object interactions from a given object image.Both are diffusion models for controllable, high-quality visual results.Source: §5 · p. 8
  • Reconstructing hand poses from synthesized interactions provides a shortcut to more plausible 3D affordance.Source: §5 · p. 8
  • Generation quality and consistency between synthesized images and extracted 3D poses can be further improved.Source: §5 · p. 8

Supplementary Materials

The supplementary material adds implementation details, ablations, robustness checks, baseline details, scene-level integration, limitations, and further qualitative results.

Supplementary Materials

  • The supplement provides LayoutNet implementation details and ContentNet ablations, along with additional qualitative results.Source: Supplementary Materials · p. 12
  • It examines paired-data construction robustness, documents baseline implementations, and details integration into scene-level affordance prediction.Source: Supplementary Materials · p. 12
  • The supplement discusses method limitations and includes visual results in an accompanying video.Source: Supplementary Materials · p. 12

A. Implementation Details

The implementation represents hand layout with location, size, and approach direction, transforms a canonical hand proxy into image space, and trains LayoutNet with diffusion-related losses and optional test-time constraints.

A. Implementation Details

  • Layout parameters encode hand location, palm size, and approaching direction, with ground truth derived from hand-prediction systems.Location and size come from a detected hand box; orientation and approach direction come from segmentation.Source: §A.1 · p. 12
  • A spatial transformer maps the canonical hand-mask template into image space using a similarity transformation determined by the layout parameters.The approaching-direction vectors are normalized before use in the transformation.Source: §A.1 · p. 12
  • The canonical proxy combines a Gaussian circle and rectangle, whose width reflects the training-set forearm-to-palm ratio.The circle uses standard deviation 1, and the rectangle uses standard deviation 2.Source: §A.1 · p. 12
  • LayoutNet uses weighted parameter and mask losses, and test-time guidance inserts noisy versions of specified constraints into diffusion steps.The unspecified constraint values are set to 0; the noise schedule is defined by α_t and its cumulative value ᾱ_t.Source: §A.1 · p. 12

A.2. ContentNet (Sec3.2)

The ContentNet appendix compares latent-diffusion and cascaded-diffusion variants. AffordDiff-LDM underlies the main paper’s quantitative results, while GLIDE has higher contact recall and is perceptually preferred despite a worse FID score.

A.2. ContentNet (Sec3.2)

  • AffordDiff-LDM implements ContentNet in latent space, following Rombach et al. [67], to generate realistic high-resolution 256² human–object interaction images conditioned on predicted layout and the input object image.Source: §A.2 · p. 13
  • AffordDiff-GLIDE generates 64² images first, then upsamples them by a factor of four using cascaded diffusion.Source: §A.2 · p. 13
  • AffordDiff-GLIDE achieves 90.8% contact recall versus 87.1% for AffordDiff-LDM, while AffordDiff-LDM scores 99.0 FID versus 121.6 for AffordDiff-GLIDE.The main paper’s quantitative results, including user studies and ablations, use AffordDiff-LDM.Source: §A.2 · p. 13
  • GLIDE is perceptually preferred for more realistic finger articulations, whereas LDM produces less blurry images with sharper, more realistic hand texture.Source: §A.2 · p. 13

A.3. Constructing Paired Training Data (Sec3.3)

The paired-data construction is applied to cluttered HOI images and to both HOI4D and EPIC-KITCHEN. Object crops use padding that helps the model learn relative hand-object scales.

A.3. Constructing Paired Training Data (Sec3.3)

  • Objects are cropped with 80% squared padding before resizing, so objects appear in similar sizes while hands appear in different sizes.This lets the model learn relative-scale priors, such as a smaller hand grasping a kettle than a mug.Source: §A.3 · p. 13
  • The paired-data construction is reported as robust to clutter and is applied to both HOI4D and EPIC-KITCHEN.The examples are shown in Figures 9 and 10.Source: §A.3 · p. 13
  • When an HOI image contains multiple hands, the construction randomly selects one hand to remove.Source: §A.3 · p. 13

A.4. Baselines Implementation

The appendix describes implementation choices for image-synthesis baselines and a crop-based approach to generating interactions for a selected object in a scene.

A.4. Baselines Implementation

  • Pix2Pix concatenates the predicted layout and object image as input to six ResNet blocks that output an HOI image.Its discriminator uses the object-only image, splatted layout, and generated HOI image.Source: §A.4 · p. 13
  • The VAE KL-loss weight is swept across five values, with 1e −3 selected for the highest contact recall; GANHand’s RGB grasp branch is finetuned for another 10k iterations.GANHand inputs unavailable in HOI4D are set to zeros, and additional reconstruction losses are used.Source: §A.4 · p. 13

A.5. Scene Integration

  • For scene-level prediction, detected object crops are padded with boundary values when expanded crops would include distracting neighboring objects.This crop choice allows interaction generation for the object of interest.Source: §A.5 · p. 13

A.6. Limitation and Failure Cases

The method’s generalization is limited when object appearance departs substantially from training data, and its training-set bias restricts some viewpoints and hand-entry directions.

A.6. Limitation and Failure Cases

  • The model may fail on appearances far from training data, including cluttered scenes, extreme lighting, and very large or very small objects.Source: §A.6 · p. 13
  • Training-set bias prevents the current model from generating hands entering from the top of the frame or from a third-person view.Source: §A.6 · p. 13
  • The authors identify consistency of hand appearance and extracted hand poses as areas for further improvement.Source: §A.6 · p. 13

B. Qualitative Results

The qualitative results include paired-data examples, comparisons of generated interactions and extracted poses, and demonstrations of editing, guided synthesis, and scene integration.

B. Qualitative Results

Related papers

Loading 2303.12538v3…