PaperlayerView exact revision on arXiv

Source-linked AI summary

Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation

Yuanchen Ju, Kaizhe Hu, Guowei Zhang, Gu Zhang, Mingrun Jiang, Huazhe Xu

arXiv:2401.07487v1cs.ROcs.CV

TL;DR

Robots lack the interaction experience needed to generalize affordances to unseen objects, although human-object videos offer a resource for learning interactions. Robo-ABC builds video-derived affordance memory, retrieves similar objects, and maps their contact points to novel objects; it reports a 31.6% success-rate improvement over previous end-to-end methods and real-robot success across object categories.

  • Problem

    Existing video-based affordance methods can predict for familiar objects but struggle to generalize to unseen objects, while robots lack innate interaction experience.

    Source: §1 · p. 2

  • Method

    Robo-ABC stores interaction experience from human videos, retrieves visually or semantically similar objects, and uses diffusion-model correspondence to map their contact points to novel objects.

    Source: §1 · p. 2

  • Results

    31.6% improvement in success rate over previous end-to-end methods is reported, alongside 85.7% real-robot success over 7 object categories.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Robo-ABC demonstrates unsupervised zero-shot generalization using a small-scale memory without manual annotation or additional training.

    Source: §1 · p. 2, §6 · p. 8

    Affordance outputs require visual usability checks, and low-resolution data or lighting problems can affect correspondence accuracy.

    Source: D. Limitations · p. 14

Abstract

from arXiv · show

Enabling robotic manipulation that generalizes to out-of-distribution scenes is a crucial step toward open-world embodied intelligence. For human beings, this ability is rooted in the understanding of semantic correspondence among objects, which naturally transfers the interaction experience of familiar objects to novel ones. Although robots lack such a reservoir of interaction experience, the vast availability of human videos on the Internet may serve as a valuable resource, from which we extract an affordance memory including the contact points. Inspired by the natural way humans think, we propose Robo-ABC: when confronted with unfamiliar objects that require generalization, the robot can acquire affordance by retrieving objects that share visual or semantic similarities from the affordance memory. The next step is to map the contact points of the retrieved objects to the new object. While establishing this correspondence may present formidable challenges at first glance, recent research finds it naturally arises from pre-trained diffusion models, enabling affordance mapping even across disparate object categories. Through the Robo-ABC framework, robots may generalize to manipulate out-of-category objects in a zero-shot manner without any manual annotation, additional training, part segmentation, pre-coded knowledge, or viewpoint restrictions. Quantitatively, Robo-ABC significantly enhances the accuracy of visual affordance retrieval by a large margin of 31.6% compared to state-of-the-art (SOTA) end-to-end affordance models. We also conduct real-world experiments of cross-category object-grasping tasks. Robo-ABC achieved a success rate of 85.7%, proving its capacity for real-world tasks.

1. Introduction

Robots struggle to generalize affordances to unseen objects, despite the availability of human-object videos that capture interaction experience. Robo-ABC retrieves related objects from video-derived memory and transfers their contact points, improving reported success rates in zero-shot and real-world cross-category settings.

1. Introduction

  • Existing video-based affordance methods can predict for familiar objects but struggle with unseen objects, while robots lack innate interaction experience.Human-object videos offer interaction insights, including temporal and motion context.Source: §1 · p. 2
  • Robo-ABC retrieves visually or semantically similar objects from human-video interaction memory, then uses diffusion-model semantic correspondence to map contact points to novel objects.The transferred affordance points guide selection from AnyGrasp’s grasping-position prior for robot deployment.Source: §1 · p. 2
  • The framework transfers interaction experience from human videos without manual annotation, additional training, or pre-coded knowledge.The paper evaluates zero-shot and cross-category generalization and reports real-robot experiments.Source: §1 · p. 2
  • 31.6% improvement in success rate over previous end-to-end methods is reported for zero-shot affordance generalization.Real-robot experiments also report an 85.7% success rate over 7 object categories.Source: §1 · p. 2

2. Related Works

Prior work learns affordances from human videos and studies semantic correspondence and cross-category manipulation, but often remains limited by familiar instances, viewpoints, training, or segmentation requirements. Robo-ABC emphasizes video-derived interaction memory and foundation-model correspondence for transfer beyond seen categories.

2.1. Learning visual affordance from human videos

  • Video-based affordance work learns visual representations, grasp priors, and dexterous skills, while contact-region prediction methods [60] remain tied to in-domain instances and viewpoints.Robo-ABC instead uses a small interaction memory to support transferable manipulation in new scenes.Source: §2.1 · p. 2
  • Robo-ABC distinguishes itself by capturing semantic correspondence beyond seen object categories to guide zero-shot manipulation.Human-object videos provide temporal and motion context for understanding interactions.Source: §2.1 · pp. 2–3

2.2. Semantic correspondence for robotics

  • Prior robotic correspondence methods [6] [15] are described as limited to same-category instances, additional training, or user-provided goal images.Robo-ABC explores correspondence knowledge embedded in foundation models, avoiding additional training or part segmentation.Source: §2.2 · p. 3
  • Diffusion-model features are described as versatile for mapping similar points across categories, motivating their use for zero-shot transfer.The paper presents this foundation-model approach as off-the-shelf visual guidance for manipulation.Source: §2.2 · p. 3

2.3. Generalizable robot manipulation

  • Cross-category point-cloud methods typically require substantial annotated data or effective object-part segmentation.This contrasts with Robo-ABC’s motivation to use foundation models for affordance generalization beyond categories.Source: §2.3 · pp. 3–4
  • Some foundation-model and NeRF manipulation agents [15] operate on known instances, with limited generalization to novel instances.Figure 3 visualizes inferred contact points as the span of object categories increases.Source: §2.3 · p. 4, Figure 3 · p. 4

3. Method

Robo-ABC extracts contact-point affordances from human videos, retrieves similar objects from memory, and transfers their contact points to unfamiliar objects for manipulation. The method then uses the transferred points to guide a robot grasp.

3.1.1 Affordance representation

  • Robo-ABC defines affordance as human–object contact points and samples them from the hand–object intersection in the first frame where a grasp is detected.A hand–object detector identifies contact and provides hand and object bounding boxes; skin segmentation locates the intersection area.Source: §3.1.1 · p. 4

3.1.2 Contact points mapping

  • Contact points from an obscured contact frame are mapped to a clear, nearby frame using a homography between consecutive video frames.The method selects a sharp, unobstructed frame within a window around the contact frame, then crops the object for memory.Source: §3.1.2 · p. 4

3.2. The most similar object retrieval from memory

  • For retrieval, Robo-ABC compares CLIP image features using cosine similarity, searching within the same category for seen objects and across all categories for new ones.The cropped target image and memory images are encoded with the same image encoder.Source: §3.2 · p. 5

3.3. Semantic correspondence mapping for affordance generalization

  • Semantic correspondence maps retrieved contact points to the target object using diffusion-model features extracted from source and target images.The features are extracted during denoising after noise is added to the target image; eight source rotations and flips address orientation mismatches, with the highest-similarity result selected.Source: §3.3 · p. 5

3.4. Deployment in the real world

  • For deployment, AnyGrasp generates 7-DOF grasp candidates from the scene point cloud, and the candidate nearest the transferred contact point becomes the robot’s grasp pose.The selected pose guides the robot’s manipulation task.Source: §3.4 · p. 5

4. Experiments

The experiments evaluate Robo-ABC’s zero-shot affordance generalization, including cross-category transfer and real-world grasping. They define a manipulation-focused dataset and contact-point metrics, then compare Robo-ABC with prior approaches and test it on a robot.

4.1.1 Evaluation Dataset

  • The evaluation set uses manipulation-feasible objects from AGD20K’s “hold” category, excludes objects too large to grasp, and adds doors and drawers.“Seen” and “unseen” indicate whether objects of that category appear in the affordance memory.Source: §4.1.1 · p. 6, Table 1 · p. 6

4.1.2 Evaluation Metrics

  • Success Rate measures predicted points inside ground-truth masks and is the primary metric; NSS evaluates normalized ground-truth saliency at predicted points, while DTM measures distance to the mask.The success-rate threshold is 122; DTM is normalized by image-diagonal length.Source: §4.1.2 · p. 6

4.1.3 Baselines and results

  • 60.7% success rate, 31.6% higher than the second-best method, LOCATE; Table 2 also reports Robo-ABC surpassing all baselines on three metrics.The reported NSS and DTM are significantly better than other methods, with results closer to the center of the ground-truth mask.Source: Table 2 · p. 6, §4.1.3 · p. 6

4.2. Cross-Category Affordance generalization

  • With one source category fixed, Robo-ABC maps its human-video contact points to target objects across categories, including a fork handle to a flower stem.Source-to-target similarity decreases across the examples; Figure 5 reports that Robo-ABC has superior success rates in the vast majority of evaluated categories.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3. Real-World robot experiment

  • 85.7% success rate across seven object categories, compared with the baseline’s 68.6%, in real-world grasping experiments.The setup pairs Robo-ABC’s contact point with grasp candidates from AnyGrasp, selecting the pose with the least translational distance to that point.Source: §4.3 · p. 7

5. Ablation Study

The ablations compare retrieval and correspondence choices, memory size, and the number of retrieved images. CLIP-B32 and DIFT perform best among the tested encoders and correspondence models, while memory-size and retrieval-count effects vary by model and inference-time cost.

5. Ablation Study

  • The ablation varies memory category count, retriever encoder, retrieved-image count, and semantic-correspondence model.Source: §5 · p. 7

5.1. Retriever & Semantic correspondence model

  • CLIP-B32 performs best among the tested retrievers, while DIFT performs best among correspondence models in Table 3.CLIP-B32+LPIPS is slightly weaker, and DINO-VIT is the weakest correspondence model.Source: §5.1 · p. 7, Table 3 · p. 8

5.2. Memory size of seen categories

  • DIFT and SD-DINO improve as seen categories increase, while DINO-VIT peaks at 36 categories and then declines.The experiment holds the retriever encoder fixed, using CLIP-B32, to examine memory size.Source: §5.2 · p. 8, Table 4 · p. 8

5.3. Number of retrieved images

  • More retrieved images improve performance across the three tested models, but may lengthen inference time.The highest-DIFT-similarity image supplies the predicted contact point.Source: §5.3 · p. 8

6. Conclusion

Robo-ABC builds an affordance memory from human videos, retrieves visually and semantically similar objects, and uses semantic correspondence to transfer affordances. The appendix describes end-to-end baselines and how their outputs are adapted for contact-point evaluation.

6. Conclusion

  • Robo-ABC combines a human-video affordance memory, visual and semantic retrieval, and diffusion-model-based correspondence for zero-shot generalization.The conclusion states that this requires no manual annotation, additional training, or human prior.Source: §6 · p. 8

A.1. End-to-end Affordance Model

  • LOCATE is characterized as one-shot because it needs a source image, unlike Robo-ABC’s zero-shot approach.Source: §A.1 · p. 12
  • Most end-to-end baselines predict contact heatmaps, which the evaluation converts to their top five points; LOCATE instead requires an action label.The evaluation focuses on object contact rather than post-contact hand trajectory.Source: §A.1 · p. 12

A.2. Semantic Correspondence Methods

The section describes zero-shot semantic correspondence approaches and their roles in Robo-ABC. It contrasts dense feature matching with special-token optimization and selects DIFT for its balance of accuracy and speed.

A.2. Semantic Correspondence Methods

  • Semantic correspondence maps a pixel in a source image to a semantically similar pixel in a target image.The section notes that this can extend across object categories and that foundation visual models can do it zero-shot.Source: §A.2 · p. 12
  • Dense feature methods match source and target pixel features by cosine similarity, requiring no training and running in about 30s on an A40 GPU.SD-DINO combines DINO and DIFT features.Source: §A.2 · p. 12
  • LDM-SC optimizes a special token to attend to the source pixel, then uses its target-image attention to identify the matching pixel; each mapping takes about 5 min on an A40 GPU.Source: §A.2 · p. 12
  • Robo-ABC selects DIFT for semantic correspondence because it has the most balanced accuracy and inference speed.Source: §A.2 · p. 12

B. Implementation Details

The implementation builds an affordance buffer from EpicKitchens-100 videos and samples contact points from hand-object interaction. It filters video quality and adjusts sampled points to keep them on the object.

B.1. Selection of Affordance Buffer

  • The affordance buffer is extracted from EpicKitchens-100 videos after removing clips with poor lighting, low clarity, or persistent object obstruction.Motion blur detection selects a clear frame near contact.Source: §B.1 · pp. 12–13
  • Contact points are sampled in the intersection of object and hand bounding boxes, then centered on the average mapped position and resampled within a four-pixel-radius circle.This procedure is intended to keep sampled points on the object despite small surfaces and homography mapping errors.Source: §B.1 · p. 13
  • Table 6 lists the object categories included in the affordance memory.Source: Table 6 · p. 13

B.2. Evaluation Dataset

These sections describe the evaluation data, retrieval and mapping choices, robot setup, and correspondence results. DIFT-based methods provide the best correspondence results while also running faster.

B.2. Evaluation Dataset

  • The evaluation adds door and drawer objects to LOCATE, and marks every possible interactive position on cabinets with multiple handles.Ground-truth labels were collected using Labelme.Source: §B.2 · p. 13

B.3. Memory Retrieval

  • Retrieval comparisons use CLIP-B32, CLIP-B16, and LPIPS; the CLIP-B32+LPIPS retriever selects five semantic matches, then chooses the one with minimum LPIPS.The final visual-similarity selection uses a pretrained VGG network.Source: §B.3 · p. 13
  • Across mask thresholds, Robo-ABC exceeds the other baselines by a large margin.Figure 6 reports success rates at different mask thresholds but provides no specific values in the supplied passage.Source: Figure 6 · p. 13

B.4. Affordance Mapping

  • For multiple source contact points, the experiments map each point separately and average the mapped results, rather than averaging points before mapping.The passage says the latter method was more effective, but the separately mapped approach yielded slightly better results and was used in experiments.Source: §B.4 · p. 13

B.5. Robot Hardware Setup

  • The real-world setup uses a Franka Panda robot with an externally mounted, calibrated RealSense L515 camera.Figure 7 shows the workspace.Source: §B.5 · p. 13, Figure 7 · p. 13

C.1. Correspondence

  • DIFT-based methods yield the best correspondence results and faster inference than the other visualized correspondence methods.Figure 8 visualizes the methods on the same source image.Source: §C.1 · p. 14, Figure 8 · p. 14

C.2. Real Robot Results

Real-robot success rates are reported for object categories, with each experiment repeated five times after repositioning. In the four categories shown, Robo-ABC improves on VRB for bowls and rackets and ties it for bottles and scissors.

C.2. Real Robot Results

  • Each experiment repositions the object, attempts a grasp five times, and reports the overall average success rate.The paper also reports rates for seven object categories along with VRB in Table 7.Source: §C.2 · p. 14
  • The reported comparison pairs Robo-ABC with VRB across the listed bowl, bottle, racket, and scissor categories.Table 7 is captioned “Real-World Success Rate.”Source: §C.2 · p. 14, Table 7 · p. 14
  • Robo-ABC records 4/5 vs. VRB 3/5 for bowls and 5/5 vs. 2/5 for rackets, while both methods score 5/5 for bottles and 4/5 for scissors.The table reports real-world success rates; the supplied rows show these four object categories.Source: §C.2 · p. 14, Table 7 · p. 14

D. Limitations

The authors identify manual review of extracted affordances and source-video quality as limitations. Low resolution and lighting problems can affect semantic correspondence accuracy.

D. Limitations

  • All affordance outputs extracted from human videos need to be visualized to check their usability.This review is part of the affordance-extraction process.Source: D. Limitations · p. 14
  • The original dataset’s low resolution can affect the accuracy of finding semantic correspondences.Source: D. Limitations · p. 14
  • Some videos are unusable because of lighting problems, which can affect subsequent semantic correspondence accuracy.Source: D. Limitations · p. 14

Related papers

Loading 2401.07487v1…