PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Human Activities and Object Affordances from RGB-D Videos

Hema Swetha Koppula, Rudhir Gupta, Ashutosh Saxena

arXiv:1210.1207v2cs.ROcs.AIcs.CV

TL;DR

The paper asks how RGB-D videos can support descriptive recognition of long human activities and object affordances for personal robots. It jointly models sub-activities and affordances in an MRF, learns with an SSVM across alternative temporal segmentations, and demonstrates the resulting labels on a PR2 robot. On 120 videos from four subjects, it reports 79.4% affordance, 63.4% sub-activity, and 75.0% high-level activity accuracy.

  • Problem

    The paper addresses descriptive labeling of human sub-activities and object affordances from RGB-D videos for assistive robots.

  • Method

    The method jointly models object and sub-activity nodes, their spatial and temporal relations, and latent labelings over multiple temporal segmentations using an SSVM.

  • Results

    79.4% affordance, 63.4% sub-activity, and 75.0% high-level activity accuracy were achieved using multiple segmentation on the evaluated videos.

  • Takeaways & Limitations

    The labels were demonstrated in PR2 assistive tasks, where inferred affordances enabled more meaningful task performance.

Abstract

from arXiv · show

Understanding human activities and object affordances are two very important skills, especially for personal robots which operate in human environments. In this work, we consider the problem of extracting a descriptive labeling of the sequence of sub-activities being performed by a human, and more importantly, of their interactions with the objects in the form of associated affordances. Given a RGB-D video, we jointly model the human activities and object affordances as a Markov random field where the nodes represent objects and sub-activities, and the edges represent the relationships between object affordances, their relations with sub-activities, and their evolution over time. We formulate the learning problem using a structural support vector machine (SSVM) approach, where labelings over various alternate temporal segmentations are considered as latent variables. We tested our method on a challenging dataset comprising 120 activity videos collected from 4 subjects, and obtained an accuracy of 79.4% for affordance, 63.4% for sub-activity and 75.0% for high-level activity labeling. We then demonstrate the use of such descriptive labeling in performing assistive tasks by a PR2 robot.

I. INTRODUCTION

The paper targets long-duration human activities composed of variable sub-activities by jointly labeling activities and object affordances from RGB-D videos. It introduces a relational model, multiple-segmentation learning, annotated datasets, and robotic demonstrations.

  • Motivation: RGB-D sensing supports learning complex human activities despite variation, clutter, occlusion, and viewpoint changes.The paper motivates 3D perception as a basis for estimating human pose and recognizing activities in challenging indoor environments.
  • Motivation: Object affordances complement object categories by indicating how objects are used and can change across sub-activities.Examples include a pitcher becoming reachable, movable, and pourable during cereal preparation.
  • Method: The method jointly models sub-activities, object affordances, their spatial interactions, and temporal evolution in an MRF learned with an SSVM.The graph contains object and sub-activity nodes connected by object–object, object–sub-activity, and temporal edges.
  • Method: Temporal segmentation is handled by learning labelings over multiple segmentation hypotheses treated as latent variables.This addresses noisy boundaries and differences in sub-activity duration and execution across people.
  • Results: Using multiple segmentation, the method achieved 79.4% affordance, 63.4% sub-activity, and 75.0% high-level activity accuracy.With ground-truth segmentation, the corresponding accuracies were 91.8%, 86.0%, and 84.7%.
  • Datasets and evaluation: The CAD-120 dataset contains 120 long-term RGB-D activities with annotations for skeletons, objects, affordances, sub-activities, and high-level activities.The paper also reports experiments on CAD-60 and releases source code with ROS and PCL integration.

II. RELATED WORK

Prior work increasingly links human activities with objects, but existing approaches remain limited in duration, 3D relational modeling, object interactions, or human-context affordances. This paper proposes a unified framework for joint activity and affordance detection.

  • Robotic assistive tasks require understanding both object affordances and human activities in human environments.
  • Object affordances: Existing affordance methods generally predict object interaction opportunities without modeling how humans use objects in context.
  • Related activity methods remain limited to shorter actions, hand actions, pose-only recognition, or models without object–object interactions.
  • Earlier methods often treat object detection and activity recognition separately, whereas mutual context can improve both tasks.
  • The proposed framework jointly captures spatial and temporal relations between object affordances and human poses for activity detection.

III. OVERVIEW

The method represents RGB-D activity videos as spatio-temporal MRFs linking object affordances and sub-activities within and across temporal segments. Its energy function combines node and interaction potentials to jointly predict labels.

  • The pipeline tracks the human skeleton, segments and tracks objects, then constructs an MRF over the resulting spatio-temporal sequence.
  • The graph contains object nodes and sub-activity nodes for each temporal segment, with within-segment and temporal-neighbor connections.
  • Affordance–sub-activity potentials use relative geometry between objects and human joints to model context-dependent object use.
  • Affordance–affordance potentials model dependencies among objects using relative relations such as being on top of, nearby, or in front of another object.
  • Temporal potentials capture changes in sub-activities and object affordances across neighboring segments using pose, appearance, and location features.
  • The total energy combines object, sub-activity, object–object, object–activity, and temporal terms, enabling structured inference and learning.

V. OBJECT DETECTION AND TRACKING

Object tracks are built by combining detector outputs with particle-filter tracking and graph-based association across frames. Detection categories and bidirectional tracking help handle occlusions and unreliable tracks.

  • Pre-trained object detectors are run on sampled frames, and particle-filter tracking provides object trajectories for graph construction.
  • Object Detection: The detector combines appearance and shape or geometry features, including color histograms and HoG, with classifier scores.
  • Object Detection: At test time, candidate boxes near the skeleton’s hands are filtered before RGB detection and VFH-based verification.
  • Combining Object Detections with Tracking: The tracking graph is initialized from object bounding-box guesses and expanded with detections connected through the video.
  • Combining Object Detections with Tracking: Detections are categorized as merged, isolated, or ignored according to proximity and similarity to current tracks.
  • Combining Object Detections with Tracking: Isolated detections are tracked in both directions to correct tracks disrupted by partial or full occlusions.

VI. TEMPORAL SEGMENTATION

Temporal segmentation reduces the graph by grouping similar frames, but a single segmentation is unreliable because features are non-additive and sub-activity boundaries are ambiguous. The method therefore generates multiple segmentation hypotheses.

  • Temporal segmentation groups similar frames into segments representing atomic human-skeleton and object movements, reducing the number of graph nodes.
  • A mistaken single segmentation can degrade subsequent activity detection, while dynamic programming applies only when the features are additive.
  • The feature maps are non-additive, making exhaustive segmentation feature computation exponential in the number of frames and preventing dynamic programming.
  • Sub-activity boundaries are often unclear because people begin the next sub-activity before finishing the current one, so no single optimal segmentation may exist.
  • The method generates multiple segmentations using uniform and graph-based procedures while varying segment size, offset, or graph thresholds.

VII. FEATURES

The model represents objects and sub-activities with spatial, temporal, and relational features extracted from RGB-D videos. These features encode motion, pose, geometry, and human–object interactions.

  • Object features: Object node features encode scene location and within-segment motion using centroids, bounding boxes, tracking correspondences, displacement, and distance traveled.The object feature vector has dimension R180 after cumulative binning into 10 bins.
  • Sub-activity features: Sub-activity node features use upper-body skeleton joints and body-pose information computed from the RGB-D video.Joint locations are represented relative to the subject’s head.
  • Relational features: Edge features model object–object, object–sub-activity, and temporal relations within and across temporal segments.The four interaction types include two within-segment relations and two between-segment temporal relations.

VIII. INFERENCE AND LEARNING ALGORITHM

Inference seeks the best joint labeling under a difficult mixed-integer formulation. A linear relaxation enables graph-cut optimization, while exact class constraints are handled with a general-purpose MIP solver.

  • Inference: Inference is NP-hard, but an equivalent mixed-integer formulation has a linear relaxation solvable as quadratic pseudo-Boolean optimization with graph cuts.This provides an efficient approximate solution to the labeling argmax.
  • Relaxation: Auxiliary variables replace products of labeling variables in the mixed-integer formulation.These variables support the optimization constraints used in the relaxed problem.
  • Relaxation: The basic relaxation can have half-integral solutions, whose node variables take values in {0, 0.5, 1}.This follows from relaxing variables to the interval [0, 1].
  • Inference: Because each node has exactly one class label, additional constraints are imposed and the resulting problem is solved exactly with a general-purpose MIP solver.The added constraints make graph cuts unavailable for this version.
  • Runtime: 74.9 frames/second were achieved for inference and 16.0 frames/second end-to-end, including feature computation.The reported setup used a 2.93 GHz Intel processor with 16 GB of RAM.

B. Learning.

Learning uses a large-margin structural SVM formulation to estimate model parameters from labeled examples. The resulting optimization is handled as a convex quadratic program with cutting-plane and graph-cut subroutines.

  • Learning: The parameter vector w is learned from labeled training examples using a large-margin approach.The method optimizes a regularized upper bound on training error.
  • Learning: The structural formulation compares the optimal predicted labeling with alternative labelings through a loss function.The optimal prediction is denoted ˆym, while ∆(y, ˆy) defines the loss.
  • Model representation: Feature maps and parameter blocks are stacked so the model score can be written compactly as wT Ψ(x, y).This is an equivalent notation for the expanded parameter-feature expression.
  • Optimization: Training is formulated as a convex quadratic program with constraints over alternative labelings.The constraints enforce the required margin relationships.
  • Optimization: Although the quadratic program has exponentially many constraints, a cutting-plane algorithm solves it efficiently using a relaxed prediction problem.The relaxed prediction problem can be solved with graph cuts because of the loss structure.

C. Multiple Segmentations

The method treats temporal segmentation as uncertain because people vary in execution and may merge adjacent sub-activities. It samples multiple segmentations and jointly learns labelings across their hypotheses.

  • C. Multiple Segmentations: RGB-D videos may admit multiple valid temporal segmentations because execution durations and sub-activity boundaries vary across people.People may also merge two consecutive sub-activities, making a single correct segmentation unavailable.
  • C. Multiple Segmentations: The method generates a set H of segmentations using different segmentation methods and criteria.Each segmentation hypothesis is denoted hn.
  • C. Multiple Segmentations: Each segmentation hypothesis receives a discriminant function and an assignment score for mapping its segment labels to a common labeling.The objective seeks a labeling that maximizes assignment scores across segmentations.
  • C. Multiple Segmentations: The segmentation-specific labelings are treated as unobserved variables in the joint formulation.A latent structural SVM is possible in principle but becomes intractable for large hypothesis spaces.
  • C. Multiple Segmentations: An approximate two-step procedure independently learns parameters for each segmentation and then trains a shared parameter model.The procedure addresses the large space of segmentation hypotheses.
  • C. Multiple Segmentations: The iterative procedure alternates between solving for each hypothesis labeling in parallel and solving for the shared labeling.The method is guaranteed to converge and scales better as the number of hypotheses grows.

D. High-level Activity Classification.

The paper evaluates high-level activity classification using sub-activity and affordance information, with additional occlusion features for activities sharing similar interaction sequences.

  • High-level activity classification: High-level activities are classified from histograms of sub-activity and affordance labels.Occlusion features are added because stacking and unstacking can share the same sub-activity and affordance sequences.
  • Dataset: The CAD-60 dataset contains 60 RGB-D videos from four subjects performing 12 high-level activity classes.Some classes contain only one sub-activity or lack object interactions.
  • Dataset: CAD-120 contains 120 activity sequences covering ten high-level activities performed by four subjects, totaling 61,585 RGB-D frames.Each high-level activity was performed three times.
  • Object tracking: Object bounding boxes are manually labeled every 50th frame, while intervening frames are obtained by SIFT tracking with depth consistency.The resulting tracks are evaluated by overlap with generated ground-truth bounding boxes.
  • Object tracking: Tracking evaluation reports the percentage of frames reaching at least 40%, 20%, and 10% overlap with ground-truth object boxes.These overlap thresholds define the measures listed for object tracking.

C. Labeling results on the Cornell Activity Dataset 60 (CAD-60)

Experiments compare the proposed structured model with feature and interaction ablations on RGB-D activity videos, showing gains from object, activity, temporal, and multi-segmentation context.

  • Context ablations: Object context raises sub-activity micro precision by 14.1% and macro precision and recall by about 23.3% over the sub-activity-only model.Including affordance and occlusion information also improves high-level activity accuracy three-fold.
  • Context ablations: Sub-activity context increases affordance micro precision by 4.9%, macro precision by 17.7%, and macro recall by 11.1% over the object-only model.The object-only model retains object–object context, which contributes to affordance detection.
  • Context ablations: Object–object interactions increase affordance macro recall by 14.9% and macro precision by about 10.9%.The comparison indicates that human activity context alone does not always determine an object’s affordance.
  • Context ablations: Temporal interactions increase micro precision by 4.8% for affordances, 10.0% for sub-activities, and 3.3% for high-level activity.These values compare the full model with a variant without temporal edges.
  • Multiple segmentations: Merging multiple temporal segmentations increases macro precision by 5.8% for affordances and 9.1% for sub-activities over the best single segmentation.Using estimated object tracks instead of ground-truth tracks causes a reported performance drop.

E. Robotic Applications

The paper demonstrates PR2 assistance driven by detected activities and affordances, including reactive responses and object manipulation under inferred movement constraints.

  • Applications: Activity recognition enables a robot to select complementary responses, while affordance recognition guides meaningful object manipulation.The demonstrations cover both human assistance and affordance-based handling.
  • Assisting humans: The PR2 robot successfully detects three activities and performs the corresponding reactive actions in a kitchen environment.The scenarios are clearing the table after a meal, bringing water during medicine-taking, and refrigerating milk after cereal-making.
  • Evaluation setting: The experiments use seven activities, nine unique objects, and 19 object instances, including two activities unseen during training.The manipulation evaluation reports results with and without multiple observations.
  • Affordance-based manipulation: 100% accuracy is achieved for moving 15 objects correctly identified as movable.The robot was programmed to move only objects inferred to have the movable affordance.
  • Affordance-based manipulation: 80% of movable instances receive the correct constrained-movement treatment for objects inferred as drinkable, pourable, or pour-to.After multiple observations and affordance aggregation, task performance reaches 100% accuracy.
  • Implications: The authors conclude that affordances support more meaningful manipulation, while complex affordances such as pouring may require more observations.Including additional high-level activities demonstrates generalizability for affordance detection.

X. CONCLUSION AND DISCUSSION

The paper jointly labels sub-activities and object affordances in RGB-D videos, incorporating temporal segmentation and evaluating generalization to unseen subjects. It also demonstrates assistive-robot use while identifying remaining perception and planning needs.

  • 79.4% affordance, 63.4% sub-activity, and 75.0% high-level activity labeling accuracy were achieved on activities performed by subjects absent from training.
  • The model jointly labels human sub-activities and object affordances using an MRF with temporal interactions and latent alternate segmentations.
  • PR2 robot experiments demonstrated that inferred affordance labels support more meaningful assistance with humans’ daily activities.
  • The assistive-robot demonstration uses pre-programmed actions, while improved object perception, task planning, and human-robot interaction remain needed for efficient operation.
Loading 1210.1207v2…