PaperlayerView exact revision on arXiv

Source-linked AI summary

EGO-TOPO: Environment Affordances from Egocentric Video

Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, Kristen Grauman

arXiv:2001.04583v2cs.CV

TL;DR

Existing egocentric video methods often separate actions from the persistent environment, limiting representation of how spaces are used. EGO-TOPO builds topological affordance graphs from activity zones and visits, linking related zones across environments; on EPIC-Kitchens and EGTEA+, it supports scene-affordance learning and improves long-range action anticipation. The approach is also constrained by imperfect SLAM registration and trajectory sparsity in fine grids.

  • Problem

    Existing egocentric video models typically ignore the environment as a persistent physical space, while the paper seeks to model how people use it.

  • Method

    EGO-TOPO converts egocentric video into topological affordance graphs of activity zones and visits, linking zones by shared space or function.

  • Results

    On EPIC and EGTEA+, explicit environment modeling yields more robust scene-affordance models and improves over state-of-the-art long-range action-anticipation models.

  • Takeaways & Limitations

    The representation supports inferring likely object interactions in novel views and anticipating actions needed to complete long-term first-person activities.

  • Takeaways & Limitations

    Only 44% of EPIC-Kitchens frames can be accurately registered with a state-of-the-art SLAM algorithm.

Abstract

from arXiv · show

First-person video naturally brings the use of a physical environment to the forefront, since it shows the camera wearer interacting fluidly in a space based on his intentions. However, current methods largely separate the observed actions from the persistent space itself. We introduce a model for environment affordances that is learned directly from egocentric video. The main idea is to gain a human-centric model of a physical space (such as a kitchen) that captures (1) the primary spatial zones of interaction and (2) the likely activities they support. Our approach decomposes a space into a topological map derived from first-person activity, organizing an ego-video into a series of visits to the different zones. Further, we show how to link zones across multiple related environments (e.g., from videos of multiple kitchens) to obtain a consolidated representation of environment functionality. On EPIC-Kitchens and EGTEA+, we demonstrate our approach for learning scene affordances and anticipating future actions in long-form video.

1 UT Austin 2 Facebook AI Research

EGO-TOPO models egocentric environments as activity-centered spaces, addressing the gap between observed actions and persistent physical structure. It builds topological affordance maps from visits to zones, links functionally related zones across environments, and supports affordance inference and long-range action anticipation.

  • Motivation: Existing first-person models often treat video as frame chunks, while dense 3D reconstructions can fail on fast head-mounted video and overlook action-relevant structure.The paper argues that neither a pure-video nor a pure-3D view adequately represents scenes as action-affording spaces.
  • Approach: EGO-TOPO converts egocentric video into a topological map of activity zones and their rough spatial proximity.Each zone affords a coherent set of interactions rather than corresponding to a uniformly shaped 3D region.
  • Approach: The method links frames by shared physical space or shared function, allowing visually different locations such as trash cans and garbage disposals to represent the same action.This supports consolidating action-related zones across multiple environments.
  • Applications: The representation reorganizes video into visits to known zones, enabling reasoning about likely future actions and object interactions in particular zones.Figure 1 depicts the zones and the sequence in which they are visited.
  • Contribution: Compared with pure-video and pure-3D approaches, EGO-TOPO offers a concise spatial representation grounded in how people use the space.Its map is defined organically by human use rather than solely by visual sequence or geometric reconstruction.
  • Results: On EPIC and EGTEA+, explicit environment modeling produces more robust scene-affordance models and improves over state-of-the-art long-range action-anticipation models.The approach is evaluated for object-interaction inference in novel views and long-term first-person action anticipation.

2. Related Work

Prior egocentric-video methods recognize actions and model video structure, but generally do not represent the persistent, action-relevant environment. EGO-TOPO instead organizes activity around functionally meaningful zones, links them across environments, and uses them for affordance learning and long-horizon anticipation.

  • Egocentric video: Existing egocentric methods adapt third-person video models, while SLAM-based approaches provide metric spatial grounding but face failures in fast head-mounted video.Only 44% of EPIC-Kitchens frames are accurately registered by a state-of-the-art SLAM algorithm.
  • Structured video representations: Prior structured video representations encode object relations, action primitives, recurrent states, or long-term feature banks, whereas this approach grounds activity in a topological environment representation.The graph records visits to zones and supports spatially organized activity summaries.
  • Affordances: Compared with Action Maps and object-centered affordance work, EGO-TOPO links visually different zones by shared function and propagates affordance labels across them.A gas stove can link to a hotplate, and observed interactions at one functionally linked zone can augment training for another.
  • Mapping and people’s locations: Unlike dense geometric reconstructions, EGO-TOPO represents a persistent environment through activity-centric zones rather than indiscriminately modeling all 3D structure.This targets regions tied to human action, such as a kitchen sink area, instead of unrelated surfaces such as walls.
  • Affordances: The resulting graphs support both scene-affordance prediction and long-horizon action anticipation by consolidating information across visits, episodes, and environments.The paper evaluates these uses on EPIC and EGTEA+, reporting more robust affordance models and improved long-range anticipation.

4. Experiments

Experiments evaluate EGO-TOPO for scene affordance prediction and long-term action anticipation on EPIC-Kitchens and EGTEA+, comparing zone-based representations with visual, SLAM, and temporal baselines.

  • 4.1. EGO-TOPO for Environment Affordances: EGO-TOPO outperforms all other methods for environment affordance prediction on both datasets, with the consolidated variant providing the largest improvement, especially for rare classes.The evaluation uses multi-label mAP over all, rare, and frequent afforded interactions; Table 1 reports results averaged over five runs.
  • 4.1. EGO-TOPO for Environment Affordances: Visual-only clustering helps on EGTEA+ but hurts when visual features cannot capture coherent zones, whereas SLAM suffers from short motion windows, scale ambiguity, and incoherent cell assignments.Only 6% of EGTEA+ SLAM data are accurately registered, and noisy localizations also affect ACTIONMAPS.
  • 4.1. EGO-TOPO for Environment Affordances: Linking actions to discovered zones consistently improves affordance prediction, while functional alignment across kitchens strengthens the consolidated representation.Multiple actions from different times and kitchens are linked to shared zones, increasing the training signal for the affordance model.
  • 4.2. EGO-TOPO for Long Term Action Anticipation: On EPIC, EGO-TOPO outperforms all anticipation baselines across horizons and improves over the next strongest baseline by 2.4% mAP across 125 action classes.On EGTEA+, it matches complex temporal aggregation models, achieves the highest results for many-shot classes, and is slightly worse than ACTIONVLAD at K=50%.
  • 4.2. EGO-TOPO for Long Term Action Anticipation: Graph convolution improves anticipation by encoding the physical layout and interactions between zones, while kitchen-specific feature clusters provide environment-aware information.Without graph convolution, node aggregation still outperforms most baselines; including it enables performance beyond more complex temporal models.

5. Conclusion

The paper concludes that EGO-TOPO learns topological affordance graphs from egocentric activity and demonstrates their value for scene affordance learning and long-range anticipation.

  • 5. Conclusion: EGO-TOPO represents commonly used interaction zones and coherent actions across multiple kitchen environments in a topological affordance graph.The representation is learned from egocentric video of human activity.
  • 5. Conclusion: Experiments on EPIC and EGTEA+ demonstrate the viability of the environment representation for scene affordance learning and long-range anticipation.The paper identifies future applications in assisting users in unfamiliar spaces and guiding robot exploration.

Supplementary Material

The supplementary material provides demonstrations, annotation details, implementation information, and additional affordance results supporting the main experiments.

  • Supplementary Material: The supplement includes a graph-construction demonstration and additional scene-affordance results, followed by annotation, result-breakdown, and implementation details.Sections S1–S8 cover the video demonstration, crowdsourced annotations, class-level results, graph construction, model implementations, baselines, and supplementary predictions.
  • Supplementary Material: Additional affordance prediction results supplement the main paper’s Figure 6.These results are listed as part of supplementary section S8.

S1. EGO-TOPO demonstration video

The demonstration video shows how egocentric activity is converted into a topological map whose nodes represent interaction zones and whose edges represent commonly traversed paths.

  • S1. EGO-TOPO demonstration video: EGO-TOPO graphs organize video into zone visits, with nodes representing primary interaction zones and edges representing commonly traversed paths.The demonstration also presents affordance predictions over the constructed graph.
  • S1. EGO-TOPO demonstration video: Repeated visits in long videos produce more complete environment maps, while short videos can be linked to other graphs from the same kitchen.The supplementary examples include fully constructed maps from EPIC and EGTEA+ test videos.

S2. Crowdsourced affordance annotations

Affordance annotations are collected from individual video frames by asking workers to select likely interactions at the depicted location. The annotations evaluate affordance models rather than train them, with agreement filtering and dataset-specific interaction sets.

  • Workers select all likely interactions occurring at the location shown in a video clip’s center frame.
  • The annotations are used exclusively to evaluate affordance models, which are trained with single-clip interaction labels.
  • EPIC annotations cover 120 verb-noun interactions, while EGTEA+ includes all 75 interactions provided by its dataset.
  • Each image receives labels from five annotators, and only interactions supported by at least three workers are retained.
  • The resulting evaluation sets contain 1,020 EGTEA+ images and 1,155 EPIC images.

S3. Average precision per class for affordances

Class-wise analysis compares EGO-TOPO with CLIPACTION on scene affordance prediction. Gains are strongest for interactions involving objects associated with multiple physical locations, while location-specific objects are already easy to predict.

  • EGO-TOPO improves especially on interaction classes occurring in multiple physical locations, particularly among low-shot classes.
  • Interactions involving location-specific objects such as fridges, stoves, and taps are easier to predict and show limited improvement.

S4. Additional implementation details for EGO-TOPO graph creation

The supplementary graph-construction details use SuperPoint correspondences and RANSAC-based homographies to estimate frame similarity. Video sampling and visit scoring are standardized through fixed frame rates, temporal windows, and per-visit frame samples.

  • Homography estimation details: SuperPoint keypoints and homographies estimated from four random points support frame-pair similarity computation.
  • Homography estimation details: RANSAC maximizes homography inliers, and the inlier count serves as the frame-similarity measure.
  • Supplementary materials include the affordance-annotation interface and a list of annotated EPIC and EGTEA+ interactions.
  • Other details: Videos are subsampled to 6 fps, while visit scores average a nine-frame window around the current frame.
  • Other details: Each visit uses 20 uniformly sampled frames to obtain robust score estimates.

S5. Training details for affordance and long term anticipation experiments

Training uses ImageNet-pretrained ResNet-152 frame features, binary cross-entropy, and established baselines or author implementations. EPIC and EGTEA+ differ mainly in training duration while sharing the main optimizer and batch configuration.

  • Affordance learning experiments in Sec. 4.1: All models use ImageNet-pretrained ResNet-152 features as frame inputs and binary cross-entropy for training.
  • Affordance learning experiments in Sec. 4.1: For single-action clips, loss is evaluated only for the positive class while other class contributions are masked.
  • The baselines include TRAINDIST, I3D, RNN, ACTIONVLAD, TIMECEPTION, and VIDEOGRAPH implementations.
  • EPIC training runs for 100 epochs, whereas EGTEA+ uses 50 epochs under otherwise shared optimization settings.
  • Both datasets use Adam, a starting learning rate of 1e-3, decay after 80 epochs, weight decay 1e-5, and batch size 256.

S6. ACTIONMAPS implementation details

The ACTIONMAPS implementation uses semantic location similarity and matrix factorization, while supplementary figures and tables document comparison and affordance results.

  • S6. ACTIONMAPS implementation details: ACTIONMAPS uses cosine similarity between pretrained ResNet-152 features to measure semantic similarity between locations.This replaces object and scene classifier scores as side information.
  • S6. ACTIONMAPS implementation details: The implementation uses a latent dimension of 256 and sets λ = µ = 1e −3 for the RWNMF optimization objective.
  • S6. ACTIONMAPS implementation details: EGO-TOPO graphs encode frequent human paths through environments with edge thickness indicating traversal frequency.The supplementary graph examples include EPIC and EGTEA+ videos.
  • S6. ACTIONMAPS implementation details: On EPIC, EGO-TOPO outperforms CLIPACTION on most affordance classes, especially interactions involving objects usable at multiple locations.Single clip labels suffice more often when interactions are strongly tied to one physical location.
  • S6. ACTIONMAPS implementation details: Table S2 reports affordance prediction results for SLAMS across varying S × S grid sizes.

S7. SLAM implementation details

The SLAM baseline extracts and normalizes monocular trajectories before accumulating them in a uniform spatial grid, with supplementary figures showing additional zone predictions.

  • S7. SLAM implementation details: Monocular SLAM trajectories are extracted with ORB-SLAM2, unreliable or lost-tracking timesteps are removed, and coordinates are scaled to [0, 1].
  • S7. SLAM implementation details: The baseline uses a uniform grid of squares with edge length 0.2 to accumulate trajectories and construct the ACTIONMAPS matrix.
  • S7. SLAM implementation details: 61% of cells register only a single trajectory with a 20x20 EPIC grid, limiting label sharing and weakening the baseline.
  • S7. SLAM implementation details: Fig. S4 shows additional zone affordance predictions for EPIC and EGTEA+, with eight interactions displayed from each dataset.These examples supplement the main-paper results.
Loading 2001.04583v2…