PaperlayerView exact revision on arXiv

Source-linked AI summary

LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities

Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, Song-chun Zhu

arXiv:2007.15781v1cs.CV

TL;DR

Activity understanding research has often missed goal-directed intent, concurrent tasks, and multi-agent collaboration. LEMMA introduces a multi-view dataset with compositional annotations and benchmarks; its experiments report that compositional structure supports prediction across all metrics.

  • Problem

    Prior activity-understanding work largely misses goal-directed behavior, concurrent multi-task activity, and collaboration among multiple agents.

    Source: §1 · p. 2

  • Method

    LEMMA records multi-view activities across four agent-task scenarios, densely annotating compositional atomic-actions and their governing tasks, and provides recognition and anticipation benchmarks.

    Source: §1 · p. 3, §1.2 · pp. 4–5, §3 · p. 9

  • Results

    Understanding compositional action structures boosts performance on all metrics in the reported prediction results.

    Source: §4.1 · p. 12

  • Takeaways & Limitations

    The dataset and benchmarks invite study of realistic goal-directed activities and task scheduling and assignment in real-world scenarios.

    Source: §5 · p. 14

Abstract

from arXiv · show

Understanding and interpreting human actions is a long-standing challenge and a critical indicator of perception in artificial intelligence. However, a few imperative components of daily human activities are largely missed in prior literature, including the goal-directed actions, concurrent multi-tasks, and collaborations among multi-agents. We introduce the LEMMA dataset to provide a single home to address these missing dimensions with meticulously designed settings, wherein the number of tasks and agents varies to highlight different learning objectives. We densely annotate the atomic-actions with human-object interactions to provide ground-truths of the compositionality, scheduling, and assignment of daily activities. We further devise challenging compositional action recognition and action/task anticipation benchmarks with baseline models to measure the capability of compositional action understanding and temporal reasoning. We hope this effort would drive the machine vision community to examine goal-directed human activities and further study the task scheduling and assignment in the real world.

1 Introduction

The paper argues that activity understanding must account for goal-directed intent, concurrent tasks, and collaboration, dimensions largely missing from prior work. LEMMA contributes a multi-view dataset with compositional annotations and benchmarks for action recognition and anticipation.

1 Introduction

  • Activity understanding needs to represent goal-directed intent, task decomposition, concurrent execution, and coordination among agents.The paper argues that these dimensions are broadly missing from activity-understanding literature.Source: §1 · p. 2

1.2 Contributions

  • LEMMA provides third- and first-person views and dense labels linking compositional verb-noun atomic-actions to their governing tasks.The annotations are intended to support learning task scheduling and assignment.Source: §1 · p. 3

1.1 Related Work

  • Prior video datasets often lack sequential, goal-directed activities, while short atomic-action datasets are limited for long-term reasoning.Crowdsourced concurrent-action datasets may lack meaningful goal-directed tasks, and instructional datasets generally omit multi-agent or multi-task activity.Source: §1.1 · p. 4
  • Group-level multi-agent methods leave subtle collaboration and task assignment unaddressed, motivating study with real visual input.The paper contrasts this need with simulation-based environments.Source: §1.1 · p. 4
  • LEMMA captures goal-directed, decompositional, multi-task activities with multi-agent collaboration at a scale comparable to state-of-the-art benchmarks.The authors position the dataset as addressing shortcomings of prior work.Source: §1.1 · p. 4

1.2 Contributions

  • The paper benchmarks compositional action recognition and action/task anticipation, comparing and analyzing multiple baseline models.Source: §1.2 · pp. 4–5

2 The LEMMA Dataset

LEMMA uses four task-and-agent scenarios, records synchronized third- and first-person views, and annotates atomic-actions and their governing tasks. Its collection includes 324 activities, millions of images, and compositional labels spanning verbs and nouns.

2 The LEMMA Dataset

  • The dataset section profiles LEMMA through statistics intended to highlight its potential for activity understanding.Source: §2 · p. 5

2.1 Activities and Scenarios

  • Four scenarios vary the number of agents and tasks; in 2×1, an uninformed helper collaborates with an instructed leader using only nonverbal communication.The 2×2 scenario asks both participants to complete two complex tasks collaboratively.Source: §2.1 · p. 5

2.2 Data Collection

  • Two Kinect Azure cameras capture global RGB-D views, while head-mounted GoPros capture agent-specific first-person views; recordings are synchronized at 24 FPS.The setup also captures depth information and 3D human skeletons.Source: §2.2 · p. 6

2.3 Ground-truth Annotation

  • Annotations include human boxes, temporal action segments, semantic verb-noun labels, and each atomic-action’s governing task.Two workers annotate each video first; a third fine-tunes annotations when box or segment IoU is below 0.5.Source: §2.3 · pp. 6–7

2.4 Dataset Statistics

  • 324 activities comprise 127 1×1, 76 1×2, 66 2×1, and 55 2×2 scenarios, totaling 10.1 hours of video.The average activity lasts 2 minutes and the longest lasts 7 minutes.Source: Figure 3 · p. 7, §2.4 · p. 8
  • 4.6 million images include 2.9 million RGB images and 1.7 million depth images; 0.9 million primary-view RGB frames and 0.8 million action frames were annotated.Source: Figure 3 · p. 7, §2.4 · p. 8
  • After ambiguity resolution, annotations contain 24 verb classes, 64 noun classes, and 862 compositional atomic-action labels, with 641 occurring more than 50 times.Verb and noun frequencies roughly follow Zipf’s law.Source: Figure 3 · p. 7, §2.4 · p. 9
  • Verb-noun-task co-occurrences include “get” and “put” with varied nouns across nearly all tasks, as well as participant interactions in action labels.Examples include “get knife from P1 using hand” and “point-to sink.”Source: Figure 4 · p. 8, §2.4 · p. 9

3 Benchmarks

The benchmarks assess atomic-action semantics and goal-directed activity understanding, including concurrent tasks and multi-agent scenarios. They cover compositional action recognition and next-action/task anticipation across four activity scenarios.

3 Benchmarks

  • The benchmarks assess atomic-action semantics and goal-directed activity understanding with concurrent tasks, especially in multi-agent scenarios.Source: §3 · p. 9

3.1 Compositional Action Recognition

  • The compositional action recognition benchmark defines 24 templates and evaluates concurrent verbs and verb-conditioned nouns assigned to object, location, or tool positions.Source: §3.1 · p. 9, Figure 5 · p. 10

3.2 Action and Task Anticipation

  • Action/task anticipation predicts the next action segment in FPV and TPV; scenario training shares of 3/4, 1/4, 1/4, and 1/4 progressively increase difficulty to encourage generalization.Source: §3.2 · p. 10

4 Experiments

The experiments compare compositional action recognition and action/task anticipation across first- and third-person views and four activity scenarios. Results favor multi-agent anticipation models and show that view, temporal structure, and task complexity shape performance.

4.1 Compositional Action Recognition

  • Recognition uses a random 3:1 split and evaluates FPV and TPV inputs with verb, noun, and compositional-action AP, recall, and F1 at 5 FPS.The test set contains 81 activities; models are evaluated on sampled frames.Source: §4.1 · p. 11
  • Recognition compares direct I3D and SlowFast prediction with branching and sequential verb–noun models; anticipation tests segment features and long-term feature banks with other-agent FPV or primary TPV inputs.The multi-agent anticipation variants combine another agent’s FPV features or the primary TPV features with the model’s own FPV features.Source: §4.1 · p. 11, §4.2 · p. 13
  • For FPVs, verb–noun prediction boosts all metrics over direct compositional-action prediction, while sequential inference trails branching.The paper attributes the sequential model's slight shortfall to accumulated error from verb-recognition precision below 25%.Source: §4.1 · pp. 11–12, Table 2 · p. 12
  • TPV compositional-action results are significantly below FPV results amid severe occlusion, and compositional prediction brings no significant improvement over direct prediction.The authors suggest that combining TPV and FPV representations may be important for this problem.Source: §4.1 · p. 12

4.2 Action and Task Anticipations

  • Anticipation uses scenario-specific train/test splits and evaluates per-agent action and task AP, recall, and F1 across the four scenarios.Models access FPVs, TPVs, and primary-view human bounding boxes.Source: §4.2 · p. 12
  • For single-agent scenarios, temporal models generally outperform segment-feature models, and adding TPV features slightly helps task interpretation and anticipation.Task anticipation performs better in the 1×1 than the 1×2 scenario.Source: §4.2 · pp. 13–14
  • Multi-agent variants perform best among the action and task anticipation models.Table 3 reports the benchmark comparison.Source: §4.2 · p. 13, Table 3 · p. 14
  • In multi-agent scenarios, aggregating FPV and TPV features generally performs better; 2×1 performance is slightly worse than 2×2.The authors hypothesize that plans change less often in 2×2, while 2×1 requires more frequent communication to coordinate task order.Source: §4.2 · p. 14

5 Conclusions

The conclusion presents LEMMA as a dataset for natural multi-agent, multi-task daily activities, with dense annotations and benchmarks for action understanding and temporal reasoning.

5 Conclusions

  • LEMMA provides dense compositional-action and task annotations across four activity scenarios of increasing difficulty.The conclusion frames the dataset around natural multi-agent, multi-task daily activities.Source: §5 · p. 14
  • The paper proposes compositional action recognition and action/task anticipation benchmarks to measure action understanding and temporal reasoning.Source: §5 · p. 14

Supplementary Material for LEMMA: A Multi-view Dataset for LEarning

The supplied supplementary-material passages identify the paper, its authors, and their institutional affiliation.

Supplementary Material

1 Annotation Details

The annotation process builds verb and noun vocabularies and patterns, then uses video-linked tools and human review to label compositional actions and tasks.

1 Annotation Details

  • The verb vocabulary starts from EPIC-KITCHENS and related goal-directed activity datasets, then removes unrelated verbs and adds verbs for human–human interaction.After language ambiguities are resolved, 24 verbs remain.Source: §1 · pp. 19–20
  • The noun vocabulary covers 64 objects that could be interacted with across the 15 daily tasks; verb patterns specify three semantic positions for objects, locations, and tools.Source: Table 1 · p. 19, §1 · p. 20
  • Vatic is used to annotate human bounding boxes and verb patterns, while an interactive tool fills noun slots and assigns each action its governing task.Synchronized egocentric and third-person views are shown together, with agent IDs to help annotators match people across views.Source: §1 · p. 20
  • Annotators manually review completed labels, resolving ambiguities and eliminating or merging nouns with occurrence frequencies below 50.Source: §1 · p. 21

2 Implementation Details

The compositional action recognition models use a multi-branch network to train verbs, nouns, and their correspondences. The sequential model also supports multi-agent anticipation by incorporating pooled features from another agent or a third-person view.

Compositional Action Recognition

  • The sequential model uses a multi-branch network to train verbs, nouns, and their correspondences.The branching model removes the connection between the verb and noun branches by discarding additional verb embeddings.Source: §2 · pp. 21–22
  • Three verb candidates are generated from 3D CNN visual features using separate linear projections and trained with cross-entropy loss.Ground-truth action labels are sorted by their index in the verb vocabulary for supervision.Source: §2 · p. 21
  • The noun branch combines verb embeddings from GloVe with visual features and predicts nouns at their semantic positions.Training uses ground-truth verb embeddings; testing uses embeddings of the predicted top-3 verbs.Source: §2 · pp. 21–22
  • Max-pooling combines noun semantic features with verb visual features to produce a compositional-action feature supervised by binary cross-entropy.The joint loss is expressed as L = Lverb + Lnoun + Lcomp.Source: §2 · p. 22

Action and Task Anticipation

  • For two-agent anticipation, pooled egocentric features from another agent or TPV features are concatenated with the target agent’s egocentric features.For TPV features, ROIAlign extracts features corresponding to each agent’s bounding box.Source: §2 · pp. 22–23, Figure 3 · p. 22

3 Additional Experiment Results

This section presents qualitative anticipation results and examples of annotated bounding boxes and compositional actions.

Additional Experiment Results

  • Qualitative action and task anticipation results are shown on LEMMA.Figure 4 presents these results.Source: §3 · p. 23
  • Examples show annotated bounding boxes alongside compositional actions.Figure 5 presents these examples.Source: Figure 5 · p. 24

Related papers

Loading 2007.15781v1…