The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain

• Egocentric interaction study has applications including worker safety, energy-saving policies, and alerts about actions missed in a production pipeline.Source: §1 · p. 1
• MECCANO is an egocentric video dataset collected in an industrial-like motorbike-building task involving tools, small objects, and sequential actions.The authors describe the scenario as a simplification of real industrial settings, while noting its complexity.Source: §1 · p. 2
• Videos from 20 participants in Italy and the United Kingdom were annotated with interaction start and end times and active-object bounding boxes.Source: §1 · p. 2
• The dataset supports action recognition, active-object detection, active-object recognition, and egocentric human-object interaction detection.Source: §1 · p. 2
• Egocentric human-object interaction detection predicts verb and noun classes and localizes active objects, with the camera wearer as the interacting human and potentially multiple objects involved.Source: §1 · p. 2
• Baseline experiments indicate that current state-of-the-art approaches achieve limited performance on MECCANO, making it a challenging benchmark for industrial-like egocentric interactions.Source: §1 · p. 2

Datasets for Human Behavior Understanding

• HOI benchmarks include V-COCO with 26 verbs and 80 COCO objects, HICO-Det with 117 verbs and 80 objects, and HOI-A with 10 verbs and 11 objects.Table 1 provides a comparative overview of relevant datasets and tasks.Source: §2 · pp. 2–3, Table 1 · p. 3
• Egocentric datasets cover kitchens, offices, and daily-life activities and generally target action recognition, whereas MECCANO is the first industrial-like egocentric dataset annotated for EHOI detection.Table 1 compares the datasets with respect to their supported tasks.Source: §2 · p. 3, Table 1 · p. 3

Action Recognition

• Action-recognition methods include motion-based features, two-stream networks, 3D ConvNets, and SlowFast; the paper evaluates SlowFast, I3D, and 2D CNN baselines.SlowFast uses a high-frame-rate pathway for motion and a low-frame-rate pathway for semantics.Source: §2 · p. 3

HOI Detection

• HOI detection methods have mainly addressed third-person vision, using human-object triplets, graph convolutional networks, or bounding-box relationships.The paper studies HOI detection from an egocentric perspective with MECCANO.Source: §2 · p. 3

EHOI Detection

• Earlier egocentric datasets with bounding boxes did not explicitly label relationships between objects and actions for EHOI detection.The paper aims to provide a task definition, benchmark dataset, and initial baseline evaluation.Source: §2 · pp. 3–4

3.1. Data Collection

• Participants assembled a motorbike model from 49 components grouped into 16 classes, using a screwdriver, wrench, and instruction booklet.The 20 participants were recruited in Italy and the United Kingdom, and could freely use the objects and booklet during assembly.Source: §3.1 · p. 4
• Videos were captured with a head-mounted Intel RealSense SR300 at 1280x720 pixels and 12 fps.Each video covered a complete assembly; the average duration was 21.14 minutes.Source: §3.1 · p. 4

3.2. Data Annotation

• Temporal annotations mark interaction start and end times and assign one of 12 verbs; 1,401 segments (15.82%) overlap because participants can perform actions simultaneously.The dataset contains 8,857 segments; for “check,” start time is marked when the participant is visibly looking at an object.Source: §3.2 · p. 5
• Active-object annotations cover 20 classes at frames sampled every 0.2 seconds, using boxes for visible objects and estimated positions for occluded ones.Multiple simultaneously active objects are annotated, across 64,349 labeled frames.Source: §3.2 · p. 5
• The temporal segments are mapped to 61 action classes, each combining a verb with one or more objects.The action classes follow a long-tail distribution.Source: §3.2 · p. 6
• An EHOI is represented by an interaction verb and the active objects involved, and each annotation combines the verb label with active-object boxes.MECCANO is the first egocentric dataset explicitly annotated for EHOI detection.Source: §3.2 · p. 6

4. Benchmarks and Baseline Results

• MECCANO provides baselines for four tasks, with training, validation, and test splits designed to balance desk types and acquisition countries.The four tasks are action recognition, active-object detection, active-object recognition, and EHOI detection; the latter had not been treated in previous works.Source: §4 · p. 6

4.1. Action Recognition

• SlowFast achieved Top-1 accuracy of 47.82 and an F1-score of 41.05 in action recognition, with similar Top-1 and Top-5 accuracy across the baselines.The authors note that the more consistent precision, recall, and F1 gaps matter given the dataset’s long-tailed action distribution.Source: Table 3 · p. 7, §4.1 · p. 7

4.2. Active Object Detection

• 38.14% AP with no distance threshold was the best active-object detection result; MECCANO-retrained variants also achieved 33.33% AP with an average-distance threshold and 20.18% AP with the standard threshold.The original detector variants obtained approximately 11% AP, and the authors identify adapting the general object detector to MECCANO as key to performance.Source: Table 4 · p. 7, §4.2 · p. 7

4.3. Active Object Recognition

• The active-object recognition baseline found large objects easier to recognize, including screwdriver AP 60.50% and instruction-booklet AP 46.48%.The authors describe the dataset as challenging because it includes small objects and leave more specific active-object detection approaches to future studies.Source: Table 5 · p. 8, §4.3 · p. 8

4.4. EHOI Detection

• SlowFast+Faster-RCNN achieved mAP 25.93%, versus mAP role 4.92% for InteractNet and 8.45% for its context-feature extension.The authors report that processing video clips substantially boosts performance over still-image approaches, but describe EHOI detection as needing further investigation in this challenging domain.Source: Table 6 · p. 8, §4.4 · p. 8

5. Conclusion

• MECCANO provides temporal action segments and spatial active-object bounding boxes, with a taxonomy of 12 verbs, 20 nouns, and 61 unique actions.The authors report baseline experiments for action recognition, active-object detection, active-object recognition, and EHOI detection.Source: §5 · p. 9

SUPPLEMENTARY MATERIAL

• The supplementary material reports additional information about the dataset, annotation stage, and experiment implementation.It identifies itself as accompanying the MECCANO submission to WACV 2021.Source: SUPPLEMENTARY MATERIAL · p. 9
• The document organizes additional material into data collection and annotation, implementation details for compared methods, and qualitative results.Source: SUPPLEMENTARY MATERIAL · p. 9

6.1. Component classes and grouping

• The 49 motorbike components are grouped into 16 component classes, including similar-length parts as screws and similar washers, yielding 20 total object classes with tools, booklet, and partial model.The screw group combines A054 and A051; the washer group combines A053, A057, and A077.Source: §6.1 · p. 9, Figure 10 · p. 9

6.2. Data Annotation

• Temporal annotations use 12 verb classes—take, put, check, browse, plug, pull, align, screw, unscrew, tighten, loosen, and fit—with a segment around each action instance.Each segment is labeled with the verb that best describes the action it contains.Source: §6.2 · p. 9
• For each annotated video segment, frames sampled every 0.2 seconds were labeled with bounding boxes and class labels for all active objects.Annotators used a customized VIA interface with thumbnails for 20 object classes and written rules with visual examples to reduce ambiguity.Source: §6.2 · pp. 9–10, Figure 13 · p. 11, Figure 14 · p. 11
• Action classes were selected from observed verb-object combinations, using distributions of active-object counts and rules for which objects occur with each verb.Take and put each have object-specific classes plus an additional class for taking or putting multiple objects simultaneously.Source: §6.2 · p. 10
• The action taxonomy distinguishes fit rim and tire, loosening bolts with hands, three align actions, and object-specific plug and pull classes.The align classes specify screwdriver-to-screw, wrench-to-bolt, or object-to-object; plug and pull distinguish screws, rods, and, for pull, the partial model.Source: §6.2 · p. 11
• Screw and unscrew classes distinguish screwdriver use from hands, while tighten classes distinguish using a wrench from hands.Source: §6.2 · pp. 11–12
• The resulting MECCANO action taxonomy contains 61 action classes.Source: §6.2 · p. 12

7.1. Action Recognition

• Action-recognition baselines use uniformly resampled video shots to provide fixed-length clips to SlowFast, C2D, and I3D.The reported average clip length is 26.19 frames; all three baselines converged under the specified training settings.Source: §7.1 · p. 12

7.2. Active Object Detection

• For active object detection, Faster-RCNN is trained on active-object labels but evaluated as a single-class active-object detector.The model uses a ResNet-101 backbone and Feature Pyramid Network.Source: §7.2 · p. 12

7.3. Active Object Recognition

• Active object recognition uses the same model as active object detection while retaining object classes at test time.Source: §7.3 · p. 12

7.4. EHOI Detection

• EHOI baselines combine verb recognition with an active-object detector, while InteractNet predicts verbs from hands and objects, optionally adding context.SlowFast is trained on 12 verb classes; InteractNet fuses branch probability distributions by late fusion.Source: §7.4 · p. 12

8. Additional Results

• SlowFast qualitative examples show cases where it predicts the verb correctly but not the object, or the object correctly but not the verb.Source: §8 · p. 12
• Table 7 reports per-class AP and average AP for active object recognition on the test set, with mAP also reported for each test video.Source: §8 · p. 12
• Active object recognition examples include correct predictions and false positives where passive items, such as an instruction booklet or red bars, are detected as active.Source: §8 · p. 12