MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain

1. Introduction

• Industrial egocentric behavior understanding has been little explored, partly because data collection faces privacy concerns and protection of industrial secrets.Existing wearable datasets also seldom include rich multimodal signals, despite the availability of gaze and depth on wearable platforms.Source: §1 · p. 2
• MECCANO Multimodal extends the earlier dataset with depth and gaze, temporal action labels, spatial boxes for hands and next-active objects, and five behavior-understanding tasks.Twenty subjects assembled a 49-component motorbike toy in Italy and the United Kingdom using a custom headset.Source: §1 · p. 2
• MECCANO labels action start and end times, hand-object contact times, active objects and hands, and past instances of objects for next-active-object prediction.The interaction time marks the first frame of hand-object contact, when an object changes from passive to active.Source: §1 · p. 2
• The benchmark covers action recognition, active-object detection and recognition, egocentric human-object interaction detection, action anticipation, and next-active-object detection and recognition.The paper reports that these problems remain challenging in industrial-domain egocentric settings despite multimodal signals.Source: §1 · p. 2
• The paper presents the dataset and benchmark as a way to study industrial-like human behavior, where existing state-of-the-art approaches are not sufficient for the considered problems.The introduction also identifies potential uses such as machine-use monitoring, calibration scheduling, operator guidance, and alerts for anticipated dangerous actions.Source: §1 · pp. 1–2

2.1. Datasets for Human Behavior Understanding

• Prior egocentric datasets cover daily living, kitchens, and other domains; Assembly101 shares MECCANO’s assembly setting but focuses on action tasks and multimodal differences distinguish the datasets.Assembly101 benchmarks action recognition, anticipation, temporal segmentation, and mistake detection, using monochrome egocentric cameras.Source: §2.1 · pp. 3–4
• MECCANO Multimodal is described as the first egocentric industrial-like dataset combining gaze and depth, with annotations for five behavior-understanding tasks.The paper notes that prior egocentric datasets generally focus on kitchens, offices, or daily life and more often address action recognition than EHOI detection.Source: §2.1 · p. 4

2.2. Human Behavior Understanding Tasks

• The task review spans action recognition, object detection, human-object interaction, action anticipation, and next-active-object detection.The paper frames these as relevant tasks for understanding human behavior.Source: §2 · p. 2, §2.2 · p. 4

2.2.1 Action Recognition

• Action-recognition research includes two-stream and 3D ConvNet approaches, temporal modeling methods, and egocentric adaptations; the paper evaluates SlowFast on MECCANO.SlowFast uses a high-frame-rate pathway for motion and a low-frame-rate pathway for semantics.Source: §2.2.1 · p. 4

2.2.2 HOI Detection

• Prior HOI detection work is chiefly third-person; this paper addresses the task from an egocentric perspective using MECCANO Multimodal.The reviewed methods include human-centric detection, graph-based relations, and representations connecting human and object boxes.Source: §2.2.2 · pp. 4–5

2.2.3 Tasks related to EHOI Detection

• Egocentric interaction studies include affordance and hand-centric approaches, but EHOI detection has not been studied systematically; the paper formalizes and investigates it.The related work describes active objects as those in contact with a hand, contrasting them with passive objects.Source: §2.2.3 · p. 5

2.2.4 Action Anticipation

• Action anticipation has been studied in third-person and first-person settings; the paper adopts RULSTM to evaluate methods on MECCANO with gaze and depth signals.Earlier first-person work includes LSTM-based methods for encoding past features and approaches that use goal representations.Source: §2.2.4 · pp. 5–6

2.2.5 Next Active Objects Detection

• Next-active-object detection remains underexplored, with few suitable egocentric datasets and no prior study combining varied signals with an industrial-like domain in this account.The paper focuses on this task using MECCANO, which is explicitly annotated for it.Source: §2.2.5 · p. 6

3. The MECCANO Multimodal Dataset

• MECCANO Multimodal consists of egocentric videos collected in an industrial-like domain.The section describes the dataset’s acquisition modalities.Source: §3 · p. 6
• The dataset simultaneously captures RGB videos, depth maps, and gaze signals.Acquisition uses two different devices.Source: §3 · p. 6
• The RGB, depth, and gaze streams are acquired as multimodal data for egocentric behavior-understanding research.Source: §3 · p. 6

3.1. Data Collection

• The motorbike uses 49 components across 16 classes, alongside two tools, an instruction booklet, and a partial-model class, for 20 object classes total.Some similar components are grouped as screws or washers; the partial-model class represents components assembled into part of the model.Source: §3.1 · p. 7
• Twenty participants in Italy and the United Kingdom assembled a toy motorbike using components arranged in heaps on light- or dark-colored tables.Participants came from eight nationalities and ranged in age from 18 to 55; they could use the available objects and instruction booklet freely.Source: §3.1 · pp. 6–7
• A custom headset captured RGB and depth with an Intel RealSense SR300 and gaze with a Pupil Core device.RGB and depth videos ran at 12 fps, while gaze was recorded at 200 Hz; RGB resolution was 1920×1080 and depth resolution was 640×480.Source: §3.1 · p. 7

3.2. Data Alignment

• The signals were aligned temporally and spatially to associate modalities consistently and allow annotations to be used with different signals.The headset combined an Intel RealSense SR300 with a Pupil Core device.Source: §3.2 · p. 7, Figure 5 · p. 8

3.2.1 Depth Alignment

• A constant 0.4-second depth–RGB offset was corrected, yielding 301,016 aligned depth frames.The offset arose because separate sensors acquired the depth and RGB streams.Source: §3.2.1 · p. 8

3.2.2 Gaze Alignment

• For each RGB frame, gaze alignment selected the nearest-timestamp gaze position with confidence of at least 0.6.Gaze data comprises 2D pixel coordinates in the RealSense RGB frame, confidence scores, and timestamps.Source: §3.2.2 · p. 8

3.3. Data Annotation

• The dataset’s annotations support five benchmark tasks: action recognition, active-object detection and recognition, EHOI detection, action anticipation, and next-active-object detection.The dataset is designed for multimodal human-behavior understanding in an industrial-like scenario.Source: §3.3 · p. 8

3.3.1 Action-Interaction Relations

• An action tracks hand motion and object-state change, while its related interaction begins at physical contact and ends when contact breaks.The action and interaction can overlap but have different start and end times; an interaction may also span more than one verb.Source: §3.3.1 · pp. 8–9, Figure 6 · p. 9

3.3.2 Action and Interaction Temporal Annotations

• Temporal segments record action start, first contact, and action end; 8,857 segments were annotated, with 1,401 overlapping another segment.For take, put, and align, start was set 0.5 seconds before contact; for check, contact was marked when the user was clearly looking at the object.Source: §3.3.2 · pp. 9–10, Figure 9 · p. 10, Figure 8 · p. 10

3.3.3 Active Object Bounding Box Annotations

• Active objects were labeled with class and bounding box every 0.2 seconds, totaling 64,349 labeled frames.Occluded active objects receive a class and estimated 2D position; multiple simultaneously active objects can be annotated.Source: §3.3.3 · p. 10, Figure 10 · p. 10

3.3.4 EHOI Annotations

• EHOI annotations represent an interaction as a verb paired with one or more active objects, with boxes localizing each object.Unlike a single-object interaction representation, this task does not localize the camera wearer, whose subject role is fixed.Source: §3.3.4 · pp. 10–11, Figure 11 · p. 11

3.3.5 Next Active Object Annotations

• Next-active-object labels place boxes on objects that will be active at contact, sampled backward up to three seconds before an interaction when past frames are available.Labels cover 6,656 interactions (75.66% of 8,857); when an object will be taken from a pile, the pile is labeled because the specific object cannot be inferred beforehand.Source: §3.3.5 · p. 11

3.3.6 Hands Annotations

• Hand boxes were annotated during interactions and in preceding frames, yielding 169,625 boxes across 89,628 frames.A detector provided initial hand locations and sides, which annotators checked and corrected or supplemented.Source: §3.3.6 · pp. 11–12, Figure 12 · p. 12

4. Benchmarks and Baseline Results

• The benchmark covers five behavior-understanding tasks, including action recognition, active-object detection and recognition, EHOI detection, action anticipation, and next-active-object detection.The authors identify EHOI and next-active-object detection as tasks not previously treated in the state of the art.Source: §4 · p. 12
• MECCANO addresses a gap in industrial egocentric multimodal evaluation, where few public datasets support studying several behavior tasks together.The authors note that the tasks had not been studied in industrial scenarios from an egocentric perspective with multimodal observations.Source: §4 · p. 12
• The dataset is divided into training, validation, and test sets balanced for desk type and acquisition country, with each video assigned wholly to one split.The splits are used for baseline training, hyperparameter tuning, and testing, respectively.Source: §4 · p. 12

4.1.1 Baseline

• Action recognition assigns an action class to an egocentric video segment; the SlowFast baselines test RGB, depth, and gaze signals in different combinations.Gaze is used to form an attention map that weights spatial feature maps in the network.Source: §4.1 · p. 12, §4.1.1 · pp. 13–14

4.1.2 Results

• RGB-Depth-Gaze achieves the best action-recognition results across all reported measures, while gaze adds only minor gains over RGB-Depth.RGB and Depth alone have similar Top-1 Accuracy (45.16% vs. 45.13%) and AVG F1-score (46.66% vs. 46.96%); their fusion improves results.Source: Table 3 · p. 14, §4.1.2 · p. 14

4.2.2 Results

• 38.14% AP is reached by the MECCANO-trained detector without a distance threshold, outperforming the reported Hand-Object Detector variants.The original detector variants obtain approximately 11% AP; re-training and changing the threshold also improve results.Source: Table 4 · p. 14, §4.2.2 · p. 15

4.2.1 Methods

• Active-object detection localizes objects involved in interactions, whereas active-object recognition also assigns one of the dataset’s 20 object classes.The reported detection baseline adapts a hand-object detector and Faster-RCNN; the recognition baseline uses Faster-RCNN with class predictions retained.Source: §4.2 · p. 14, §4.2.1 · p. 15

4.2.2 Results

• The object-detection results indicate that small objects remain challenging, while large objects such as screwdrivers and tires have higher reported AP.The authors leave investigation of more specific active-object detection approaches to future work.Source: §4.2.2 · p. 15

4.3. EHOI Detection

• APverb requires a matching verb and object-box overlap of at least 0.5 IoU; APverb,noun additionally requires the predicted object class to match.The authors also evaluate at IoU thresholds 0.3 and 0.1.Source: §4.3 · pp. 16–17

4.3.1 Method

• The baseline combines SlowFast verb prediction on an interaction-centered clip with Faster-RCNN detection and recognition of active objects.SlowFast is trained on the dataset’s 12 verb classes, and the object detector is the model used for active-object recognition.Source: §4.3.1 · p. 17

4.3.2 Results

• Depth-only obtains the best mAPverb values, while RGB-Gaze obtains the best mAPverb,noun values across the reported IoU settings.The depth-only mAPverb values are 29.10, 31.81, and 33.81; RGB-Gaze mAPverb,noun values are 21.38, 24.04, and 25.04.Source: §4.3.2 · p. 17
• Adding gaze attention to depth does not improve verb recognition, whereas RGB-Gaze is second-best on the reported lower-IoU verb measures.The authors suggest depth focuses attention on hands relevant to motion and verb discrimination; verb-and-noun scores are lower because they also require the correct object class.Source: §4.3.2 · p. 17

4.4.1 Method

• Action anticipation predicts an egocentric action from an observed segment preceding its start, with different anticipation times evaluated.The RULSTM baseline uses RGB, depth, gaze, object-centric, and hand-centric branches; RGB features replace the optical-flow branch used by the original approach.Source: §4.4 · p. 18, §4.4.1 · p. 18

4.4.2 Results

• The best anticipation model uses gaze, object-centric, and hand-centric branches, reaching Mean Top-5 Recall of 32.25 on the validation set.Using all five available signals does not guarantee the best performance, and RGB plus Depth performs worse than RGB alone in the reported ablation.Source: §4.4.2 · p. 18, Table 7 · p. 19
• The anticipation ablation compares combinations of five branches—RGB, Depth, Objects, Gaze, and Hands—and test results consider different anticipation times.The supplied results describe the validation-set best combination; the test-set table is identified as reporting results across anticipation times.Source: §4.4.2 · pp. 18–19, Table 7 · p. 19, Table 8 · p. 19

4.5.1 Method

• Next-active-object detection predicts the classes and bounding boxes of objects involved in a future interaction from an earlier temporal segment.The four Faster-RCNN baselines vary whether training uses active-object annotations, next-active-object annotations, or both.Source: §4.5 · p. 19, §4.5.1 · p. 19

4.5.2 Results

• Training with both active-object and next-active-object annotations gives the best reported next-object results: mAP 14.20 and mAP50 26.40.Training only on next-active-object annotations is not enough for reasonable performance.Source: §4.5.2 · pp. 19–20, Table 9 · p. 20

5. Conclusion

• MECCANO provides RGB, depth, and gaze data with temporal action and interaction segments and spatial boxes for active objects, next-active objects, and hands.Its taxonomy contains 12 verbs, 20 nouns, and 61 unique actions.Source: §5 · p. 20
• The dataset includes baseline experiments for five challenging behavior-understanding tasks, which the authors describe as demonstrating the usefulness of its multimodality.The paper also points to MECCANO as suitable for exploring tasks beyond those covered in this work [2].Source: §5 · p. 20
• The authors state that future work will explore approaches to improve performance on the proposed tasks.Source: §5 · p. 20

• The supplementary document reports additional information about action-interaction relations, the proposed dataset, and its annotation stage.Source: SUPPLEMENTARY MATERIAL · p. 20
• The material accompanies the MECCANO manuscript submitted to Computer Vision and Image Understanding in 2022.Source: SUPPLEMENTARY MATERIAL · p. 20

6. Next-Active Objects Annotations

• Annotators used a customized VIA project with object-class thumbnails and written rules to support next-active-object labeling and reduce ambiguity.The guidelines define cases such as next active objects in piles and occluded next active objects, with visual examples.Source: §6 · p. 20
• Figure 25 compares the number of MECCANO interactions with the number that include labeled past frames.Source: §6 · p. 20

7. Hands Annotations

• Figure 26 reports statistics related to the hand annotations.Source: §7 · p. 20, Figure 26 · p. 23
• Annotators corrected Hand Object Detector class errors and refined the bounding boxes around hands.Figure 27 illustrates the procedure by contrasting detector predictions with annotator corrections.Source: §7 · p. 20, Figure 27 · p. 23