PaperlayerView exact revision on arXiv

Source-linked AI summary

DYAD: A Multimodal Dataset of Co-Located Human Assistance

Akhil Ajikumar, Mahya Qorbani, Sakib Reza, Sean Andrist, Mohsen Moghaddam

arXiv:2609.09023v1cs.RO

TL;DR

Co-located assistance requires estimating task state and need before choosing a response, yet existing data do not link those decisions to requests, interventions, and outcomes. DYAD records that interaction structure and evaluates selected components; its mode results differ across RGB, causal metadata, and a privileged trigger map.

  • Problem

    A co-located assistant must estimate task state and need before selecting a response, but existing datasets do not jointly link requests, task state, triggers, interventions, and outcomes.

    Source: §I · p. 1

  • Method

    DYAD records 20 performers assisted by one co-located helper, linking 528 task steps and 611 requests to 851 assistance events under synchronized sensing, and evaluates three selected reference tasks.

    Source: §VI · p. 8

  • Results

    Step understanding yields 17.4 segment F1@0.1, response baselines 3.58–3.69/5 under oracle context, and mode anticipation 0.548±0.007 RGB macro-F1 versus 0.624 causal metadata.

    Source: §VI · p. 8

  • Takeaways & Limitations

    The linked timeline supports future benchmarks for need detection, grounding, timing, triggers, and embodied assistance, while outcome prediction requires more diversity.

    Source: §VI · p. 8

    The dataset covers 20 sessions from one gearbox task, site, and helper; participant-grouped evaluation tests new performers only within that setting, and the single-helper design entangles mode labels with personal habits.

    Source: §V · p. 7

Abstract

from arXiv · show

An embodied assistant working beside a person must track task state, recognize help seeking, choose how to intervene, and produce an appropriate response. Existing procedural datasets richly describe individual execution, while interactive datasets capture remote verbal instruction or undifferentiated co-working. They do not jointly link a co-located helper's verbal and physical interventions to performer requests, task state, assistance triggers, and outcomes. We introduce DYAD (DYadic Assistance Dataset), a synchronized multimodal record of human-human assistance during gearbox assembly. Across 20 sessions, one trained helper follows a guidance-first policy while assisting HoloLens 2 wearers. DYAD links 528 task-step intervals and 611 performer requests with 851 valid assistance records spanning verbal and physical help. DYAD's annotations span the assistance process; three reference tasks evaluate selected components rather than an end-to-end system: causal step understanding, pre-onset mode anticipation, and instructor response generation. On 829 eligible mode events, the strongest four-seed RGB mean is 0.548 +/- 0.007 macro-F1; causal metadata reaches 0.624 and a privileged trigger mapping 0.915, revealing information not recovered from pre-onset RGB. DYAD's contribution is not scale, but a linked interaction structure spanning help seeking, intervention choice, execution, and outcome under egocentric and workspace sensing.

I. INTRODUCTION

A co-located assistant must estimate task state and need before choosing and producing a response, but existing datasets do not link those decisions to requests, interventions, and outcomes. DYAD supplies that interaction structure and evaluates three components separately.

I. INTRODUCTION

  • Procedural, remote-guidance, and co-located collaboration datasets lack a co-located timeline linking task progress, requests, triggers, interventions, and outcomes –.Without that linkage, a model may recognize actions or errors but cannot learn how responses depend on task state, initiation, and assistance need.Source: §I · p. 1
  • DYAD links 528 task-step intervals and 611 requests to 851 valid assistance records across 20 gearbox-assembly sessions, using synchronized egocentric and workspace sensing.One trained helper follows a guidance-first policy, so mode labels describe that policy rather than a universal prescription.Source: §I · p. 1
  • Three reference tasks separately assess causal task-state estimation, mode selection for an upcoming intervention, and verbal response generation; they are not an end-to-end assistant.Continuous need detection is not benchmarked because non-intervention negatives were not sampled, and outcome prediction is excluded because 812 of 827 labeled outcomes are effective.Source: §I · p. 1, Figure 1 · p. 2
  • 0.624 macro-F1 with causal metadata exceeds the strongest four-seed RGB mean of 0.548, while a privileged trigger map reaches 0.915.Step and response baselines also remain limited under causal or privileged inputs.Source: §I · p. 1

II. RELATED WORK

Prior datasets provide procedural execution labels or interactive assistance records, but differ in whether they capture co-located modality choice and linked assistance outcomes. Related work also motivates causal evaluation and separates response content from perception errors.

a) Procedural data

  • Procedural datasets provide execution, action, pose, step-completion, error, and multi-view supervision, but do not record whether help was requested or whether a helper’s response resolved the need –.Ego4D records long-form egocentric activity; IKEA ASM, Assembly101, IndustReal, and Ego-Exo4D add assembly labels, errors, or paired views and commentary.Source: §II · p. 2

b) Interactive assistance

  • HoloAssist captures remote verbal instruction, IndEgo records co-located peers without linked request–intervention–outcome events, and Vinci answers questions from live egocentric video –.DYAD centers on linked co-located requests, verbal and physical interventions, and outcomes.Source: §II · p. 2

c) Causal understanding and response evaluation

  • Causal online recognition avoids using an observed intervention that would leak DYAD’s mode target, while oracle context isolates response content from perception and dialogue-history errors –.Progress-aware online segmentation motivates causal inputs and frame- and segment-level metrics; response evaluation combines BERTScore with a task-specific judge.Source: §II · p. 2
  • PACE uses action-completion estimates to time proactive robotic assistance, whereas DYAD records both cognitive and physical assistance needs.DYAD mode prediction conditions on an upcoming event; need detection is separate and requires non-intervention negatives.Source: §II · p. 2

A. Collection Rationale and Protocol

The collection pairs a HoloLens 2 performer assembling a gearbox with a nearby helper, using task features intended to elicit varied assistance needs. A single trained helper applies a guidance-first policy across the recorded sessions.

A. Collection Rationale and Protocol

  • Twenty recorded sessions pair a HoloLens 2 wearer assembling a physical gearbox from a CAD reference with a nearby helper, following six pilot sessions to refine the task and protocol.The sessions total 5 h 58 min, averaging 17 min 55 s and ranging from 8 min 57 s to 29 min 40 s.Source: A. Collection Rationale and Protocol · p. 2
  • Similar parts, ordering constraints, out-of-reach items, and unstable subassemblies elicit search, confusion, ordering, reach, and stabilization needs without scripting individual mistakes.The reference task is designed to surface assistance needs through the work setting rather than staged errors.Source: A. Collection Rationale and Protocol · p. 2
  • The same trained helper follows a guidance-first policy: allow 5–10 s for self-correction, intervene after a visible stall, and use physical help for reach or stabilization.Mode labels describe behavior under this helper’s policy, not a universal prescription.Source: A. Collection Rationale and Protocol · p. 2

B. Sensing, Release, and Annotations

DYAD combines synchronized sensing with aligned task-step, request, and assistance-event annotations. Its event schema captures intervention details and interaction patterns, while the reported counts and reliability audit specify the scope of those records.

B. Sensing, Release, and Annotations

  • The planned release includes synchronized modalities, and deidentified ego RGB and depth, gaze, hand joints, head pose, Kinect RGB-D, timestamps, annotations, task materials, and benchmark code are specified for controlled access.Audio was collected but will not be released; deidentified request and helper transcripts will be provided, and faces are blurred.Source: B. Sensing, Release, and Annotations · p. 3, Table II · p. 3, Figure 2 · p. 3
  • Three aligned layers contain 528 task-step intervals, 611 transcribed performer requests, and 851 valid assistance events with mode, initiation, hard-stop, outcome, trigger, object, and utterance annotations.The export has 852 rows; removing one malformed duplicate leaves 851 valid events, of which 850 have mode labels and 829 have eligible pre-onset RGB windows.Source: B. Sensing, Release, and Annotations · pp. 3–4
  • The reliability audit blindly relabeled one full session and prespecified windows from seven others, matching 144 of 147 reference assistance events within 1 s before adjudication.The audit covered eight participants and 17.3% of the corpus; agreement was also assessed for task-step and request streams.Source: B. Sensing, Release, and Annotations · p. 4
  • Verbal guidance is 2.7 times as common as physical help; reach and stabilization are about 92% physical, while decision, ordering, and orientation problems are 95–100% verbal.Out-of-reach help is usually requested, whereas ordering violations are commonly caught by the helper; outcome prediction is not benchmarked because 812 of 827 labeled outcomes are effective.Source: B. Sensing, Release, and Annotations · p. 4

IV. REFERENCE TASKS

The reference tasks evaluate selected components of the assistance loop rather than an end-to-end assistant. Their protocols distinguish causal inputs from post-hoc information and treat the mode target as one helper’s observed behavior under the collection policy.

Reference-task scope

  • The protocols evaluate task-state estimation, mode selection conditional on an upcoming intervention, and verbal response generation; they do not measure an end-to-end system.The tasks have no shared score.Source: §IV · p. 4
  • Assistance-need detection is not benchmarked because non-intervention intervals are not sampled as negatives.Source: §IV · p. 4
  • The mode target represents one helper’s observed action under the collection policy, not a universal prescription.The contracts also guard against treating post-hoc trigger labels or ground-truth dialogue history as causal inputs.Source: §IV · p. 4

A. Assistance-Mode Anticipation

Mode anticipation predicts a helper’s demonstrated verbal or physical intervention from pre-onset inputs, assuming an intervention will occur. Results show that causal metadata outperforms RGB, while trigger-based performance is an annotation upper bound rather than deployable prediction.

Protocol

  • The task predicts the helper’s demonstrated verbal or physical mode from a 3 s window ending 0.5 s before onset, assuming an intervention will occur.The gap reduces leakage from the helper’s first reach or utterance.Source: A. Assistance-Mode Anticipation · p. 4
  • The RGB protocol uses participant-grouped folds and prohibits current-event triggers, initiation, outcome, speech, task-step labels, and participant identity.Macro-F1 is primary because physical assistance is the minority class; the causal metadata control uses timing and prior counts but no pixels, trigger, step label, or transcript.Source: A. Assistance-Mode Anticipation · p. 4

Results and limitations

  • 0.548±0.007 macro-F1 is the highest four-seed RGB mean (X3D-S); causal metadata reaches 0.624 and the privileged trigger map reaches 0.915.Source: A. Assistance-Mode Anticipation · p. 5
  • For cannot_find_part, VideoMAE physical recall is 0.103 versus metadata’s 0.793.The trigger mapping is an annotation upper bound because triggers are assigned after annotators see the completed intervention.Source: A. Assistance-Mode Anticipation · pp. 4–5
  • All scores aggregate 829 unique out-of-fold predictions, but the task cannot establish whether or when help is needed or whether a mode is universally appropriate or robot-feasible.The evaluation target is one helper’s action under one policy, not universal appropriateness or robot feasibility.Source: A. Assistance-Mode Anticipation · p. 5

B. Causal Assembly-Step Understanding

Causal assembly-step understanding tests current-step recognition and temporal boundary estimation using only observations available at decision time. The results show metric trade-offs across models and gains from temporal decoding components, while remaining bounded to a closed vocabulary and one procedure.

Protocol

  • The task predicts one of eight assembly steps from ego RGB sampled at 1 Hz, using only frames at or before each decision time to estimate steps and boundaries.Future frames, requests, assistance labels, and helper utterances are withheld.Source: B. Causal Assembly-Step Understanding · p. 5

Baselines and results

  • No model dominates every metric: Qwen3-VL-2B has the highest segment F1 (17.4), Qwen3-VL-8B the highest edit score (28.3), and Qwen3-VL-4B the highest localization mAP (8.9).The majority baseline’s stronger frame metrics reflect class prevalence rather than useful temporal localization.Source: B. Causal Assembly-Step Understanding · p. 5
  • Removing PMI reduces segment F1 from 17.4 to 12.4; removing minimum dwell reduces it to 15.6, while slightly increasing localization mAP and MoF.Removing EMA lowers every reported metric.Source: B. Causal Assembly-Step Understanding · p. 6

Limitation

C. Instructor Response Generation

The response-generation benchmark tests concise, context-grounded replies to performer questions using oracle task and interaction history. GPT-4.1 scores higher on task-aware final score, while GPT-4o is briefer and has higher contextual-embedding similarity; the task excludes physical-only responses and relies on an LLM judge not validated on DYAD.

Protocol

  • The benchmark generates concise, contextually grounded helper utterances for questions matched to verbal assistance within a window from 5 seconds before to 15 seconds after request end.Requests answered solely through physical action are excluded; the development set has 120 pairs and the participant-disjoint test set has 256.Source: C. Instructor Response Generation · p. 6

Context

  • Oracle context supplies procedure knowledge, annotated task and interaction history, a gearbox image, and the current question, while withholding the current trigger.Ground-truth task and dialogue history isolate response quality from upstream perception and accumulated model errors; this is not a deployable video-to-response setting.Source: C. Instructor Response Generation · p. 6

Baselines, metrics, and results

  • GPT-4o and GPT-4.1 receive identical prompts, while Claude Opus 4.5 receives the same oracle context and human response as reference; evaluation combines task-aware judge scores with BERTScore.The judge scores correctness, completeness, contextual relevance, actionable clarity, brevity, and holistic quality on a 0–5 scale.Source: C. Instructor Response Generation · p. 6
  • GPT-4.1 has the higher task-aware final score, while GPT-4o is briefer and has higher contextual-embedding similarity to the reference.Final-score means vary by session: 2.756–4.364 for GPT-4o and 2.833–4.364 for GPT-4.1; the authors treat them as representative baselines, not a model ranking.Source: C. Instructor Response Generation · p. 7, Table XIV · p. 7

Limitation

  • The task excludes requests answered only through physical action, tests two proprietary model families, and uses an LLM judge not directly validated by human ratings on DYAD.The five development sessions were used for prompt engineering, while held-out sessions were not used to revise the prompt or evaluation.Source: C. Instructor Response Generation · pp. 6–7

V. DISCUSSION

The discussion highlights DYAD’s linked assistance structure and reports distinct findings across its reference tasks and observed interaction patterns. It also bounds what the dataset and benchmarks establish, and identifies collection practices that could strengthen future data.

Why use DYAD?

  • DYAD centers the assistance interaction by linking a designated helper’s verbal and physical interventions to requests, task state, triggers, and outcomes.This differs from datasets centered on scale, broader activities, remote verbal instruction, or co-located peer collaboration.Source: §V · p. 7
  • The three reference tasks evaluate selected components, while the release also supports further benchmarks once new evaluation protocols are defined.Potential tasks include continuous need detection, trigger recognition, request–response matching, hard-stop prediction, object grounding, and sensing ablations.Source: §V · p. 7

What the reference tasks show

  • The mode-anticipation ordering is privileged trigger map ≫ causal metadata > RGB, while RGB stacking does not reliably improve on metadata.Weak VideoMAE physical recall for within-search events further indicates that pre-onset appearance alone does not explain the helper’s policy.Source: §V · p. 7

From mode prediction to assistive policy

  • A mode-prediction gate could route verbal decisions to dialogue or AR guidance and physical decisions to action selection, but DYAD does not benchmark need inference, planning, safety, or motor execution.Handovers and stabilization require different branches, grounding, control, and safety assumptions; human actions are not robot commands.Source: §V · p. 7

Interaction patterns and collection lessons

  • Assistance events outnumber explicit requests by 1.39:1, and 270 of 851 interventions are helper-initiated.Out-of-reach help is usually requested, while ordering violations are often caught proactively; hard stops also concentrate in ordering and orientation errors.Source: §V · p. 7
  • The annotation export lacked persistent event IDs and mixed session-relative with absolute-style clocks, requiring ID reconstruction and stored clock transforms before windowing.Future collections should record IDs and clock transforms at capture time, distinguish missing from out-of-taxonomy values, and permit nested or multilabel assistance when speech and physical help overlap.Source: §V · p. 7

Limitations

  • DYAD’s 20 sessions come from one gearbox task, site, and helper, so participant-grouped evaluation tests new performers only within that setting.The single-helper design controls policy variation but entangles mode labels with that helper’s habits; physical assistance and several triggers also have limited support.Source: §V · p. 7
  • Mode anticipation assumes a known intervention and cannot detect need; step understanding assumes a fixed vocabulary, and response generation uses privileged context and an unvalidated LLM judge.Outcome prediction is not reported because 812 of 827 labeled outcomes are effective; meaningful modeling requires targeted failure and recovery collection.Source: §V · p. 7

VI. CONCLUSION

DYAD records co-located human assistance during gearbox assembly and links task steps and requests to assistance events. Its reference tasks evaluate selected components, with results varying across task, context, and input type.

Conclusion

  • DYAD records 20 performers assisted by one co-located human helper, linking 528 task steps and 611 requests to 851 assistance events under synchronized sensing.The linked records span verbal and physical assistance under a single helper.Source: §VI · p. 8
  • On 829 eligible mode events, RGB reaches 0.548±0.007 macro-F1, causal metadata reaches 0.624, and the privileged trigger upper bound is 0.915.The same summary reports 17.4 segment F1@0.1 for step understanding and instructor-response baselines of 3.58–3.69/5 under oracle context.Source: §VI · p. 8
  • The timeline supports future benchmarks for need detection, grounding, timing, triggers, and embodied assistance, while outcome prediction requires more diversity.The paper presents DYAD’s linked assistance structure as making components observable before integration into robotic or AR assistance policies.Source: §VI · p. 8
Loading 2609.09023v1…