Source-linked AI summary
AI-based worker guidance in assembly and disassembly operations using multimodal ego/exo-centric data capture and structured task knowledge
Vivek Chavan, Jörg Krüger
TL;DR
Assembly and disassembly still depend on expert knowledge that is difficult to capture and reuse, especially for long-horizon, context-dependent procedures. The paper combines multimodal egocentric and exocentric demonstrations with narration to extract validated task structure and guide workers during execution. A real-world workstation disassembly study shows reliable action recognition, precedence-graph construction, and valid next-step recommendation, including from a single expert demonstration.
Problem
Existing systems provide limited support for capturing and reusing expert procedural knowledge in long-horizon assembly and disassembly tasks.
Method
The approach synchronizes egocentric and exocentric recordings with narration, extracts a validated precedence graph, and combines it with video representations for context-aware guidance.
Results
The workstation disassembly case study reliably recognizes actions, constructs precedence graphs, and recommends valid next steps, with meaningful guidance possible from a single expert demonstration.
Takeaways & Limitations
Structured precedence-graph reasoning supports advisory worker assistance for realistic industrial disassembly and other procedural tasks.
Takeaways & Limitations
Real-time mistake detection, generalization across similar but non-identical products, and transfer to robotic learning remain future work.
Abstract
from arXiv · showhide
Assembly and disassembly processes rely on expert knowledge that is difficult to document, reuse, and transfer. This paper presents a data-centric approach for extracting structured task knowledge from expert demonstrations using egocentric and exocentric recordings. Temporal and multimodal information from video and narration is jointly encoded to derive structured task representations that enable procedural documentation and context-aware worker guidance. The approach is evaluated on a real-world disassembly case study, demonstrating that video-based representations capture procedural structure and execution context beyond static image-based methods. The results highlight the potential of egocentric video understanding for repair, training, and circular manufacturing applications. Project website: https://indego-assistant.github.io/
1. Introduction
Industrial assembly and disassembly face difficult, variable conditions that limit rigid automation and make long-horizon procedures hard to model. The paper addresses this gap by extracting structured task knowledge from combined egocentric and exocentric expert demonstrations for context-aware guidance.
- High-mix, low-volume production and Circular Economy demands make flexible assembly and disassembly increasingly important.
- Contact-rich operations involve tight tolerances, non-linear dynamics, and deformable objects that challenge traditional automation.
- Disassembly variability from wear, damage, missing components, and undocumented conditions makes predefined CAD-based trajectories unreliable.
- Existing multimodal AI approaches often address short-horizon tasks or static observations, while long-horizon procedures remain difficult because of temporal complexity and context dependence.
- The framework combines egocentric and exocentric expert data to capture procedural knowledge in a precedence graph and support context-aware next-step guidance.
2. Methodology
The methodology converts synchronized expert recordings, narration, and task structure into validated procedural graphs and video representations. During execution, recognized actions are mapped to graph state to recommend the next permissible step and provide adaptive guidance.
- 2.1. Egocentric and exocentric data capture [A]: A combined setup records first-person video, gaze, audio narration, head and hand motion, plus a stationary third-person workspace view.Egocentric sensing captures hands-on interactions, while exocentric sensing supplies global scene context and spatial relationships.
- 2.1. Egocentric and exocentric data capture [A]: Experts narrate actions, reasoning, constraints, and common errors while synchronized sensor streams provide inputs for procedural modelling and worker guidance.
- 2.2. From ego/exo-centric recordings to structured task representation [B-E]: WhisperX transcribes narration into timestamped text that temporally links continuous vision with procedural intent and task structure.
- 2.2. From ego/exo-centric recordings to structured task representation [B-E]: LLaMA 3.1 extracts tasks, non-linear dependencies, and parallel branches from narrated demonstrations, enabling automatic precedence-graph construction from one sequential recording.
- 2.2. From ego/exo-centric recordings to structured task representation [B-E]: The expert reviews and corrects the directed acyclic precedence graph before it becomes a structural prior for video recognition and guidance.Graph nodes denote procedural steps, and edges encode precedence and dependency relations.
- 2.4. Worker guidance and task execution support [G-K]: During execution, online egocentric action recognition updates execution history, which a language model maps to graph state to generate the next permissible instruction.Guidance can be delivered through text, narration, and retrieved example recordings.
- 2.4. Worker guidance and task execution support [G-K]: The framework can incorporate manufacturer documentation, structural priors, and component recognition linked to a digital product passport for traceability and lifecycle-aware disassembly.
3. Case study
The case study evaluates the approach on workstation disassembly as a representative WEEE task with interacting components, tool use, and nontrivial dependencies. Its corrected precedence graph captures both fixed sequences and interchangeable removal branches.
- 3. Case study: WEEE disassembly must accommodate wear, missing components, undocumented modifications, and diverse end-of-life variants.These uncertainties limit rigid product-specific procedures and complicate process modelling.
- 3. Case study: The IndEgo case study uses synchronized first-person video, third-person video, and spoken expert narration from industrial assembly and disassembly recordings.
- 3. Case study: Workstation disassembly tests structured task representation, video understanding, and worker guidance across multiple components, tool usage, and procedural dependencies.AI-generated outputs are compared with manually annotated ground-truth sequences for procedural verification.
- 3. Case study: The final precedence graph spans preparation through cleanup and represents both strict dependencies, such as Cabinet → Duct → RAM, and parallel component-removal branches.
- 3. Case study: The procedure includes safety and workplace checks, internal-component removal in partly variable order, and final completion checks, sorting, and tool return.
4. Experimental evaluation
The pipeline is evaluated in two demonstration settings, using structured outputs to assess video-label generation, procedural-graph extraction, and worker guidance. Multiple demonstrations improve recognition robustness, while a single demonstration already supports meaningful procedural structure.
- Experimental setup: Two configurations compare one untuned expert demonstration with four recordings used to fine-tune the video model.Both settings generate structured transcripts, video-label pairs, a precedence graph, and procedural instructions; guidance is evaluated on a separate execution recording.
- Quantitative results: High video-label pair IoU indicates strong agreement with manual annotations.
- Quantitative results: PG generation achieves high step coverage and edge-level F1, indicating reliable extraction of procedural steps and dependencies.
- Guidance evaluation: A single expert demonstration yields meaningful procedural structure for action recognition and worker guidance.
- Guidance evaluation: Multiple demonstrations improve recognition accuracy by adding robustness and covering greater execution variability.
- Guidance evaluation: Occasional recognition errors are tolerated without disrupting overall procedural flow, supporting reliable assistive recommendations.
- Practical considerations: End-to-end processing of a complete expert recording requires approximately 1 h to produce the multimodal task model.
5. . Conclusions and outlook
The paper concludes that multimodal egocentric and exocentric data can generate structured assistance for assembly and disassembly. Its case study supports action recognition, precedence-graph construction, and valid next-step recommendations, while future work targets broader automation capabilities.
- Conclusions: The approach automatically extracts procedural knowledge from expert demonstrations and supports context-aware worker guidance during execution.
- Conclusions: A real-world workstation disassembly case study shows reliable action recognition, precedence-graph construction, and valid next-step recommendation.
- Conclusions: Meaningful procedural guidance can be derived from one expert demonstration, with multiple recordings improving robustness.
- Conclusions: The method is particularly valuable for disassembly and reassembly, where explicit procedural knowledge such as precedence graphs is typically unavailable.
- Outlook: Future work includes real-time mistake detection, cross-product procedural understanding, and transferring egocentric knowledge to robotic learning.
CRediT authorship contribution statement
The contribution statement assigns authorship across writing, visualization, validation, software, methodology, investigation, data curation, conceptualization, supervision, resources, funding, and formal analysis.
- Contributions: Vivek Chavan contributed to writing, visualization, validation, software, methodology, investigation, data curation, and conceptualization.
- Contributions: J€org Kr€uger contributed to writing, validation, supervision, resources, methodology, funding acquisition, formal analysis, and conceptualization.