Source-linked AI summary
Agentic AI-enabled Semantic Commissioning of a Cognitive Digital Twin for Reconfigurable Manufacturing
Yangyang Liu, Xun Xu, Jan Polzer
TL;DR
Bespoke commissioning remains a bottleneck because CDT construction must integrate semantic specifications with live industrial states rather than geometry alone. The paper proposes a LangGraph-orchestrated agentic workflow combining RAG for static documentation and MCP for dynamic telemetry binding, with HITL support. The system achieved 97.2% mAP@0.5 and reduced deployment lead time by 95% to an average of 2 hours, while the authors identify dependence on digital documentation as a limitation.
Problem
Bespoke CDT commissioning remains a severe industrial bottleneck because semantic mapping between heterogeneous physical assets and virtual counterparts is largely manual and time-consuming.
Method
The paper orchestrates LLM-based agents through LangGraph, combining RAG-based specification extraction, MCP-based telemetry binding, and HITL feedback within a CDT workflow.
Results
97.2% mAP@0.5 and a 95% reduction in deployment lead time were achieved, completing reconfiguration in an average of 2 hours.
Takeaways & Limitations
The results demonstrate feasible autonomous orchestration of high-fidelity, live-synchronised CDT commissioning in a robotic machining cell.
Takeaways & Limitations
The RAG-based enrichment assumes that digital documentation is available.
Abstract
from arXiv · showhide
Rapid bespoke commissioning of the Cognitive Digital Twin (CDT) is a major challenge in reconfigurable manufacturing. Traditional digital twin (DT) construction methods primarily focus on geometric reconstruction, often neglecting the deep semantic integration and functional interoperability necessary for autonomous reasoning. This paper proposes an agent-based, AI-driven workflow to automate end-to-end CDT debugging. The system utilises LangGraph as a multi-agent orchestration engine to achieve dual-path synthesis: the semantic path extracts technical specifications from unstructured documents using Retrieval Augmented Generation (RAG), while the functional path autonomously discovers and binds to real-time industrial telemetry data using Model Context Protocol (MCP). Experimental validation in a robotic machining cell demonstrates that the system achieves a mean average accuracy (mAP) of 97.2% in perception and reduces the deployment cycle from several weeks to an average of 2 hours, marking a paradigm shift from manual scripting to autonomous orchestration.
I. INTRODUCTION
The paper proposes an agentic workflow for rapidly commissioning a semantically rich, functionally interoperable CDT in reconfigurable manufacturing. It combines perception, document-based semantic enrichment, live telemetry binding, knowledge integration, and human feedback.
- Existing DT construction often prioritises geometric reconstruction while overlooking semantic enrichment and functional interoperability.A functional CDT must represent technical specifications and connect to live operational states, not merely mirror physical geometry.
- The proposed workflow uses vision-based monitoring to detect layout changes and retrieve corresponding 3D models for spatial reconstruction.
- RAG extracts static technical parameters from unstructured PDFs, while MCP enables autonomous discovery and subscription to OPC UA telemetry nodes.Together, the two paths provide static semantic enrichment and dynamic state synchronisation.
- Extracted spatial, static, and dynamic information is synthesised into a knowledge graph and rendered through a lightweight Three.js interface.
- Human-in-the-Loop intervention handles perceptual uncertainty, with feedback stored in persistent memory to improve future autonomy and robustness.
- 95% reduction in deployment lead time was reported, reducing commissioning from several weeks to several hours in a physical testbed.The implementation maintained high semantic fidelity while reducing bespoke commissioning time.
A. Evolution and Architecture of CDT
CDT research has evolved toward cognitive, knowledge-rich models, but bespoke commissioning remains a major barrier to industrial deployment. Semantic mapping is still largely manual and must address heterogeneous data, bespoke configurations, knowledge extraction, and system fragility.
- CDT architectures extend traditional digital twins with cognitive functions such as perception, reasoning, and learning.The field has progressed from ontology-driven models toward multidimensional cognitive models using knowledge graphs and related technologies.
- Bespoke commissioning remains a severe bottleneck when CDT systems move from laboratory settings to industrial shop floors.
- Semantic mapping between heterogeneous physical assets and virtual counterparts remains largely manual, time-consuming, and labour-intensive.
- Four reported challenges are heterogeneous data, bespoke configurations, complex knowledge extraction, and system fragility and maintenance.
B. LLM-driven Autonomous Agents in Manufacturing
Manufacturing agentic frameworks are progressing from high-level planning toward autonomous low-level execution. Graph-based orchestration, industrial interfaces, and HITL architectures support stateful, interoperable, and auditable workflows.
- Agentic frameworks emphasise iterative reasoning and autonomous tool-calling for multi-step engineering tasks beyond standalone LLM operation.
- Recent developments shift agentic manufacturing systems from high-level task planning toward autonomous low-level execution.
- LangGraph addresses complex, stateful manufacturing workflows by helping agents directly manipulate industrial assets through MCP and IIoT interfaces.
- Agents require understanding of Sequential Function Charts under IEC61131-3 to interpret and generate executable industrial logic.
- HITL architectures combine agentic reasoning with user interfaces to support transparency and auditability during real-world production deployment.
C. Knowledge Representation and Standardised Interfacing
Knowledge representation and standardised interfaces are central to CDT interoperability. The paper combines established representation standards with RAG and MCP to connect technical specifications and live industrial telemetry.
- ISO 23247 defines entity attributes, while RAMI 4.0 supports structural alignment within the administrative shell.
- RAG extracts specifications from unstructured documents to help bridge the semantic gap during data integration.
- MCP provides an open, modular standard for connecting LLM agents with external contexts and industrial tools.The paper applies MCP to automate binding to real-time industrial telemetry within a CDT framework.
III. AGENTIC FRAMEWORK REALISATION AND EXPERIMENTAL VALIDATION
The section presents a physical robotic machining cell implementation of the cognitive-function CDT framework and evaluates it through perception and system-integration experiments.
- The proposed framework is instantiated in a real-world–inspired robotic machining cell using an LLM-driven agentic workflow.
A. Experimental Setup & Bespoke Commissioning Context
The experimental setup is a reconfigurable machining cell with robots, conveyors, and RGB-D perception, designed to reflect frequent production-layout changes. Its stack combines YOLOv11, LangGraph-based reasoning, and Neo4j semantic memory.
- The testbed is a reconfigurable discrete manufacturing cell comprising three core entity groups.
- Two industrial robots perform assembly and machining tasks, while two automated conveyors support material handling.
- An Intel RealSense D435i RGB-D camera captures the cell’s spatial layout from above.
- Frequent production rotation requires layout adjustments and equipment swaps to accommodate different product variants.
- The technology stack uses YOLOv11 for detection and pose estimation, GPT-5.1 via LangGraph for multi-agent reasoning, and Neo4j as the knowledge-graph backbone.
B. The Agentic Bespoke Commissioning Workflow
The workflow treats bespoke commissioning as a dynamic reasoning task: it enriches detected equipment semantically, binds entities to telemetry through MCP, and persists the resulting configuration in a knowledge graph.
- The workflow replaces hard-coded commissioning scripts with a dynamic reasoning process for bespoke configuration.
- Stage I Hybrid Semantic Enrichment: YOLOv11 detections trigger MinerU-powered RAG, which extracts equipment attributes from PDF manuals and links visual bounding boxes to physical capabilities.
- Stage II Protocol Binding via MCP Tool-calling: MCP mediates protocol binding by mapping entity types to telemetry tools, including robot 6-DOF joint states and conveyor linear velocity.
- Stage III HITL and Memory Persistent: The final stage synthesises extracted information into a Neo4j knowledge graph that serves as the CDT’s semantic foundation.
- The configuration links geometric entities, RAG-extracted parameters, and OPC UA telemetry addresses for semantic persistence.
- Three.js loads and animates the 3D scene from the knowledge graph using real-time telemetry, while bidirectional OPC UA writes can send control signals to the shop floor.
- Human-in-the-loop intervention is triggered by low perceptual confidence, and corrections are stored in persistent memory for future autonomous actions.
C. Experimental Results and Evaluation
The evaluation compares the Agentic workflow with manual CDT commissioning and assesses perception, deployment efficiency, and semantic fidelity in a robotic machining context. The workflow combines visual monitoring with reconstructed and knowledge-enriched digital scenes.
- Perception Performance: YOLOv11 perception was evaluated through object detection and pose estimation in complex workshop environments.The evaluation used 588 annotated images and an iterative model-assisted plus human-correction training strategy.
- Commissioning Efficiency: The study compares autonomous commissioning with manual 3D asset alignment, parameter entry from manuals, and OPC UA node binding.These activities define the traditional bespoke commissioning baseline.
- Commissioning Efficiency: Several weeks are required for a skilled engineer to configure four major manufacturing entities manually.The manual process includes data binding and spatial configuration.
- Commissioning Efficiency: The Agentic pipeline completes commissioning from raw camera input to live Three.js rendering in an average of 2 hours.The cycle covers the autonomous pipeline through live visualisation.
- Results and Fidelity: 95% deployment-lead-time reduction is reported relative to the multi-week manual commissioning process.The generated CDT also integrates RAG-extracted parameters and live industrial data streams into knowledge-enriched virtual entities.
- Visualisation: The evaluation materials include a DT Editor view for spatial reconstruction and initial geometric-semantic mapping, plus a live dashboard for MCP-enabled telemetry synchronisation.These views represent the transition from reconstructed scene geometry to live industrial data integration.
IV. CONCLUSION & FUTURE WORK
The paper presents a LangGraph-orchestrated, LLM-driven CDT workflow that connects physical perception with live-synchronised digital representations. In a robotic machining cell, it reports high perceptual reliability and substantially shorter deployment lead time, while identifying documentation availability as a current boundary.
- LangGraph orchestrates a multi-agent CDT workflow using LLM reasoning to bridge physical perception and live-synchronised digital representations.
- The workflow demonstrates autonomous CDT commissioning through YOLOv11 perception, RAG-based static-parameter extraction, and MCP-based dynamic industrial binding.
- The RAG enrichment currently assumes that digital documentation is available.Future work targets multimodal MLLMs for legacy blueprints and hand-drawn diagrams, and a distributed KG for cross-factory learning.