Source-linked AI summary
IRWOZ 2.0: A Large Language Model-driven Dialogue Dataset for Industrial Robot Conversations
Chen Li, Dimitrios Chrysostomou
TL;DR
Industrial HRI dialogue datasets require accurate domain-specific annotations, but manual collection and correction can introduce noise and remain labor-intensive. IRWOZ 2.0 combines LLM-based generation with manual and automated quality refinement, producing 390 dialogues across four industrial domains. Its benchmark results include a GPT-2 BLEU-4 increase from 0.1651 to 0.5604 over IRWOZ, alongside improved joint goal accuracy, while some perplexity and slot-accuracy trade-offs remain.
Problem
Industrial dialogue datasets face annotation noise and must preserve specialized terminology and task-relevant information during correction.
Method
IRWOZ 2.0 combines Mistral and Claude-3.5 generation with structured prompts, manual correction, automated typo correction, and hybrid validation.
Results
GPT-2 BLEU-4 increases from 0.1651 to 0.5604, while GPT-2 joint goal accuracy rises from 0.801 to 0.824 on IRWOZ 2.0 versus IRWOZ.
Takeaways & Limitations
IRWOZ 2.0 provides a more comprehensive industrial HRI dataset for developing dialogue systems across Assembly, Delivery, Position, and Relocation.
Takeaways & Limitations
IRWOZ 2.0 has slightly higher perplexity and lower slot accuracy for some model sizes than IRWOZ.
Abstract
from arXiv · showhide
IRWOZ has improved industrial human-robot interaction (HRI) dialogue systems through domain-specific annotations. However, its initial version contains substantial noise in dialogue states and utterances, limiting state-tracking accuracy. We introduce IRWOZ 2.0, which addresses these limitations through large language model (LLM) enhanced generation (Mistral/Claude-3.5) and quality refinements. Our improved dataset expands to 390 dialogues across 4 industrial domains (Assembly, Delivery, Position, Relocation), featuring manual corrections and automated typo removal. Benchmark experiments on dialogue state tracking demonstrate significant improvements, with GPT-2's BLEU-4 score increasing from 0.1651 to 0.5604 compared to original IRWOZ. To support industrial HRI research, we publicly released IRWOZ 2.0 dataset at https://ieee-dataport.org/documents/irwoz-20-large-language-model-driven-dialogue-dataset-industrial-robot-conversations
1 Introduction
Industrial HRI dialogue systems need high-quality, domain-specific data, but manual collection and annotation introduce noise and limit scalable coverage. IRWOZ 2.0 addresses these issues with LLM-assisted correction and generation across four industrial domains.
- Motivation: Manual Wizard-of-Oz collection is costly, time-consuming, and difficult to scale across diverse industrial scenarios.Multi-annotator involvement can also introduce inconsistencies that limit comprehensive domain coverage.
- Motivation: More than 20% error rates have been reported in manually annotated industrial dialogues, affecting state tracking and response generation.The original IRWOZ contains substantial noise in annotations and utterances.
- Approach: IRWOZ 2.0 combines automated LLM error detection and correction, structured domain-specific prompting, and hybrid validation with expert review.The framework uses Mistral and Claude-3.5 for dataset enhancement.
- Dataset: 390 dialogues span Assembly, Delivery, Position, and Relocation, expanding coverage while targeting diverse industrial interaction scenarios.The dataset uses LLM-assisted generation to broaden industrial domain coverage.
2 Related Work
Related dialogue datasets provide structured resources for task-oriented systems, but most emphasize general domains rather than industrial robotics. IRWOZ 2.0 combines LLM scalability with domain-specific knowledge to address that gap.
- Task-oriented dialogue datasets: Task-oriented dialogue datasets support systems that model dialogue states, action spaces, and domain-specific knowledge bases.They are commonly created through manual annotation or dialogue collection.
- Task-oriented dialogue datasets: MultiWOZ and SGD broaden multi-domain dialogue research through large collections and schema information describing entities and actions.SGD includes over 20,000 dialogues across 16 domains.
- Research gap: Existing benchmark datasets primarily target general domains such as travel and hospitality, leaving industrial robotics underrepresented.Industrial robotics requires specialized vocabulary and task-specific interactions.
- LLMs for dataset correction: LLMs can correct typographical and grammatical errors, but industrial dataset correction must preserve specialized terminology and task-relevant information.This domain-specific preservation requirement distinguishes industrial robotics from generic text correction.
- LLMs for dataset generation: LLM-generated datasets offer scalability and conversational diversity, whereas traditional datasets provide stronger domain-specific knowledge and structured annotations.IRWOZ 2.0 adopts a hybrid approach combining these advantages for industrial conversations.
3 Dataset Corrections
IRWOZ 2.0 corrects errors in the original Wizard-of-Oz dataset through manual verification and Claude-3.5-assisted typo correction. The process targets incomplete markups, mis-annotations, and textual errors while preserving industrial terminology.
- Original dataset: The original IRWOZ was collected through Wizard-of-Oz interactions across Delivery, Position, Assembly, and Relocation domains.Participants played wizard and shop-floor-worker roles using interfaces connected to robot resources and APIs.
- Error types: Original-dataset errors include typos, incomplete markups, and values assigned to incorrect slot types.Examples include extracting only “shop” from “shop floor” and labeling a task-related response as small talk.
- Manual corrections: Manual correction verifies each slot value against the user utterance and each response against dialogue context and task requirements.These checks address mis-annotations and incomplete markups requiring contextual judgment.
- Manual corrections: 2.2% of turns contain incomplete markups and 3.2% contain incorrect annotations.These figures summarize the statistical analysis of manual corrections.
- Automated corrections: 4.3% of turns contain typos or misspellings corrected through Claude-3.5-assisted processing.Claude-3.5 was configured to recognize and preserve industrial terminology while correcting textual errors.
4 Dataset Generation
IRWOZ 2.0 is generated through a prompt-driven framework using Claude-3.5 and Mistral to produce industrial HRI dialogues across multiple domains. Its statistics describe short operator interactions, varied response lengths, and broader flexibility for task-related responses than small talk.
- Overall Framework: The framework uses task-specific prompts, Claude-3.5, and Mistral to generate dialogues for industrial HRI domains.Prompts define dataset structure, database information, and behavioral requirements for the language models.
- Prompt Design: Prompt instructions specify dialogue structure, database information, and behavioral constraints for generating and verifying belief states and system actions.The prompts include user input, dialogue belief state, system action, system responses, and rules for handling unavailable products.
- Prompt Design: The prompts incorporate industrial terminology, disambiguation, confirmation, error handling, and both task-related and small-talk responses.These requirements are intended to represent realistic operator-robot interactions across industrial tasks.
- Data Statistics: 390 dialogues were generated across Assembly, Delivery, Position, and Relocation, with each domain containing mostly one-turn dialogues.Across domains, approximately 54%-55% of dialogues have one turn, 33%-35% have two turns, and 10%-11% have three turns.
- Data Statistics: Task-related responses are longer and more variable than user utterances and small-talk responses across the four industrial domains.Average task-response lengths range from 21.35 to 26.71 words, while small-talk responses average 10.30 to 12.90 words; Delivery peaks at 62 tokens in the distribution.
5 Benchmark Evaluation
IRWOZ 2.0 generally improves dialogue generation and goal accuracy over IRWOZ across GPT-2 variants, although some perplexity and slot-accuracy trade-offs remain.
- BLEU performance: IRWOZ 2.0 consistently improves BLEU scores over IRWOZ across gpt2, gpt2-medium, and gpt2-large models.For gpt2, BLEU-1 rises from 0.3816 to 0.7370 and BLEU-4 from 0.1651 to 0.5604.
- Dialogue state tracking: 0.824 joint goal accuracy is achieved by gpt2 with IRWOZ 2.0, up from 0.801 with IRWOZ.The improvement is reported for the base gpt2 model, while larger variants favor the original dataset on JGA.
- Trade-offs: IRWOZ 2.0 shows slightly worse perplexity and lower slot accuracy for some larger models than IRWOZ.For gpt2-medium and gpt2-large, slot accuracy is 0.956 and 0.960 versus 0.979 and 0.983 for IRWOZ.
- Overall assessment: The reported BLEU and JGA gains are treated as outweighing minor perplexity trade-offs for dialogue quality and goal achievement.The comparison emphasizes language generation and goal accuracy as the more indicative measures in this evaluation.
6 Human Evaluation
Human evaluation of sampled dialogues reports strong accuracy, domain knowledge, engagement, and human-like interaction scores, while identifying room for richer technical detail and language variation.
- Evaluation setup: 23 raters evaluated 40 randomly sampled dialogues from each industrial domain using 1-to-10 scores.Participants included shop floor workers, robotics engineers, computational linguists, and researchers.
- Accuracy: Accuracy scores reach 8.13 for Q1 and 8.39 for Q2, indicating strong contextual understanding of user intents.The evaluation links these scores to coherent task-oriented interactions.
- Knowledge: Knowledge scores are 8.48 for Q1 and 7.39 for Q2, showing industrial-domain expertise alongside room for better technical-detail integration.The lower Q2 score is associated with task-execution phases.
- Engagement: Engagement scores of 8.04 and 8.30 suggest effective turn-taking but challenges in complex multi-step interaction flows.The reported limitation concerns maintaining dynamic interaction during complex operations.
- Human-likeness: Human-like evaluation reaches 8.70 for politeness, while language variation scores 7.74 and leaves room for more diverse expressions.The highest single-question score is attributed to politeness maintenance.
7 Conclusions & Future Work
The paper presents IRWOZ 2.0 as a broader and more accurate industrial HRI dialogue dataset, with benchmark improvements and a public non-commercial release.
- Conclusion: IRWOZ 2.0 combines LLM-based generation with manual and automated correction across Assembly, Delivery, Position, and Relocation.The conclusion characterizes the resulting dataset as more comprehensive and accurate.
- Conclusion: Benchmarking reports improvements in BLEU scores, Joint Goal Accuracy, and other key metrics across model architectures.The conclusion summarizes the benchmark comparison with the predecessor dataset.
- Future work: Future work may expand industrial domains, integrate multimodal data, and explore transfer learning across industrial scenarios.These directions are presented as proposed future work rather than completed contributions.
- Dataset release: The IRWOZ 2.0 dataset is available for non-commercial use through IEEE DataPort.The paper also identifies the original IRWOZ repository separately.
Appendix A Prompt for correction
The appendix presents the prompt used to remove typos from the IRWOZ dataset.
- Prompt for correction: Figure A1 shows the defined prompt for removing typos from the IRWOZ dataset.
Appendix B Dialogue turn distribution of each domain
Appendix B presents the generated-dialogue turn distribution across four domains.
- Figure B2 shows the turn distribution of generated dialogues for four domains.
- Appendix B uses Figure B2 to summarize dialogue-turn patterns across the domains.
- The distribution is presented as a domain-by-domain comparison.
Appendix C Generated Dialogue Samples
Appendix C provides generated dialogue samples for the Assembly task.
- Figure B3 shows two generated dialogue samples for the Assembly task.
Appendix D Human Evaluation - Questions and Results
Appendix D presents randomly selected LLM-generated dialogue examples from four industrial domains and describes a human evaluation of dialogue quality.
- Figure B3 provides two generated Assembly-task dialogue samples for comparison of LLM-generated outputs.
- Table B1 reports human evaluation results for dialogue quality across four categories with two questions each.
- The human-evaluation table counts evaluator grades from G3 through G10, with zero responses marked by an en dash.
- Figures D4 and D5 show randomly selected LLM-generated dialogues from Assembly, Delivery, Position, and Relocation.