Source-linked AI summary
Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions
Jun Hatori, Yuta Kikuchi, Sosuke Kobayashi, Kuniyuki Takahashi, Yuta Tsuboi, Yuya Unno, Wilson Ko, Jethro Tan
TL;DR
Robots must interpret spoken instructions despite diverse expressions and ambiguity. The paper integrates object detection, language understanding, and interactive clarification, achieving effective instruction following in simulation and physical-robot experiments.
Problem
Robots must interpret spoken instructions despite diverse expressions and inherent ambiguity in human intent.
Method
The system integrates deep-learning object detection, natural-language processing, and dialogue-based clarification for unconstrained spoken instructions.
Results
73.1% end-to-end success rate was achieved for picking and placing with external operators in the physical robot experiment.
Takeaways & Limitations
Interactive clarification enables the robot to resolve ambiguous instructions by asking operators for additional explanations.
Abstract
from arXiv · showhide
Comprehension of spoken natural language is an essential component for robots to communicate with human effectively. However, handling unconstrained spoken instructions is challenging due to (1) complex structures including a wide variety of expressions used in spoken language and (2) inherent ambiguity in interpretation of human instructions. In this paper, we propose the first comprehensive system that can handle unconstrained spoken language and is able to effectively resolve ambiguity in spoken instructions. Specifically, we integrate deep-learning-based object detection together with natural language processing technologies to handle unconstrained spoken instructions, and propose a method for robots to resolve instruction ambiguity through dialogue. Through our experiments on both a simulated environment as well as a physical industrial robot arm, we demonstrate the ability of our system to understand natural instructions from human operators effectively, and how higher success rates of the object picking task can be achieved through an interactive clarification process.
I. INTRODUCTION
The paper addresses unconstrained spoken instructions, which combine diverse expressions with ambiguity, by integrating language understanding, object detection, and interactive clarification. It introduces a comprehensive dialogue-based system and a dataset of miscellaneous real-world objects annotated with spoken instructions.
- Motivation: Spoken instructions are challenging because they lack predefined structures and limited vocabularies, and may ambiguously refer to multiple objects.Examples include informal descriptions such as “brown fluffy thing” when several fluffy objects are present.
- Contribution: The system combines object detection and referring-expression models to map diverse spoken expressions to objects without explicit object-class information.The design supports zero-shot recognition of unseen objects.
- Contribution: Interactive clarification lets the robot ask questions and use verbal and visual feedback to narrow the operator’s intended object.The robot highlights candidate objects while the operator provides additional explanations.
- Novelty: The paper presents the first comprehensive system combining interactive clarification with unconstrained spoken instructions in human–robot dialogue.The authors position this integration as a novel contribution relative to prior robotics work.
- Evaluation resources: A new challenging dataset contains miscellaneous real-world objects annotated with spoken language instructions.The dataset is intended to support evaluation of varied spoken references in realistic settings.
II. RELATED WORK
Prior work studied natural-language interaction, spatial and cardinality expressions, multimodal understanding, and clarification, but clarification research used simpler controlled settings. This paper builds on those directions for spoken object picking in more realistic environments.
- Natural-language robotics: Robotics studies have examined abstract spatial concepts, cardinality, ordinality, and recognition of multiple daily objects.These works show that positional relationships help distinguish among multiple objects.
- Clarification: Prior clarification work addressed object fetching with natural language and pointing, but used at most six known, labeled objects in a simple controlled environment.Its robot feedback was limited to binary confirmation such as “This one?”.
- Multimodal foundations: Related multimodal research includes image captioning, visual-semantic mapping, image retrieval, dense object or relation descriptions, and referring-expression tasks.These areas provide foundations for linking language with visual regions.
- Related tasks: Visual question answering answers open-ended questions about images, while object detection identifies objects within images.This paper applies object detection to objects described by spoken expressions in human–robot interaction.
III. TASK DEFINITION
The task is to understand spoken instructions that direct a robot to pick an object and move it to a destination in realistic, varied environments. The setup includes clutter, many known objects, unseen objects, and unconstrained language feedback.
- Task setting: The robot operates among over 100 distributed objects and must pick a specified object and move it to another box.An example instruction is “Move the tissue box to the top right bin.”
- Interactive clarification: When multiple objects match an instruction, the robot asks which one to pick and accepts unconstrained feedback about color, position, or relations.Example feedback includes “The orange one” and “The one next to the plastic bottle.”
- Realistic challenges: The environments include organized and highly cluttered layouts with occlusion, similar objects, and references based on absolute or relative position, ordinality, or cardinality.Both clutter and organization create distinct object-recognition challenges.
- Object variety: The setup uses over a hundred known objects and 22 unknown objects to evaluate generalization to new objects.Unnamed objects may require indirect or abstract expressions from the operator.
- System task decomposition: The system is statistical rather than rule-based, jointly learning language understanding and object recognition for attributes such as color, texture, size, and orientation.Instructions may enter as text or speech and are decomposed into subtasks including transcription, detection, object selection, and destination selection.
A. Candidate Object Detection
Candidate object detection uses an SSD-based detector to propose bounding boxes, while the target-selection pipeline combines visual and language features and modifies relational reasoning to support unseen objects.
- Candidate proposal: SSD scores many cropped image regions and outputs candidate-object bounding boxes after filtering low-scoring regions.The detector is modified to classify candidate regions without relying on predefined object classes.
- Feature flow: The system feeds image and text features to target object detection, but feeds only text features to destination-box selection.This separates visual target identification from instruction-based destination prediction.
- Model extensions: The object comprehension module is based on a referring-expression listener model and is modified for zero-shot recognition of unseen objects.The destination box module uses the same neural-network architecture described for the broader system.
- Target-object pipeline: The target object module selects the best bounding box from predicted candidates using the instruction, the full image, cropped object images, and geometric features.CNN and MLP components encode visual and spatial information for candidate comparison.
- Generalization: Relational features compare each candidate with all other objects rather than only objects in the same class.This modification supports recognition of seen and unseen objects and removes the need to annotate object classes.
C. Destination Box Selection
The destination box selection module uses the same neural-network architecture as target-object recognition, processing the instruction without requiring tokenization or chunking. Ambiguity is handled by comparing scores and asking for clarification when competing objects or boxes are too close.
- C. Destination Box Selection: Destination box selection uses the same neural-network architecture as target-object recognition, despite being treated as a separate task.The architecture includes word embedding, LSTM, and MLP layers.
- C. Destination Box Selection: The model processes the instruction without tokenization or chunking, learning which sentence parts to emphasize through its LSTM-based representation.
- C. Destination Box Selection: The system asks for clarification when multiple objects or boxes have scores within the corresponding ambiguity margins.Potential targets are highlighted so the operator can provide additional explanation.
- C. Destination Box Selection: Margin thresholds are calibrated carefully because overly large margins cause persistent confirmation, while overly small margins permit speculative actions under ambiguity.The proposed margin is linked to training the comprehension model with a max-margin objective.
- C. Destination Box Selection: Additional sentences are incorporated by summing the object scores calculated for each sentence.The paper reports that this approach is effective in experiments.
V. EXPERIMENT SETUP
The experiments use PFN-PIC, a cluttered real-world object dataset annotated with unconstrained instructions. Its design includes unfamiliar and repeated household objects, while validation annotations were manually reviewed for appropriateness.
- V. EXPERIMENT SETUP: PFN-PIC contains 1,180 images with bounding boxes and annotated text instructions.The dataset was created for the picking-instruction task.
- V. EXPERIMENT SETUP: Each image contains 20 objects on average distributed across four boxes in average or highly cluttered environments.Each box measures 400mm × 405mm.
- V. EXPERIMENT SETUP: The dataset includes unfamiliar commodities and household items that encourage indirect references to position and attributes such as color or texture.
- V. EXPERIMENT SETUP: Repeated objects prevent operators from identifying targets by object name alone, requiring absolute or relative positional references.
- V. EXPERIMENT SETUP: At least three crowd workers annotated each object with colloquial instructions, while some interpretation ambiguities were deliberately retained for clarification experiments.Annotators were asked to make each instruction uniquely identifying, but validation did not remove every ambiguity.
- V. EXPERIMENT SETUP: The dataset contains 25,900 objects and 91,590 text instructions, split into training and validation images.The training split has 1,160 images and 25,517 objects; the validation split has 20 images and 383 objects.
B. Details of Machine Learning Setup
The machine-learning setup trains separate detection and comprehension components using pretrained visual backbones, stochastic optimization, and regularization choices selected through validation-data trial and error.
- B. Details of Machine Learning Setup: All hyperparameters in the machine-learning setup were selected through trial and error on validation data.
- B. Details of Machine Learning Setup: The candidate object detector uses a VGG16 feature extractor pretrained on ImageNet and is trained with SGD and momentum for 60,000 iterations.Training also applies vertical image-flip augmentation.
- B. Details of Machine Learning Setup: The object comprehension model uses a 50-layer ResNet pretrained on ImageNet and minimizes a max-margin loss with margin m = 0.1.It is optimized with Adam for 120,000 iterations using randomly sampled incorrect pairs.
- B. Details of Machine Learning Setup: The language model replaces unseen words with UNK and applies word dropout plus stochastic removal of the latter half of sentences during training.The word-dropout ratio is 0.1, and latter-half sentence dropout uses a ratio of 0.05.
C. Robotic System Setup
The physical system combines an industrial robot arm, vacuum gripper, pressure sensing, stereo and RGB cameras, and software for motion and manipulation planning. Validation also reports distinct metrics for detection, destination selection, and object selection.
- C. Robotic System Setup: The robotic system uses a FANUC M10iA industrial arm equipped with a vacuum gripper.
- C. Robotic System Setup: Table I evaluates candidate object detection with average precision, destination selection with top-1 accuracy, and object selection with top-k accuracies.
- C. Robotic System Setup: A pressure sensor validates whether grasping succeeded, while stereo and RGB cameras observe the workspace and retrieve point clouds.
- C. Robotic System Setup: The software stack includes an RRT-based motion planner and a manipulation planner that queries inverse kinematics to construct pick-and-place paths.
A. Software Simulation Results
The simulation evaluates object-selection modules under colloquial spoken instructions and tests ambiguity resolution through interactive clarification. Clarification substantially improves selection accuracy, while errors concentrate in ambiguous or visually similar cases.
- Module performance: Candidate object detection and destination box selection exceeded 95% precision and accuracy, while target object selection reached 88% top-1 accuracy.The target-selection result was obtained with naturally colloquial instructions and an average of 20 objects per instance.
- Interactive clarification: The clarification experiment uses another instruction for the same image as the robot’s simulated clarification response.The selected response minimizes word overlap with the original instruction, assuming human operators would rephrase and narrow the target.
- Ambiguity analysis: Target selection accuracy was 94.9% for unambiguous instructions versus 63.6% for the top-ranked object when the system judged instructions ambiguous.Even in ambiguous cases, candidate detection identified the correct target as a potential candidate in 90.9% of cases.
- Error analysis: The system handled challenging referring expressions and clutter but was often confused by similarly shaped or colored objects; clarification resolved color and location ambiguities in analyzed examples.
C. Physical Robot Experiment Results
Physical-robot experiments evaluated unconstrained spoken instructions from seven external operators across 97 object-picking trials. The system achieved measurable end-to-end success, while performance was limited by environment differences and vacuum-grasping failures.
- Experimental setup: Seven external operators performed 97 physical-robot object-picking trials, including 34 trials with approximately 30% unknown objects.The operators used colloquial expressions without prescribed instruction patterns, and the robot could request clarification up to twice.
- Results: 75.3% target object selection accuracy and 73.1% end-to-end picking-and-placing success were achieved with external operators.The reported metrics include both object selection and complete task success in the physical-robot setting.
- Results: With approximately 30% unknown objects, target object selection reached 70.6% accuracy and end-to-end picking-and-placing success reached 67.6%.The results indicate performance remained measurable when unknown objects were mixed into the environment.
- Limitations: The 75.3% physical target-selection accuracy was below the 88.0% software-simulation accuracy, primarily because camera position, color temperature, and lighting differed.Color-temperature differences caused perceived colors such as orange to mismatch system or dataset labels such as red.
- Limitations: Vacuum grasping remained problematic for some objects: sponges were unreliable, and strong suction sometimes dented empty beverage cans.Although picking and placing alone reached 97.3%, these grasping cases were counted as failures.
VII. CONCLUSION
The paper presents a robotic system for unconstrained spoken instructions that uses interactive dialogue to clarify ambiguous object references. In realistic cluttered environments, it achieved 73.1% end-to-end picking accuracy and reduced target-selection error by 39%.
- Conclusion: The proposed system handles unconstrained spoken instructions and clarifies human intent through interactive dialogue.It integrates object-recognition and referring-expression comprehension technologies, with modifications for previously unseen objects.
- Conclusion: In a challenging realistic environment with randomly scattered miscellaneous objects, the industrial robot achieved 73.1% end-to-end picking accuracy.The conclusion presents this as the system's overall physical-robot outcome.
- Conclusion: Interactive clarification reduced target object selection error by 39%.The reported reduction concerns disambiguation of the human operator’s intention.
- Future work: Future work will extend the architecture to multiple languages and improve grasping with a combined vacuum-and-gripper end-effector.The planned multilingual extension aims to share knowledge across languages, while the grasping change targets picking accuracy.