Source-linked AI summary
Learning to Assemble Novel Structures with Unfamiliar Parts under Semantic Constraints
Jonghyuk Park, Alex Lascarides, Subramanian Ramamoorthy
TL;DR
Agents assembling novel structures may encounter semantic rules after deployment, without initially knowing the relevant concepts. This paper presents a neurosymbolic agent that learns from embodied language and demonstrations, finding that generic teacher-provided constraints improve online assembly learning beyond labels and correction memories.
Problem
Assembly demonstrations can leave semantic constraints ambiguous, especially when agents initially lack the relevant part and structure concepts needed to identify valid assemblies.
Method
A neurosymbolic agent combines visual grounding with symbolic memory and planning, updating concepts and constraints from embodied natural-language interaction and task demonstrations.
Results
Generic teacher-provided constraints consistently reduce cumulative regret beyond part labels and non-generic correction memories, without improving visual grounding.
Takeaways & Limitations
Reusable symbolic constraints communicated during deployment improve online semantic assembly learning in this simulated toy-truck benchmark through planning rather than visual grounding alone.
Takeaways & Limitations
The case-memory mechanism does not abstract over corrections or discard irrelevant literals, leaving general constraint induction for future work.
Abstract
from arXiv · showhide
This paper describes a neurosymbolic architecture for learning to assemble novel structures using evidence from embodied conversations and task demonstrations. We focus on scenarios where an agent encounters, after deployment, semantic constraints on structures--in other words, constraints as to which part types and features make valid structures--that were not available during training, and where it is initially unaware of the relevant structure and component part concepts. The agent must acquire and exploit such knowledge through user interactions while attempting assembly. We study this setting in a simulated toy truck assembly domain, learning from symbolic evidence encoded in natural language and from dense visual observations. Our experiments show that communicating semantic constraints through natural language (e.g., "dump trucks have a dumper") yields more data-efficient online adaptation than relying only on task demonstrations and/or only naming the parts through natural language.
1. Introduction
The paper addresses robotic assembly when semantic constraints are unavailable before deployment and demonstrations cannot adequately convey them. It proposes a neurosymbolic agent that adapts online from embodied natural-language interaction and task demonstrations, showing improved data efficiency when semantic constraints are communicated through natural-language generics.
- Motivation: Robotic assembly knowledge includes part geometry and joins, assembly orders, and low-level control skills, which are largely physical constraints suited to task demonstrations.The paper contrasts these established knowledge categories with semantic constraints introduced during deployment.
- Motivation: Semantic constraints can make a geometrically feasible join produce an invalid structure, such as requiring coloured dump-truck parts not to be yellow.Demonstrations alone are insufficient as a corrective signal because multiple rules may be consistent with the observed correction.
- Approach: The proposed neurosymbolic architecture supports online adaptation from embodied natural-language interaction and task demonstrations as domain knowledge arrives piecemeal during deployment.It separates dense perceptual grounding from explicit symbolic memory and planning, with visual classifiers updated through new exemplars.
- Evaluation: In a simulated toy-truck assembly domain, the study compares generic natural-language semantic constraints with baselines receiving demonstrations, part labels, and/or non-generic corrections.Truck types are defined by constraints over part subtypes and attributes; an example generic is “dump trucks have a dumper”.
- Evaluation: Communicating semantic constraints through natural-language generics improves data-efficient adaptation compared with relying only on task demonstrations and/or naming parts through natural language.The comparison is conducted in the simulated toy-truck assembly domain.
2. Semantic Assembly: Task Setting and Domain
The semantic assembly setting models parts, subassemblies, structural relations, and semantic constraints, then requires an agent to assemble a goal structure from valid parts while avoiding semantically violating distractors. The agent begins without domain-specific assembly knowledge and acquires it online through visual observations, demonstrations, and controlled natural-language interaction.
- Domain representation: A semantic assembly domain defines atomic part concepts Π, subassembly concepts Σ, and a domain theory Ω of subtype and has-a relations.Atomic parts are unary predicates over primitive objects, while subassemblies are unary predicates over composite structures; relations encode subclass and containment structure.
- Task setting: Each task provides required parts C, geometrically feasible distractors D, and an assembly goal γ ∈Σ.The objective is to construct γ using only C, because D can violate semantic constraints despite being physically usable.
- Task specifications: Valid instances of γ must satisfy both its topology graph Gγ, specifying required joins and poses, and its quantified constraints Sγ.For example, Sγ can encode that every dump truck has a dumper.
- Task execution: The agent plans and executes assembly from a single-view RGB image while engaging in supervised natural-language dialogue with a teacher.The experiments use a simulated toy-truck domain with oracle object masks, parametrized demonstrations, and an oracle controller to isolate deployment-time semantic acquisition.
- Deployment-time acquisition: At deployment, the agent lacks the concept sets Π and Σ, domain theory Ω, perceptual features, topology graphs Gγ, and semantic constraints Sγ, so it must acquire domain knowledge online.Domain-neutral primitive-action interfaces and symbolic planning remain available, while demonstrations and controlled dialogue provide complementary evidence.
3. The Neurosymbolic Agent Architecture
The architecture combines perception, dialogue, symbolic reasoning, actuation, and long-term memory in an interactive episode loop. It incrementally updates visual, symbolic, and lexical knowledge, then selects constraint-satisfying structures and plans feasible joins.
- Episode control loop: Algorithm 1 repeatedly grounds scene images, updates knowledge through dialogue or demonstrations when concepts are unknown, selects goals, and continues until teacher acceptance.The loop uses a goal utterance, scene image, visual exemplar base, and symbolic knowledge base as inputs.
- Perception and interaction: Vision converts a single-view RGB scene into candidate atomic part concepts with confidence scores and 3D geometry for join relations.Recognition confidence is propagated to the symbolic layer for downstream planning and learning.
- Perception and interaction: Dialogue management parses goals, clarifications, corrections, and generic semantic constraints, translating first-order statements into ASP clauses for planning.The protocol represents both universal constraints, such as all fire-truck fenders being red, and existential constraints, such as missile trucks having a green fender.
- Learning and memory: Long-term memory stores visual exemplars, symbolic domain and assembly knowledge, semantic constraints, and lexicon associations updated from teacher definitions, demonstrations, and utterances.Teacher-labelled positive or negative exemplars induce updated binary classifiers for atomic part concepts.
- Symbolic reasoning: Symbolic reasoning decomposes assembly into ASP goal selection and join-sequence planning, optimizing recognition compatibility and constraint violations before finding collision-free joins.Goal structures instantiate topology graphs, while motion-planner checks discard and remember infeasible joins.
4. Interactive Learning Procedures
Section 4 compares four interaction strategies that share the same core architecture but differ in how teacher feedback updates memory. Label-aware strategies support targeted visual-grounding updates, while the full approach additionally translates generic teacher statements into symbolic semantic constraints.
- Interaction strategies: Four strategies differ in the teacher feedback they can use to update memory while sharing the same architecture, task abstraction, visual backbone, planner, and oracle execution assumptions.The strategies are NoLabels+CaseMemory, LabelsOnly, Labels+CaseMemory, and Labels+TeacherRules.
- Interaction strategies: The three label-aware strategies acquire labelled part-concept exemplars through dialogue and narrated demonstrations, enabling targeted updates to visual grounding.NoLabels+CaseMemory instead relies on indirect signals such as completed-assembly analysis and teacher interruptions, which introduce additional noise.
- Semantic constraint learning: Labels+TeacherRules translates violated generic teacher statements into FOL and ASP clauses added to Sγ, whereas LabelsOnly performs no semantic-memory update.This is the full approach: it combines shared concept labels with teacher-provided generic statements encoding semantic constraints.
- Correction memory: The +CaseMemory strategies store most-specific, non-generic correction cases that discourage an attempted choice when a preferred unused part is available.These corrections are triggered by teacher interventions such as “Use this1 instead of this2” and do not interpret generic constraint statements.
- Limitations: The case-memory mechanism does not abstract over corrections or discard irrelevant literals, leaving induction of general constraints via ILP or anti-unification as future work.This limitation applies to the stored correction cases rather than to the generic-rule translation mechanism.
5. Experiments
Experiments evaluate three label-aware interaction strategies across 30 simulated toy-truck datasets, where novel truck and part subtypes are introduced online after warm-up. Label-aware strategies outperform the no-label diagnostic in regret and grounding, while teacher rules reduce semantically invalid assemblies without improving visual grounding.
- Experimental setup: Experiments use 30 datasets of 40 online assembly episodes, with five warm-up problems followed by unforeseen subtypes and distractors during task execution.The domain contains 22 atomic part types and 6 truck types, with no separate training and testing splits.
- Evaluation metrics: Cumulative regret counts uniformly weighted structural, pose, distractor-use, grounding, and planner-timeout errors across 40 planning problems.Grounding failures are addressed through “Is there a X?” questions.
- Regret results: NoLabels+CaseMemory incurs substantially higher regret than label-aware strategies, while LabelsOnly and Labels+CaseMemory have similar curves without a reliable case-memory advantage.The no-label strategy struggles to learn novel part concepts without a shared vocabulary.
- Grounding and constraint use: The three label-aware strategies achieve comparable visual-grounding F1, but Labels+TeacherRules lowers regret by using generic constraints to prevent semantically invalid assemblies.Its improvement is therefore attributed to symbolic-planner use of constraints rather than better part classification.
6. Related Work
The work builds on concept-based interactive learning and VLM-based neurosymbolic systems that connect human feedback, perception, language, and symbolic reasoning. It differs by addressing agents initially unaware of relevant structural concepts and by studying generic semantic constraints communicated as verified symbolic knowledge during task execution.
- Concept-Based Interactive Learning: Concept-based interactive learning uses concept-level corrections, prototype interactions, and self-explanations to revise neural or neurosymbolic models.These approaches typically revise models with a largely fixed task vocabulary.
- Concept-Based Interactive Learning: Unlike prior concept-based approaches, the agent overcomes unawareness of part predicates, subassembly concepts, topology, and semantic constraints.
- VLM-Based Neurosymbolic Systems: Recent VLM-based neurosymbolic systems map perception and language into symbolic structures, while human-guided systems integrate communication, learning, reasoning, and planning.The paper studies a complementary regime in which a human teacher communicates generic semantic constraints as verified symbolic knowledge during task execution.
7. Conclusion and Future Directions · Appendix A. ASP Encodings of Planning Subproblems · Appendix B. Detailed Account of Agent-Teacher Interaction Flow
The paper presents deployment-time neurosymbolic learning through controlled interaction and demonstrations, with ASP encodings for planning and a detailed teacher–learner protocol for acquiring missing knowledge during assembly.
- 7. Conclusion and Future Directions: The neurosymbolic agent adapts during task execution by updating visual exemplars, symbolic memory, and plans despite initially lacking relevant assembly concepts and constraints.The study focuses on deployment-time adaptation from controlled natural-language interaction and demonstrations.
- 7. Conclusion and Future Directions: Teacher-provided generic constraints improve online assembly learning in the studied benchma
- Appendix A. ASP Encodings of Planning Subproblems: The ASP encoding in Figures 4–6 represents the goal-selection planning subproblem and assumes a multi-shot solver such as Clingo.
- Appendix A. ASP Encodings of Planning Subproblems: The ASP encoding in Figures 7–8 represents join-sequence planning and additionally assumes ASP-modulo-theory solving, again like Clingo.
- Appendix B. Detailed Account of Agent-Teacher Interaction Flow: The interaction protocol expands the compact description in §2.4 into the complete set of teacher–learner branches used in the experiments.
- Appendix B. Detailed Account of Agent-Teacher Interaction Flow: Each task starts with a teacher-specified goal; demonstrations with natural-language narration teach new geometric relations or join poses, while verbal definitions suffice for known intermediary concepts.If the agent is unaware of the target, the teacher provides either a full demonstration or a verbal definition depending on context.
- Appendix B. Detailed Account of Agent-Teacher Interaction Flow: When visual recognition fails for a required atomic part, the agent requests clarification and the teacher points to the correct object.The part is guaranteed to exist in C.
- Appendix B. Detailed Account of Agent-Teacher Interaction Flow: After execution errors, the teacher elicits the intended action and corrects either a grounding error or a semantic-constraint violation.The agent’s explanation exposes its current beliefs and helps identify the relevant knowledge gap.
Appendix C. Vision Processing Module: Implementation details
The vision processing module converts RGB observations into structured representations through object detection, point cloud extraction, and pose estimation. Its implementation uses learned part-concept classifiers, COLMAP-based reconstruction, and ground-truth poses in simulation.
- Vision processing pipeline: The module parses RGB sensor inputs into structured representations for downstream processing through three visual subtasks: object detection, point cloud extraction, and pose estimation.Object detection includes predictions of object types.
- Object detection and classification: Object detection uses binary segmentation masks to localise instances and prioritises learning object classes because classification supports identifying novel parts.The implementation uses lightweight binary classifiers for currently known atomic part concepts, trained from positive and negative exemplars in long-term memory and teacher interactions.
- Point cloud extraction: Each newly encountered atomic part concept is represented by a point cloud extracted from multi-view images with known poses, using COLMAP for reconstruction.Per-pixel embeddings are computed at lower resolution and used as point descriptors for photogrammetric reconstruction.
- Pose estimation: 6D object pose estimation supports accurate joining at specified relative poses, but experiments use ground-truth poses from the simulated environment after each manipulation action.The passage notes recent progress in few-shot and zero-shot RGB pose estimation using vision foundation models.
Appendix D. Per-strategy Details on Learning from Visual Data … Appendix E. Full Description of Toy Truck Assembly Domain
Appendix D details how agents learn visual grounding and 3D representations under different interaction strategies, while Appendix E specifies the toy-truck domain’s concepts, topology, and semantic constraints. Label-aware agents acquire visual exemplars directly from language, whereas NoLabels+CaseMemory relies on indirect signals and must infer unfamiliar parts from demonstrations.
- D.1. Learning Visual Grounding of Parts: Visual grounding performance is determined by binary classifiers induced from positive and negative exemplar sets.The learning process depends on acquiring labelled concept exemplars through teacher interactions.
- D.1. Learning Visual Grounding of Parts: Label-aware agents obtain exemplar labels from natural-language statements triggered by grounding mistakes, failed searches, or full demonstrations.Examples include “This is a dumper” and “This is not a quad cabin.”
- D.1. Learning Visual Grounding of Parts: NoLabels+CaseMemory agents update exemplar sets indirectly through episode analysis and teacher interruptions rather than shared-vocabulary concept labels.Graph matching supplies positive exemplars, while interrupted invalid joins provide pairwise negative labels; both acquisition processes are susceptible to noise.
- D.2. Learning 3D Geometries of Novel Parts and Joins: Across four interaction strategies, learning point-cloud representations of atomic parts and joining poses uses the same process, but strategies differ in when extraction is triggered.NoLabels+CaseMemory relies on its own judgments to identify novel atomic part types.
- D.2. Learning 3D Geometries of Novel Parts and Joins: NoLabels+CaseMemory may mistake familiar parts for novel ones or miss unfamiliar parts, causing downstream task-execution errors.Novel types are registered when demonstrated objects receive low likelihood scores across currently known types.
- Appendix E. Full Description of Toy Truck Assembly Domain: The simulated toy-truck domain contains 22 atomic part types, their supertypes, and subassembly concepts organized through hypernymy and hyponymy relations.Subassemblies include goal concepts and semantically meaningful intermediary structures.
- Appendix E. Full Description of Toy Truck Assembly Domain: Assembly topology graphs describe how subassemblies compose from atomic parts or other subassemblies, while subtype graphs inherit from the truck graph and add semantic constraints.The domain theory also includes holonymy and meronymy relations, and the constraint table lists constraints that correction memories may not provide.
Appendix F. Breakdown of Cumulative Regrets
Appendix F breaks down the averaged cumulative regret curves by five agent error types. Labels+TeacherRules reduce distractor-usage mistakes over time, while NoLabels+CaseMemory’s apparent post-warm-up increases reflect its need for full demonstrations on novel truck types and averaging across randomized introductions.
- Breakdown of cumulative regrets: Labels+TeacherRules agents make fewer distractor-usage mistakes as learning proceeds.
- Breakdown of cumulative regrets: NoLabels+CaseMemory curves seemingly increase after warm-up because novel truck types require full demonstrations, unlike label-aware agents for which verbal definitions suffice.These demonstrations do not count towards any error type.
- Breakdown of cumulative regrets: After sufficient episodes, NoLabels+CaseMemory agents begin showing learning progress again for most error types.Novel truck types are introduced at random points in each dataset, so averaging across datasets can create a false impression of increasing errors.
- Breakdown of cumulative regrets: Figure 13 breaks down the averaged cumulative regret curves into five agent error types.The breakdown corresponds to the cumulative regret curves reported in the main paper.
Appendix G. Auxiliary Metric: Unique Motion Planner Calls
Unique motion-planner calls measure collision-free joining-path checks within ASP planning. Label-aware strategies reduce these calls relative to NoLabels+CaseMemory, with TeacherRules benefiting from early semantic filtering during goal selection.
- The auxiliary metric counts accumulated unique calls to the integrated motion planner for collision-free joining-path checks.These calls are part of the ASP planning procedure.
- NoLabels+CaseMemory makes substantially more motion-planner calls than label-aware strategies because poorer grounding produces more invalid object choices and replanning.Invalid choices can trigger additional replanning before assembly succeeds.
- Among label-aware strategies, Labels+TeacherRules makes fewer calls by rejecting or penalising semantically invalid object-role assignments before geometric collision checks.This filtering occurs in the goal-selection ASP subproblem and can eliminate distractor choices that remain geometrically feasible.