Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Corrections

• Prior LLM-based methods adapt plans or low-level rewards from language feedback, but often handle corrections rigidly rather than arbitrary forms of human guidance.Source: §I · p. 1
• In the scissors example, a planning correction makes the robot open the drawer before grasping, while later low-level commands correct the grasping pose.Source: §I · p. 1, Figure 1 · p. 2
• DROC addresses planning and execution failures by interpreting online corrections, distilling reusable knowledge, and retrieving it for future tasks.The system uses an LLM to infer how to respond, what to remember, and what to retrieve.Source: §I · p. 1
• DROC improves adaptation to new objects and configurations while reducing the number of human corrections needed over time.The introduction reports higher success rates and fewer corrections than Code as Policies and simpler distillation or retrieval variants.Source: §I · p. 1

LLM-powered Robotics

• Prior systems use LLMs for task planning, code generation, reward design, and fine-grained feedback, but existing work does not handle both planning-level and low-level manipulation corrections.Source: §II · p. 2

Knowledge Base for LLMs

• Existing LLM-based robot policies focus on immediate correction responses rather than distilling knowledge for long-term generalization from corrections.Source: §II · p. 2
• DROC combines summarizing multiple feedback rounds with autonomously partitioning corrections into high-level and low-level reusable adjustments.Source: §II · p. 2

Learning from Language Corrections

• Unlike offline language-correction methods or approaches requiring restrictive grounding, demonstrations, or extensive corrections, DROC responds to online corrections without training data for interpreting them.Source: §II · p. 2

Problem Formulation

• DROC decomposes manipulation into a task planner T that maps instructions to skills and a skill composer S that maps each skill to a code policy.Humans can issue corrections while skills execute, and the system repeats correction and execution until the skill or plan is fixed.Source: DISTILLATION AND RETRIEVAL OF ONLINE… · pp. 3–4
• The framework measures learning through the amortized number of corrections across tasks and aims to reduce this quantity over time.Source: DISTILLATION AND RETRIEVAL OF ONLINE… · p. 3

Correction Handling

• The correction handler C classifies feedback as high-level or low-level, then triggers replanning or code generation with relevant interaction history.Low-level corrections use selected history rather than the entire log to avoid burying information needed for reasoning.Source: DISTILLATION AND RETRIEVAL OF ONLINE… · pp. 3–4
• DROC grounds fine-grained corrections in object-centric frames and scales vague movement distances to the related object’s size.Geometric properties such as surface normals help represent the object frame.Source: DISTILLATION AND RETRIEVAL OF ONLINE… · p. 4

Knowledge Distillation

• The extractor E summarizes interaction histories into reusable task knowledge, variables, modified plans or code, and updated object states stored in a dictionary knowledge base.Plan-level knowledge includes constraints and preferences, while skill-level knowledge includes parameters such as grasp poses and pull distances.Source: DISTILLATION AND RETRIEVAL OF ONLINE… · p. 4

Knowledge Retrieval

• The retriever R selects semantically related tasks and then uses visual similarity to retrieve object-specific knowledge when language alone is insufficient.DINO-V2 features support visual-visual retrieval, while CLIP supports visual-semantic retrieval; drawer-handle shape can determine the appropriate grasp.Source: DISTILLATION AND RETRIEVAL OF ONLINE… · pp. 4–5, Figure 2 · p. 4

Skill-Level Corrections

• DROC outperforms baselines on skill-level corrections, requiring fewer interventions while supporting generalization to unseen task settings.Skill-level results average six experimental rounds, with each iteration using a different task setting.Source: §IV · p. 5, Figure 3 · p. 6
• DROC uses fewer than half as many corrections as Code as Policies in the first round, indicating more effective responses to online feedback.The comparison uses the same prompts and function APIs across baselines.Source: §IV · p. 5
• Distilled knowledge reduces corrections for variations of the same task, while cross-object history transfer further reduces online corrections.Transfer reuses histories from other objects and from a related drawer-placement task.Source: §IV · p. 5
• Visual retrieval helps DROC select the appropriate policy when cup orientation, rather than cup color, determines how the cup must be handled.The Hang Cup on Rack ablation contrasts knowledge sets with different visual features.Source: §IV · p. 5

Plan-Level Corrections

• Illustrative plan-level cases show DROC replanning after identifying a plan error, distilling constraints, and retrieving experiences through semantic and visual similarity.The examples cover both correction-driven replanning and ambiguity in task planning.Source: Figure 4 · p. 6, §IV · p. 6
• For plan-level test tasks, DROC requires fewer corrections than naive retrieval, showing that distillation and retrieval support generalization across task scenarios.The evaluation reports average test-task corrections over three trials and omits train-task differences because both systems share the correction module.Source: §IV · p. 6, Table II · p. 6

• DROC enables robots to respond to arbitrary online language corrections, distill generalizable information, and retrieve relevant knowledge for novel tasks.Source: §V · p. 6
• DROC outperforms baselines for both low-level and high-level corrections and generalizes knowledge within the same tasks and across similar tasks.Source: §V · p. 6