Source-linked AI summary
Robot Representation and Reasoning with Knowledge from Reinforcement Learning
Keting Lu, Shiqi Zhang, Peter Stone, Xiaoping Chen
TL;DR
KRR and RL each lack capabilities provided by the other: KRR is not designed to learn from experience, and RL is ill-equipped to use declarative knowledge. KRR-RL combines logical-probabilistic reasoning with model-based learning, and evaluations report improved navigation and delivery outcomes.
Problem
KRR paradigms are strong at representing and reasoning with declarative knowledge but are not designed to learn from experience, while RL agents learn through interaction but are ill-equipped to use declarative knowledge.
Source: §I · p. 1Method
Results
Navigation success averaged 83.8% for KRR-RL versus 26.8% for the baseline, and delivery evaluations reported higher reward and request fulfillment with lower question-asking cost when learning was enabled.
Source: §IV · pp. 5–6Takeaways & Limitations
The experiments suggest learned RL knowledge can be represented and used by KRR reasoning to dynamically generate planning models for robot tasks.
Source: §V · p. 6When a task arrives, the robot currently optimizes task completion without accounting for potential knowledge learned during that task.
Source: §V · p. 6
Abstract
from arXiv · showhide
Reinforcement learning (RL) agents aim at learning by interacting with an environment, and are not designed for representing or reasoning with declarative knowledge. Knowledge representation and reasoning (KRR) paradigms are strong in declarative KRR tasks, but are ill-equipped to learn from such experiences. In this work, we integrate logical-probabilistic KRR with model-based RL, enabling agents to simultaneously reason with declarative knowledge and learn from interaction experiences. The knowledge from humans and RL is unified and used for dynamically computing task-specific planning models under potentially new environments. Experiments were conducted using a mobile robot working on dialog, navigation, and delivery tasks. Results show significant improvements, in comparison to existing model-based RL methods.
I. INTRODUCTION
KRR represents declarative knowledge but is not designed to learn from task experience, while RL learns policies from interaction but is ill-equipped to use human-provided declarative knowledge. The paper integrates logical-probabilistic KRR with model-based RL so learned transition probabilities and human knowledge can support runtime planning.
I. INTRODUCTION
- KRR represents and reasons with declarative knowledge, but is not designed to learn from experience; RL learns policies through interaction but is ill-equipped to use declarative knowledge.The paper frames these limitations as complementary motivations for integration.Source: §I · p. 1
- P-log represents and reasons with both human knowledge and knowledge learned through RL.P-log is a logical-probabilistic KRR paradigm; the robot reasons at runtime to produce probabilistic transition systems for policy generation.Source: §I · p. 1
- Model-based RL supplies a world model that can update the robot’s declarative knowledge base and be combined with human knowledge.Unlike model-free RL, its output includes a world model usable by planning algorithms.Source: §I · p. 1
- KRR-RL learns action transition probabilities through model-based RL and incorporates them into probabilistic reasoning alongside human knowledge.The resulting reasoning supports dynamic construction of task-specific planning models at runtime.Source: §I · p. 1
II. RELATED WORK
Prior work has combined KRR and RL to use knowledge in learning, and has connected relational reasoning or logical-probabilistic reasoning to planning. The paper distinguishes its approach by tightly coupling logical-probabilistic KRR with model-based RL to reason over probabilities learned from interaction.
II. RELATED WORK
- Earlier KRR-RL work used logical action knowledge to guide exploration and improve learning, while this work directly reasons with probabilities learned from RL.The paper characterizes those methods’ KRR capabilities as limited to logical action knowledge.Source: §II · pp. 1–2
- Relational RL methods exploit state and action structure in current tasks, whereas this framework supports KRR factors beyond state and action representations, such as time in navigation,,, [26].The cited work includes relational temporal-difference learning and affordance relations that forbid specific actions.Source: §II · p. 2
- Related logical-probabilistic planning methods compute priors or world dynamics, but require supplied world models rather than learning them,, [20], [7].The paper contrasts this with its integration of model-based RL.Source: §II · p. 2
- The paper presents tightly coupled logical-probabilistic KRR and model-based RL as a first, with KRR representing and reasoning over knowledge from humans and RL.Other robot reasoning and learning architectures are described as relatively complex and functionally broad,,,.Source: §II · p. 2
III. INTEGRATED KRR-RL FRAMEWORK
The framework addresses the trade-off between modeling domain factors for completeness and keeping learning computationally feasible. It combines learned dynamics with human knowledge to construct task-oriented partial models for planning.
III. INTEGRATED KRR-RL FRAMEWORK
- When a task arrives, KRR dynamically constructs a partial world model that excludes unrelated factors, then planning algorithms compute a task-oriented controller.The knowledge base combines human knowledge about variable dependencies with probabilities learned through interaction.Source: §III · p. 2
- Including factors such as human walkers or sunlight can improve model completeness, but representing them in the state space can make learning computationally infeasible.The passage estimates that modeling ten human-occupancy cells makes the state space about 1,000 times larger.Source: §III · p. 2
A. A General Procedure
The general procedure separates learning world dynamics while the robot is free from task-time construction of a planning model. Learning is restricted to goal-oriented endogenous variables, while reasoning can account for exogenous variables that affect outcomes.
A. A General Procedure
- Endogenous variables are goal-oriented factors the robot actively changes, while exogenous variables affect action outcomes but are not changed by the robot; both sets depend on the task.For navigation, position is endogenous and current time is exogenous; RL learns in spaces specified by endogenous variables.Source: A. A General Procedure · p. 2
- While free, the robot selects tasks using human-provided guidance, learns with R-Max, and uses learned probabilities to update the KRR knowledge base.If learning is interrupted by a task arrival, the probabilities learned so far are used for the update.Source: A. A General Procedure · p. 3
- For a task-specific transition system, KRR reasons over endogenous variables while also using exogenous variables to compute transition probabilities.The resulting controllers omit exogenous variables directly, but their parameters account for all variable values.Source: A. A General Procedure · p. 3
B. An Instantiation on a Service Robot
The service-robot instantiation represents navigation, dialog, and delivery tasks in P-log, then constructs task-oriented controllers from declarative knowledge and learned navigation dynamics.
B. An Instantiation on a Service Robot
- Delivery combines question-asking to identify a service request with navigation to fulfill it, under unreliable speech recognition and probabilistic obstacles.If the request is misidentified, the robot returns to the shop, identifies the correct request, and redelivers; the second dialog is assumed to identify it correctly.Source: B. An Instantiation on a Service Robot · pp. 3–4
- P-log represents rigid knowledge with sorts such as time, person, item, and room, and models request distributions and available delivery actions.For example, the model assigns probability 0.8 to people requesting delivery to their own rooms.Source: B. An Instantiation on a Service Robot · p. 3
- The robot's navigation domain uses grid cells with geometric relations, while delivery tasks specify an item, room, and person.Navigation actions are unreliable, and obstacles can probabilistically trap the robot.Source: B. An Instantiation on a Service Robot · p. 3
- Current and next state spaces and probabilistic rules represent action transitions; a rightward move succeeds with probability 0.8, with failures landing in nearby cells.The transition probabilities are learned through model-based RL.Source: B. An Instantiation on a Service Robot · pp. 3–4
- RMax learns navigation-action success rates at different positions, which update delivery-action success rates and support computing delivery success probabilities.For aligned requests and actions, delivery requires one navigation task; a mismatch requires three navigation tasks.Source: B. An Instantiation on a Service Robot · p. 4
- The delivery transition model combines navigation probabilities to estimate request-fulfillment probability, using a learned partial world model and a value-iteration policy.The model-construction procedure reasons over exogenous variables when computing transitions for task-relevant variables.Source: B. An Instantiation on a Service Robot · pp. 3–4
IV. EXPERIMENTS
Experiments test whether navigation-learned knowledge transfers across navigation, dialog, delivery, and unfamiliar settings, including an illustrative real-robot dialog trial.
a) Learn from and applied to navigation tasks
- Knowledge extraction from shop-room1 to shop-room2 accelerated learning, with a more significant improvement in the larger navigation domain.Figure 4 compares 30 × 30 and 50×50 grid domains; each data point averages 1,000 trials.Source: Figure 4 · p. 5, §IV · p. 5
b) Learn from navigation and applied to delivery
- With learning, KRR-RL produced higher overall reward and request-fulfillment rate, and lower question-asking cost; reward improvements were statistically significant at blocking rates 0.1, 0.5, and 0.7.The reported p-values for overall reward were 0.028, 0.035, and 0.049, respectively.Source: Table I · p. 5, §IV · p. 5
- KRR-RL reduced dialog question-asking cost for deliveries to both room2 and room4; room4 deliveries required less cost because room4 is closer to the shop.Figure 5 presents completion-percentage CDFs against QA cost for the two delivery destinations.Source: Figure 5 · p. 5, §IV · p. 5
Information collected in dialog
- As br rose from 0.1 to 0.7, mean belief entropy decreased, and the robot gathered the most information before the more difficult deliveries to room5.The authors interpret the entropy pattern as evidence that navigation learning adjusts dialog information gathering to task difficulty.Source: Table II · p. 6, §IV · p. 6
Generate controllers for new circumstances
- Under an unknown setting, KRR-RL achieved an average 83.8% navigation success rate, versus 26.8% for a baseline that randomly selected a setting-specific policy.The agent learned controllers under three settings with decreasing navigation-action success rates; KRR-RL dynamically constructed policies for the unknown setting.Source: §IV · p. 6
An Illustrative Trial on a Robot
- In the real-robot dialog, navigation learning made the robot risk not confirming the room, while prior knowledge of Bob’s office let it skip asking about the person.Figure 6 tracks item, person, and office beliefs over five turns; the robot became confident in the request by turn four.Source: §IV · p. 6, Figure 6 · p. 6
V. CONCLUSIONS AND FUTURE WORK
KRR-RL combines logical-probabilistic reasoning with model-based RL to incorporate learned world dynamics into task-specific planning models. Robot experiments suggest the approach can use RL-learned knowledge to generate task-oriented action policies; future work includes source-task sequencing and balancing task completion against learning.
V. CONCLUSIONS AND FUTURE WORK
- KRR-RL learns transition probabilities with model-based RL and incorporates them into logical-probabilistic reasoning to construct efficient, run-time task-specific planning models.Source: §V · p. 6
- Experiments with a simulated and physical mobile robot on delivery tasks involving navigation and dialog suggest learned RL knowledge can support reasoning and task-oriented action policies.Source: §V · p. 6
- Future work includes sequencing source tasks for target-task performance and balancing service-task completion against RL; currently, the robot optimizes task completion without considering knowledge gained.Source: §V · p. 6