PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Action Models with Conditional and Quantified Effects via Uncertainty-Guided Exploration

Jeffrey Jewett, William Solow, Sandhya Saisubramanian

arXiv:2608.30955v1cs.AI

TL;DR

Accurate planning requires action models that represent conditional and quantified effects, but learning them online is difficult because the hypothesis space is large and observations can be noisy. OHCAM combines soft-consistency beliefs, dynamic hypothesis expansion, and disagreement-driven exploration; across six domains it completes more tasks with fewer samples than baselines, with additional validation on a Kinova Gen3 robot.

  • Problem

    Learning expressive conditional and quantified action models from limited online interactions is challenging because possible effects form a combinatorial space and observations may be noisy.

  • Method

    OHCAM maintains a belief over hypotheses, uses noise-tolerant soft consistency, expands simple models only when needed, and selects actions maximizing disagreement among plausible hypotheses.

  • Results

    OHCAM completed >95% of tasks on five of six benchmark domains, while the next best approach completed only two domains even with expert data.

  • Takeaways & Limitations

    OHCAM learns expressive action models from limited interactions while remaining robust to noisy observations and improving downstream planning performance.

  • Takeaways & Limitations

    Learning quantified effects assumes identifiability and locality, requiring matching quantified effects to have a single explanation and affected objects to relate to action parameters.

Abstract

from arXiv · show

Accurate action models are critical for effective planning. Existing action-model learning methods largely assume simple action representations or become computationally intractable when learning conditional and quantified effects. We present Online Hypothesis-Driven Conditional Action Model Learning (OHCAM), an online approach for learning action models with conditional and quantified effects from limited interactions with the environment. OHCAM maintains a belief over hypothesized action models and actively selects informative actions to reduce uncertainty by maximizing disagreement among competing hypotheses, while being robust to noisy observations. To enable scalability, OHCAM begins with a small set of simple action model hypotheses and expands to more complex conditions only when the current hypotheses become inconsistent with the data. Experiments on six benchmark planning domains demonstrate that OHCAM is sample efficient in learning action models that solve substantially more tasks than baselines, even with observation noise. We validate OHCAM on two tasks using a Kinova Gen3 robot, demonstrating the real-world applicability of our approach.

Introduction

Learning conditional and quantified action models online is challenging because real-world effects depend on context, affect additional objects, and are observed noisily. OHCAM addresses these challenges through belief-guided exploration and dynamically expanded hypotheses, achieving stronger benchmark and robotic-task performance.

  • Motivation: Context-dependent outcomes and effects on nearby objects require expressive conditional and quantified action models, but their combinatorial hypothesis space limits scalability and noise robustness.Noisy state estimates can obscure the true action dynamics during deployment.
  • Motivation: Existing methods largely learn unconditional STRIPS effects or restrict learnable conditions to control the complexity of conditional-model learning.These restrictions limit applicability to complex environments.
  • OHCAM: OHCAM maintains a belief over candidate models, uses soft consistency for noisy observations, and expands from simple hypotheses only when data support greater complexity.Branch-and-bound search avoids exhaustive reasoning over the combinatorial hypothesis space.
  • OHCAM: OHCAM selects actions that maximize disagreement among plausible hypotheses, making exploration sample efficient.The approach actively gathers interactions that reduce uncertainty about competing models.
  • Evaluation: Across six benchmark domains and two Kinova Gen3 tasks, OHCAM achieves higher task completion, improved sample efficiency, and practical online learning in noisy environments.The benchmark and hardware evaluations support its applicability beyond simulation.

Related Works

Prior action-model learning work mainly targets STRIPS representations, while methods for conditional and quantified effects face noise, hypothesis-space, or generalization limitations. Online approaches emphasize informative action selection because real-world data collection is costly.

  • Action Model Learning: Most AML methods infer STRIPS models with unconditional effects restricted to action parameters, limiting representation of conditional behaviors and effects on additional objects.Logical induction can also eliminate the ground-truth formula after a single noisy observation.
  • Conditional and Quantified Effects: Methods for conditional and quantified effects include deictic references, clustering, LAMP, and Conditional-SAM, but face permissive conditions, rapidly growing hypothesis spaces, or restrictions on quantified variables.The cited approaches trade expressiveness against tractability or noise robustness.
  • Capability Comparison: Table 1 compares action-model learning approaches across capabilities required for real-world environments.The table is framed around capability comparison rather than a single performance metric.
  • Online AML: Online AML selects actions and incrementally updates models from observed outcomes, making sample-efficient active exploration important when executions are time consuming.Recent work distinguishes competing hypotheses through multi-step plans rather than purely myopic exploration.

Problem Formulation

The paper studies online inference of symbolic action preconditions and effects when actions may have conditional and quantified outcomes. The agent must choose informative executions under a limited budget despite noisy observations.

  • Setting: A planning domain consists of predicates describing object attributes and relations, parameterized actions, and action models specifying preconditions and effects.Preconditions determine applicability, while effects describe state changes.
  • Setting: The agent knows available macro-actions and their parameters but must learn how those actions affect the environment online.This setting reflects robotics scenarios where exact action consequences are unknown before deployment.
  • Conditional and Quantified Effects: Conditional effects apply only when their context condition is satisfied, while quantified effects can influence objects outside the action parameters through universally quantified variables.Quantified effects apply to all objects satisfying the corresponding condition.
  • Online Learning: The learning objective is to infer action preconditions and effects from observed state transitions and success indicators using actively selected actions under budget B and noisy observations.The agent observes states before and after execution together with whether the action succeeded.

Hypothesis-Driven Action Model Learning

OHCAM learns conditional and quantified action models by combining probabilistic hypothesis scoring with dynamic expansion and disagreement-based exploration. Its search begins with simple candidates and selectively refines them to avoid exhaustive reasoning over an exponential hypothesis space.

  • Hypothesis Scoring: OHCAM factorizes model learning into hypothesis sets for preconditions and potential effect literals, estimating each hypothesis from predicted versus observed state changes.Execution-literal comparisons yield true-positive, true-negative, false-positive, and false-negative counts.
  • Hypothesis Scoring: A soft-consistency likelihood reduces but does not eliminate hypotheses after incorrect predictions, while a simplicity prior penalizes complex antecedents.The score balances predictive fit with model complexity.
  • Search Challenge: Finding the MAP hypothesis is difficult because candidate sets grow combinatorially with the number of literals and possible conditional effects.The computational challenge motivates structured search rather than exhaustive evaluation.
  • Dynamic Hypothesis Expansion: Dynamic expansion starts with simple hypotheses and incrementally adds antecedent literals using best-first branch-and-bound search.The search refines only hypotheses with sufficient potential to improve the current best score.
  • Belief-Guided Exploration: In the blocks example, OHCAM selects actions with high predictive disagreement, shifts posterior weight after observations, and replaces a hypothesis with a more complex one to preserve diversity.The figure illustrates belief updates and adaptive hypothesis expansion during interaction.
  • Dynamic Hypothesis Expansion: Admissible upper bounds prioritize refinements that could surpass the current best hypothesis, and termination occurs when no frontier bound exceeds that score.This procedure returns a MAP hypothesis while controlling explored candidates.
  • Dynamic Hypothesis Expansion: Restricting refinement literals to those with promising score bounds and sufficient true negatives reduces branching relative to considering every bound literal.The refinement set contains literals that plausibly correlate with the effect and offset the simplicity penalty.

4 // Belief-guided exploration

The procedure imagines trajectory rollouts and identifies the first action of the best rollout.

  • Trajectory rollouts are imagined as part of the procedure.
  • The rollout process connects imagined trajectories to an action-selection step.
  • The procedure selects the first action of the best rollout.

13 // Dynamic hypothesis expansion

OHCAM expands quantified-effect hypotheses dynamically while using posterior disagreement to plan informative action sequences. It updates beliefs after execution and returns the highest-posterior action model when the budget is exhausted.

  • Dynamic quantified hypotheses: Quantified effects explain impacts on objects outside action parameters using universally quantified variables and locality relationships.Affected objects are connected to action-parameter objects through observed state relationships.
  • Dynamic quantified hypotheses: OHCAM forms candidate UQV sets from state-graph trees connecting affected objects to action-parameter objects, starting with the simplest trees.Tree nodes become UQVs and edges become antecedent conditions.
  • Dynamic hypothesis expansion: The simplicity prior penalizes hypotheses with more conditions and UQVs, while dynamic expansion adds complexity only when existing hypotheses cannot explain effects.The quantified-hypothesis prior includes a penalty factor β^|Q| for additional UQVs.
  • Belief-guided exploration: BALD selects actions expected to reduce posterior uncertainty by maximizing disagreement among plausible hypotheses.Disagreement may concern action applicability or the resulting state and effects.
  • Belief-guided exploration: The exploration utility accounts for applicability uncertainty because effects are observed only when actions execute successfully.Predictive entropy decomposes into uncertainty about applicability and individual effects.
  • Online learning: OHCAM scores receding-horizon imagined rollouts by cumulative information gain, executes the selected action, and reweights or expands hypotheses until budget B is exhausted.The final output is the MAP action model M′.

Experiment Setup

The evaluation combines six simulated planning domains with two Kinova Gen3 hardware tasks. Methods are compared across online, random, expert, and active-learning data under fixed training and task-completion protocols.

  • OHCAM is evaluated on six benchmark planning domains and two real-world tasks using a Kinova Gen3 robot.
  • Baselines: The baselines include offline learners of conditional and quantified effects and OLAM, an online active learner limited to STRIPS effects.Conditional-SAM is evaluated with three maximum antecedents and ground-truth UQV type.
  • Data collection: Three data regimens are compared: OHCAM-collected active data, uniformly random actions, and expert demonstrations with 20% random actions.
  • Domains: The simulation domains are Satellite, Briefcase, Miconic, Maintenance, CityCar, and Nurikabe; two additional domains support hardware evaluation.
  • Training and evaluation: Each domain uses ten training and ten harder evaluation problems, and every method receives up to one hour for learning.Evaluation problems contain more objects or longer plans.
  • Metrics: Task completion rate measures the average fraction of evaluation tasks solved by open-loop plans without failed actions, while sample efficiency is evaluated at 10-action intervals.Offline methods are evaluated on increasing data subsets.

Simulation Results and Discussion

Across six simulation domains, OHCAM achieves stronger task completion and sample efficiency than the baselines, while degrading gracefully under observation noise. Its actively collected data also benefits other learners.

  • Task completion: >95% of tasks were completed by OHCAM on five of six domains, whereas the next-best approach reached that level on only two domains with expert data.OHCAM consistently performed better than or similarly to the baselines.
  • Task completion: Conditional-SAM learned overly restrictive conditions in larger domains because its strict safety threshold required more data than the budget allowed.
  • Task completion: LAMP often exceeded the one-hour training limit because it exhaustively enumerated hypotheses.
  • Task completion: Cluster&Intersect failed to generalize to additional evaluation objects because it learned an action for each training-observed object combination.
  • Sample efficiency: Within 100 actions, OHCAM trained on its actively collected data recovered expert-demonstration performance without prior domain knowledge.OHCAM-collected data also enabled higher problem-solving rates for other baselines than random data.
  • Robustness to noise: With noise increasing from zero to 40%, OHCAM degraded gracefully, while baselines declined sharply; under 10% noise, its simplicity prior prevented false effects.OHCAM-collected data was more robust to noise than expert data.

Robot Hardware Results

The robot evaluation uses a Kinova Gen3 arm and compares action-model learners on tabletop manipulation tasks. OHCAM solves all evaluation tasks despite perception noise, while simpler and baseline methods perform worse.

  • Hardware and setup: The setup uses a Kinova Gen3 7 DoF arm and an Intel RealSense D435i camera for tabletop manipulation.Actions are implemented as manipulation skills parameterized by object locations.
  • Evaluation: Figure 6 reports task completion rates across 10 MagnetBlocks evaluation tasks and five SpongeWorld tasks.The figure also shows example start and goal configurations.
  • Results: OHCAM solves all evaluation tasks in both domains using online-collected data despite perception noise.The baselines fail because they cannot address noise in the data.
  • Results: The naïve model without conditional effects solves only 20% of MagnetBlocks tasks and 40% of SpongeWorld tasks.In MagnetBlocks, the baselines learn a model incapable of finding a plan for any task.

Conclusion

OHCAM combines dynamic hypothesis expansion with belief-guided exploration to learn expressive action models from limited interactions while remaining robust to noisy observations. Benchmark and robot experiments report improved sample efficiency and downstream planning performance over state-of-the-art baselines, with future work targeting partial observability and larger relational domains.

  • Conclusion: OHCAM combines dynamic hypothesis expansion and belief-guided exploration to learn expressive action models from limited interactions under noisy observations.The framework targets conditional effects and quantified variables.
  • Conclusion: Experiments on benchmark planning domains and a Kinova Gen3 robot report higher sample efficiency and downstream planning performance than state-of-the-art baselines.The evaluation spans both simulated planning domains and physical robot tasks.
  • Conclusion: Future work will address partially observable settings and scaling OHCAM to larger relational domains.

Appendix A: Theoretical Results and Proofs

The theoretical results establish that dynamic hypothesis expansion returns a MAP hypothesis while bounding search complexity by evidence-supported conditional-effect complexity. In the noiseless case, the returned hypothesis and generated search depth are bounded by the true environment’s simplest perfectly predictive effect.

  • MAP correctness: Dynamic hypothesis expansion returns a MAP hypothesis under a finite literal set and best-first branch-and-bound search.The proof shows the MAP hypothesis remains refinable, is eventually generated, and is preserved by the termination criterion.
  • Search bounds: The search automatically bounds generated hypothesis depth by the MAP hypothesis rather than a predefined maximum-depth hyperparameter.Only hypotheses with potential exceeding the current best score are refined, and refinement can be restricted to a thresholded literal set.
  • Search bounds: The complexity-to-noise ratio ζ represents the minimum prediction errors that must be eliminated to justify adding one literal.The proof uses ζ to show that removing a MAP-condition literal decreases true negatives sufficiently for that literal to remain in the refinement set.
  • Noiseless complexity bound: In the noiseless case, the MAP hypothesis complexity satisfies |Cmap| ≤ |C∗|, so generated hypotheses have depth no greater than the true conditional effect.Thus dynamic expansion does not search beyond the complexity of the simplest true effect that perfectly predicts the environment.
  • Likelihood and prior: The simplicity prior α and noise parameter ϵ jointly control how conditional hypotheses are penalized and how missed predictions are tolerated.The paper sets α = 0.1 for sparse conditional effects and uses ϵ = 0.1 for hardware experiments, corresponding to 11% noise.

Appendix B.2: Additional Results

Appendix B.2 adds predictive-power and hypothesis-expansion analyses, showing how learned models predict outcomes and how OHCAM limits hypothesis growth across benchmark domains. OHCAM generated fewer hypotheses than Conditional-SAM in every domain, while Conditional-SAM reached 637 thousand hypotheses at depth 3 on Nurikabe.

  • Predictive power: Predictive power measures how accurately learned models predict action applicability and effects, complementing task success evaluation.Applicability and effect precision and recall are averaged over applicable states and actions.
  • Predictive power: Cluster&Intersect could not be reliably evaluated because non-mutually-exclusive action branches let planners choose among outcomes, unlike the deterministic environment.Conditional-SAM also timed out on multiple domains despite predicting 100% of the evaluation dataset, because its quantified preconditions were complex.
  • Number of hypotheses expanded: OHCAM expands hypotheses according to the true conditional-effect depth instead of enumerating every hypothesis up to a fixed antecedent depth.The analysis motivates minimizing hypothesis enumeration as domains and effect complexity increase.
  • Number of hypotheses expanded: 637 thousand hypotheses were generated by Conditional-SAM at d = 3 on Nurikabe, while OHCAM generated fewer hypotheses than Conditional-SAM in every domain.OHCAM learned a depth-3 conditional effect on Nurikabe while still producing fewer hypotheses than Conditional-SAM at d = 1.
  • Benchmark domains: The benchmark suite spans six domains with varying action, predicate, conditional-effect, and quantified-effect complexity.The domains are Satellite, Briefcase, Miconic, Maintenance, CityCar, and Nurikabe.
  • Benchmark domains: CityCar was the most challenging domain because a quantified effect required a rare interaction involving a destroyed road and a car on it.The analysis also observed that overly complex preconditions affected task completion.
  • Noise considerations: Robot-collected data contained perception and execution noise, including object misidentification, depth-map errors, and occasional sponge slippage.These observations required the learning algorithms to distinguish noise from genuine conditional effects; approximately 5% of action failures during data collection resulted from perception noise.

Appendix D.2: Robot Data Collection and Evaluation

The robot study collected online interaction data in MagnetBlocks and SpongeWorld, then evaluated learned PDDL models on tasks requiring conditional-effect knowledge. Data collection was time-consuming and involved realistic perception noise, while evaluation reran plans only after failures attributable to perception or execution noise.

  • Data Collection: All hardware experiments used a human supervisor because irreversible conditional effects could apply unsafe torque to the robot.One example is placing a block over another with a magnet block between them.
  • Data Collection: 250 skill executions were collected in each of MagnetBlocks and SpongeWorld, with data collection taking approximately 45 minutes to 1.25 hours.Most time was spent executing skills and resetting the environment rather than selecting actions.
  • Robot Evaluation: Learned PDDL models were evaluated with Fast Downward on ten MagnetBlocks tasks and five SpongeWorld tasks.Seven MagnetBlocks tasks and four SpongeWorld tasks required knowledge of conditional effects.
  • Robot Evaluation: Each task was run once per learned model, with reruns only when perception or execution noise caused failure.The evaluation targeted online learning under noisy observations and sample efficiency, rather than robot execution accuracy.
Loading 2608.30955v1…