Source-linked AI summary
Learning Unknown Constraints without Unsafe Data via Optimality and Counterfactual Regularization
Zhouyu Zhang, Chih-Yuan Chiu, Glen Chou
TL;DR
Constraint learning from demonstrations must address CIOC's reliance on known dynamics and ICRL's reliance on exploration that may violate unknown constraints. CF-KKT learns local dynamics and combines demonstration optimality with synthetic counterfactuals; simulations report improved constraint recovery and safety over offline ICRL baselines.
Problem
CIOC typically assumes known dynamics and structured constraints, whereas ICRL supports unknown dynamics and flexible constraints but often needs online exploration that may cause unsafe violations.
Source: §1 · p. 2Method
CF-KKT learns local dynamics from constraint-satisfying expert demonstrations and combines KKT-inspired conditions with synthetic reward-improving counterfactuals to learn constraints without observed violations.
Source: §1 · p. 2Results
Takeaways & Limitations
Learned local dynamics can support constraint recovery without requiring observed constraint-violating trajectories, and can also enable CIOC-based parameter recovery when the constraint parameterization is known.
Source: §1 · p. 2Counterfactual quality depends on learned dynamics away from the demonstrations.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Learning from demonstrations (LfD) provides a framework for inferring unknown constraints from locally optimal, constraint-satisfying expert behavior. Existing approaches largely fall into two paradigms, constrained inverse optimal control (CIOC) and inverse constrained reinforcement learning (ICRL). CIOC exploits optimality conditions such as the Karush--Kuhn--Tucker (KKT) conditions but typically assumes known dynamics and structured constraint representations. Meanwhile, ICRL accommodates complex unknown constraints and unknown transition dynamics but often requires extensive online exploration, during which unsafe constraint violations may occur. In this work, we introduce Counterfactual KKT (CF-KKT), a constraint learning framework that leverages learned dynamics and locally optimal demonstrations to recover unknown constraints without requiring known dynamics or additional risky exploration, thereby combining the data efficiency and safety advantages of CIOC with the flexibility of ICRL. First, we use a locally learned differentiable dynamics model to impose KKT-inspired optimality conditions directly on the demonstrations. Second, we use the learned dynamics to generate reward-improving counterfactual behaviors near the demonstrations, revealing behaviors that would be preferable in the absence of the unknown constraint and thus providing synthetic infeasible data. When the constraint parameterization is known, the same learned-dynamics framework enables direct CIOC-based parameter recovery, and we characterize its sensitivity to dynamics misspecification. Across high-dimensional robotic control tasks, our approach learns neural constraint representations with improved safety and data efficiency relative to state-of-the-art offline ICRL baselines.
1 INTRODUCTION
Constraint learning from locally optimal demonstrations faces a trade-off: CIOC uses optimality but typically needs known dynamics, while ICRL supports unknown dynamics but may require risky exploration. CF-KKT uses learned dynamics and counterfactuals to learn unknown constraints without observed violations, and the paper also studies known-parameter recovery, dynamics misspecification, and simulated performance.
1 INTRODUCTION
- CIOC-based methods use demonstration KKT conditions but assume known smooth dynamics and, in many cases, structured constraints, whereas ICRL accommodates unknown dynamics and flexible neural constraints.ICRL methods often require costly, data-heavy online exploration that may cause unsafe violations; offline variants can still depend on potentially violating data.Source: §1 · pp. 1–2
- CF-KKT learns local dynamics from constraint-satisfying expert demonstrations and combines KKT-inspired optimality conditions with synthetic reward-improving counterfactual trajectories.The counterfactuals provide information about constraint-violating states without executing violating trajectories in the environment.Source: §1 · p. 2
- For known closed-form constraint parameterizations, a two-stage framework learns dynamics without additional demonstration data and then recovers constraint parameters through CIOC.Source: §1 · p. 2
- The paper analyzes how recovered constraints respond to system dynamics misspecification.Source: §1 · p. 2
- In simulation across low- and high-dimensional black-box systems, CF-KKT improves constraint recovery and downstream safety over offline ICRL methods while using structurally less data.The evaluated data are safe.Source: §1 · p. 2
2 RELATED WORK
Related work contrasts CIOC's use of demonstration optimality with ICRL's flexibility under unknown dynamics and constraints. This work combines those strengths using learned local dynamics and synthetic counterfactual evidence.
2 RELATED WORK
- CIOC is data-efficient because it exploits demonstration optimality, but many approaches require known differentiable dynamics and structured constraints, limiting use with high-dimensional or neural representations.GP-based approaches relax parameterization assumptions but scale poorly with dataset size and still require known dynamics.Source: §2 · p. 2
- ICRL accommodates unknown dynamics or rewards but typically relies on online exploration and constrained-RL subroutines, with high computational and sample costs and potential unsafe exploration.Offline ICSDICE avoids online exploration but still uses a fixed dataset containing behaviors not guaranteed to satisfy the unknown constraint.Source: §2 · p. 3
- The proposed approach combines CIOC's safety and data-efficiency advantages with ICRL's representational flexibility through learned local dynamics and synthetic reward-improving counterfactuals.Unlike Chou et al. (2019), it does not rely on hit-and-run sampling alongside known parametric constraints and explicit dynamics.Source: §2 · p. 3
3 PRELIMINARIES AND PROBLEM FORMULATION
This section defines the constrained-learning settings and contrasts ICRL with CIOC. It then specifies recovery of neural constraints from optimal, constraint-abiding demonstrations when both dynamics and constraints are unknown.
3 PRELIMINARIES AND PROBLEM FORMULATION
- The section proceeds from ICRL and CIOC formulations to the constraint-recovery problem considered in the work.Source: §3 · p. 3
3.1 INVERSE CONSTRAINED REINFORCEMENT LEARNING (ICRL)
- The forward CMDP problem maximizes expected discounted reward subject to hard constraints at every time step, with states evolving according to the dynamics map.Source: §3.1 · p. 3
- ICRL recovers unknown constraints from reward-maximizing, constraint-satisfying demonstrations, while exploratory online actions can be costly or unsafe in safety-critical settings.Offline ICSDICE uses expert and additional policy-generated datasets, but the latter may contain constraint-violating behavior.Source: §3.1 · p. 3
3.2 CONSTRAINED INVERSE OPTIMAL CONTROL (CIOC)
- CIOC computes optimal state-control trajectories by minimizing cumulative negative reward subject to the dynamics and constraint maps.Source: §3.2 · p. 4
- CIOC recovers constraints by requiring expert trajectories to satisfy the KKT system, including complementarity between constraint values and their dual variables.When the constraint form is known, recovery reduces to finding parameters that satisfy those conditions; some low-dimensional cases can be solved efficiently as mixed-integer programs.Source: §3.2 · p. 4
- CIOC exploits demonstration optimality but typically needs closed-form dynamics and structured constraints; ICRL supports unknown dynamics and flexible constraints but needs additional behavior data for off-expert coverage.Source: §3.2 · p. 4
3.3 PROBLEM STATEMENT
- The recovery problem assumes known reward and constraint-abiding optimal expert trajectories, while both the dynamics and the unknown constraint are to be recovered or represented.The target constraint is represented by a trainable neural network.Source: §3.3 · p. 4
4 KKT-INFORMED CONSTRAINT LEARNING
The section shares a learned local dynamics model across parametric CIOC recovery and neural constraint learning. CF-KKT combines expert-side KKT supervision with reward-improving counterfactual rollouts to provide synthetic supervision for infeasible behavior.
4 KKT-INFORMED CONSTRAINT LEARNING
- Both recovery procedures use a differentiable local dynamics model fitted from expert transitions, without additional demonstration data.The learned model is used for both inference procedures.Source: §4 · p. 4
4.1 LEARNING PARAMETRIC CONSTRAINTS VIA MIP
- For known closed-form constraint parameterizations, the method applies CIOC to learned dynamics and minimizes expert-trajectory stationarity residuals subject to feasibility and complementary-slackness conditions.The work also analyzes the MIP’s sensitivity to dynamics misspecification.Source: §4.1 · p. 5
4.2 COUNTERFACTUAL KKT (CF-KKT) CONSTRAINT LEARNING
- For neural constraints, CF-KKT uses transition-wise KKT supervision with bounded auxiliary multipliers; full trajectory KKT training produced Inf/NaN losses on BlockedAnt and HalfCheetah.The transition-wise loss penalizes stationarity residuals, primal constraint violation, and complementary-slackness violation, while discouraging large negative constraint values on expert transitions.Source: §4.2 · p. 5
- CF-KKT perturbs expert controls and rolls them out from the same demonstrated state using learned dynamics, comparing their value-augmented returns with the expert-action baseline.A value estimate accounts for consequences beyond the finite rollout horizon.Source: §4.2 · p. 6
- Under Lemma 1’s assumptions, a counterfactual with greater value-augmented return than the globally optimal feasible expert implies constraint violation at some rollout timestep.In practice, positive value-augmented advantage is used as a soft selection criterion because the assumptions need not hold exactly.Source: §4.2 · pp. 6–7
- For selected counterfactuals, CF-KKT trains the average learned constraint value along the rollout to exceed a positive margin, using fixed synthetic samples for constraint updates.Averaging provides a denser, smoother loss gradient than a max-based or single-step objective.Source: §4.2 · p. 7
5 EXPERIMENTS
The experiments evaluate constraint recovery and downstream control across known parametric constraints, continuous mazes, and high-dimensional tasks. Results show improved recovery and safety in the reported comparisons, with trade-offs in some settings.
5 EXPERIMENTS
- The study compares three increasingly unstructured settings and evaluates against ICSDICE, SMODICE, and IQ-Learn, with adaptations for baselines that do not recover explicit constraint masks.In the known-parametric setting, it also compares MIP with parametric ICSDICE.Source: §5 · p. 7
5.1 KNOWN PARAMETRIC CONSTRAINTS UNDER UNKNOWN DYNAMICS
- MIP closely recovers true θ on Swimmer, Quadcopter, and HalfCheetah; its BlockedAnt estimate is −2.05 versus −3.0, while parametric ICSDICE estimates −1.23 on HalfCheetah and −3.99 on BlockedAnt.The larger BlockedAnt MIP error is consistent with larger KKT residuals in its demonstrations.Source: §5.1 · p. 7, Table 1 · p. 8
5.2 UNKNOWN CONSTRAINT GEOMETRY IN CONTINUOUS MAZES
- 0.707 mean IoU for CF-KKT versus 0.425 for ICSDICE and 0.379 for SMODICE; CF-KKT recovers constraint geometry more accurately in every maze.The largest gains over ICSDICE are on Disk (0.815 versus 0.428) and Obstacle (0.590 versus 0.276).Source: §5.2 · p. 8, Table 2 · p. 8
- 0.174 → 0.042, 0.200 → 0.085, 0.140 → 0.080, and 0.209 → 0.093 are CF-KKT’s violation reductions versus ICSDICE on Obstacle, Wide, Disk, and Zigzag, respectively.CF-KKT retains comparable goal-reaching performance across the four environments.Source: §5.2 · p. 9, Table 3 · p. 9
5.3 HIGH-DIMENSIONAL NEURAL CONSTRAINT LEARNING
- 34.6%–96.8% fewer mean violations than ICSDICE across five tasks; CF-KKT also improves return on BlockedAnt and BlockedHopper, while other tasks show safety–return trade-offs.On Walker_ls, SMODICE has fewer violations but substantially lower return.Source: §5.3 · p. 9, Table 4 · p. 9
6 CONCLUSION
The paper presents learned-dynamics approaches for recovering unknown constraints from demonstrations, including CF-KKT for neural constraints. Experiments report improved recovery and downstream safety over offline ICRL baselines, alongside limitations in assumptions and model dependence.
6 CONCLUSION
- The framework learns unknown constraints from expert demonstrations under unknown dynamics, using CIOC-based recovery for known parametric constraints and CF-KKT for neural constraints.CF-KKT combines KKT-inspired expert supervision with synthetic reward-improving counterfactuals to provide off-demonstration constraint information without violations.Source: §6 · p. 9
- Experiments on robotic control tasks show improved constraint recovery and downstream safety over offline ICRL baselines.Source: §6 · p. 9
- The approach assumes a known differentiable reward, omits cross-time coupling in its transition-wise KKT objective, and its counterfactual quality depends on learned dynamics away from demonstrations.Loss weights are also environment-dependent.Source: §6 · p. 10
GENERATIVE AI USE DISCLOSURE
The disclosure describes generative AI assistance in preparing the manuscript and states that the authors reviewed the arguments and verified the reported results. The appendix collects environment and dataset details omitted from the main experimental section.
GENERATIVE AI USE DISCLOSURE
- Generative AI assisted with mathematical statements, experimental-methodology feedback, manuscript organization and presentation, language editing, and equation and table formatting.Source: GENERATIVE AI USE DISCLOSURE · p. 10
- The authors state that they reviewed and revised AI-assisted mathematical arguments and produced and verified all reported experimental results.Source: GENERATIVE AI USE DISCLOSURE · p. 10
A BENCHMARK ENVIRONMENTS AND DATASETS
- The appendix collects environment and dataset details omitted from the main experimental section.Source: A BENCHMARK ENVIRONMENTS AND DATASETS · p. 13
A.1 KNOWN-PARAMETRIC EXPERIMENTS
The experiments cover structured parametric constraints, neural constraint recovery in navigation and MuJoCo tasks, and implementation and planning details. They use feasible expert demonstrations, while some baseline and evaluation procedures also use suboptimal data.
A.1 KNOWN-PARAMETRIC EXPERIMENTS
- One expert trajectory per environment is used to fit local dynamics and recover parametric constraints for HalfCheetah, BlockedAnt, Swimmer, and Quadcopter.HalfCheetah and BlockedAnt use a position lower bound; Swimmer and Quadcopter use spherical obstacle avoidance.Source: §A.1 · p. 13
A.2 CONTINUOUS-MAZE EXPERIMENTS
- The maze experiments use four 2D navigation tasks with unknown wall-attached, compact, or non-convex unsafe regions, without giving the learner the state-only constraint structure.Each environment and seed has approximately 30,000 feasible expert transitions; ICRL baselines additionally receive 7,000 suboptimal transitions that may contain violations.Source: §A.2 · p. 13
A.3 MUJOCO NEURAL-CONSTRAINT EXPERIMENTS
- Five MuJoCo locomotion tasks pair task rewards with state constraints that are not explicitly penalized by those rewards.The datasets contain approximately 5 × 10^4 expert transitions in 50 trajectories and 2 × 10^5 benchmark suboptimal transitions; ground-truth labels are reserved for evaluation and oracle diagnostics.Source: §A.3 · pp. 13–14, Table 6 · p. 14
B EXPERIMENT DETAILS
- The MuJoCo evaluation retains suboptimal data for downstream policy extraction, but CF-KKT does not use that data during constraint learning.The protocol is retained for consistency with ICSDICE and SMODICE, which require a suboptimal dataset as constraint learners.Source: B EXPERIMENT DETAILS · p. 15
B.1 MAZE PLANNING VISUALIZATIONS
- Figure 3 visualizes representative maze plans, distinguishing predicted unsafe boundaries, true unsafe regions, oracle routes, executed unsafe timesteps, and goals.The legend maps these to dashed contours, green, purple, yellow, and blue stars, respectively.Source: §B.1 · p. 15
B.2 ROBUSTNESS TO LEARNED REWARD MODELS
The learned-reward ablation tests whether preserving reward gradients matters for CF-KKT. Additional diagnostics compare its KKT-only component with online ICRL methods and sweep counterfactual rollout and acceptance choices.
B.2 ROBUSTNESS TO LEARNED REWARD MODELS
- Constraint violations fall 133.5→51.1 on HalfCheetahWithObstacle and 65.1→37.3 on BlockedHopper with reward-gradient regularization.Across all five environments, gradient regularization reduces violations; the results support preserving local reward-gradient structure when using a learned reward approximation.Source: §B.2 · pp. 15–16, Table 7 · p. 16
B.3 ADDITIONAL BENCHMARKING AGAINST ONLINE ICRL METHODS.
- The KKT-only learner has lower constraint violation and higher feasible reward than BC2L and MECL, and is comparable to GACL in BlockedHalfCheetah.Unlike the online baselines, it relies on initial expert demonstrations; this is a diagnostic of the KKT signal, not the full CF-KKT method.Source: §B.3 · p. 16, Figure 4 · p. 17
B.4 METHOD SWEEP ON COUNTERFACTUAL GENERATION
- H = 12, 24, and 48 yield comparable downstream performance, while H = 24 has the lowest mean violation on all five environments; positive-advantage gating also has lower violation than quantile-based acceptance.The quantile rule retains the top 25% of candidates by advantage, whereas CF-KKT uses the positive-advantage gate.Source: §B.4 · p. 17, Table 8 · p. 17
C EMPIRICAL VALIDATION OF THE COUNTERFACTUAL PREMISE
The diagnostics assess whether positive value-augmented advantage selects genuinely infeasible counterfactual rollouts, including when violations emerge only after multiple steps. The accompanying ablation and metrics define how expert-side and counterfactual supervision are evaluated.
C EMPIRICAL VALIDATION OF THE COUNTERFACTUAL PREMISE
- Bag precision is at least 0.922 and enrichment exceeds 1 in all five environments, so selected counterfactual rollouts identify trajectory-level infeasibility and concentrate more violations than the benchmark suboptimal distribution.The selection rule uses learned dynamics and continuation-value estimates; ground-truth labels are used only for this diagnostic, not for CF-KKT training.Source: C EMPIRICAL VALIDATION OF THE… · pp. 17–18
- Violations can emerge late: BlockedHopper's rate rises from 0.08 at h = 1 to 0.72 at h = H, and HalfCheetah's rises from zero to 0.83.These delayed effects support multi-step rollouts for constraints whose consequences propagate through the dynamics.Source: C EMPIRICAL VALIDATION OF THE… · p. 18
- State precision ranges from 0.749 to 1.000, supporting the mean constraint value as a dense surrogate although the theoretical guarantee requires only one violation somewhere in a rollout.The empirical results indicate selected rollouts commonly contain violations across multiple states.Source: C EMPIRICAL VALIDATION OF THE… · p. 18
D.1 EXPERIMENTAL SETTING
- Seven controlled conditions isolate expert-side feasibility, activity, complementary-slackness, counterfactual, and KKT-only supervision, alongside the complete CF-KKT objective.Other experimental settings are held fixed across conditions.Source: §D.1 · p. 19
D.2 METRICS
- Mask admission rate compares the share of benchmark suboptimal transitions admitted with the ground-truth share, while ROC AUC measures held-out feasible/infeasible ranking independently of threshold.Ground-truth labels are used only for evaluation.Source: §D.2 · p. 19
D.4 ANALYSIS
The ablations show that counterfactual and KKT supervision provide complementary signals: counterfactuals discriminate off-demonstration states, while KKT supervision helps calibrate a useful feasibility threshold. Their combination supports downstream performance that AUC alone does not ensure.
D.4 ANALYSIS
- CF-only and full CF-KKT have the two highest mean AUC values, 0.778 and 0.749, while KKT-only reaches 0.564.KKT-only is near chance on four environments but reaches 0.976 AUC on HalfCheetah.Source: §D.4 · p. 20
- CF-only has ρθ = 0 on all five environments, rejecting every transition at the fixed threshold; counterfactuals improve AUC on four of five tasks but can make masks overly restrictive.KKT-only tends toward the opposite error: its admission rate exceeds the ground-truth rate on four of five environments.Source: §D.4 · p. 20
- Mean absolute admission-rate deviation is 0.123 for full CF-KKT, versus 0.526 for CF-only and 0.232 for KKT-only.Removing the activity term increases admission rate in every environment with only modest AUC changes; complementary slackness adds little without the stationarity residual.Source: §D.4 · pp. 20–21
- CF-only suffers severe return collapse on every task, whereas CF-KKT achieves substantially higher return on all five and improves both return and violation count on Ant_ls and BlockedAnt.The comparison shows that discrimination must be paired with a useful operating threshold for downstream control.Source: Table 13 · p. 21, §D.4 · p. 21
E.1 SENSITIVITY ANALYSIS FOR MIP-BASED INFERENCE OF CONSTRAINT PARAMETERS
The section analyzes how dynamics-model weight perturbations affect the constraint parameters recovered by MIP-based inference, under local regularity conditions.
E.1 SENSITIVITY ANALYSIS FOR MIP-BASED INFERENCE OF CONSTRAINT PARAMETERS
- If the learned dynamics are C4, strict complementarity holds at the nominal solution, and the KKT Jacobian is nonsingular, the nearby solution is unique.The proposition establishes this result within neighborhoods of the nominal dynamics weights and primal-dual solution.Source: §E.1 · p. 22
- Locally, the recovered primal-dual solution’s change is bounded by a sensitivity coefficient times the dynamics-weight perturbation, plus a second-order remainder.The coefficient is the norm of the inverse KKT Jacobian multiplied by the dynamics-parameter derivative at the nominal solution.Source: §E.1 · pp. 21–22
- Strict complementarity excludes simultaneous tightness of the primal and dual inequalities but does not require every inequality constraint to be active.It makes the local equality representation of complementarity unambiguous; the active set need not be fixed a priori.Source: §E.1 · pp. 21–22
- At active-set transitions, generalized-equation sensitivity results may apply under suitable regularity, but the solution map need not be differentiable.The proposition’s differentiable local bound applies under its stated strict-complementarity conditions.Source: §E.1 · p. 22