Source-linked AI summary
Command-Space Counterfactual Explanations for Pareto-Conditioned Reinforcement Learning
Joanikij Chulev, Hendrik Baier
TL;DR
PCNs make multi-objective reinforcement-learning decisions through user-specified return commands, but their local command-to-action mapping remains opaque. This paper introduces CF-ZOO, which searches for minimally changed commands that induce foil actions, succeeding in 79.9% of tested cases versus 40.4% for WHITE-CW.
Problem
Although desired-return commands are interpretable, the local mapping from state and command to action in PCNs remains opaque.
Method
CF-ZOO searches in command space for minimally changed desired-return commands that make the same policy choose a specified foil action.
Results
79.9% of cases succeeded with CF-ZOO, compared with 40.4% for WHITE-CW across 797 tested cases.
Takeaways & Limitations
The explanations express how a user's requested trade-off would need to change for the same policy to select a different action at the same state.
Takeaways & Limitations
A successful counterfactual is local and does not imply that the counterfactual command is Pareto-optimal; failures may reflect optimizer failure or unsupported foils.
Abstract
from arXiv · showhide
Pareto Conditioned Networks learn multiple multi-objective reinforcement learning behaviours by conditioning a single policy on a desired return command. However, the local mapping from command and state to action remains opaque. We propose command-space counterfactual explanations for PCNs: given a fixed state, original command, and foil action, we search, in a black-box setting, for a minimally changed desired-return command under which the same trained policy would choose the foil. Our contributions are threefold. First, we formulate PCN explanations as return-command interventions, using a return-only PCN variant that avoids the added ambiguity of horizon-conditioning. Second, we adapt adversarial machine learning methods to reinforcement-learning explanations. Third, we introduce a boundary-seeded directional search that improves over purely local optimization in the command-action landscape, resulting in our proposed approach CF-ZOO. The resulting explanations are actionable and intuitively expressed in the user's own preferences: "If your trade-off had shifted slightly towards X, the agent would have chosen Y."
1 Introduction
The introduction frames PCNs as compact multi-behaviour policies whose interpretable commands still leave state-to-action decisions opaque. It proposes command-space counterfactuals and CF-ZOO to explain which minimal trade-off change would produce a different action under the same policy.
- Multi-objective reinforcement learning represents trade-offs among objectives such as safety, efficiency, and cost using vector-valued rewards and policies.
- PCNs represent multiple behaviours with one neural policy conditioned on desired return and horizon, but their local command-to-action mapping remains opaque.
- The proposed explanation asks what minimal change to a PCN’s desired-return command would make the same policy choose a different action in the same state.
- CF-ZOO repurposes adversarial and black-box optimization, including finite-difference zeroth-order queries, to expose action dependence on requested multi-objective trade-offs.
- In Deep Sea Treasure, the example contrasts moving toward treasure value 5 with continuing toward value 8 under command Rt = (8.000, −2.000), which makes the submarine move right.
2 Background
MORL extends reinforcement learning to vector-valued rewards and uses Pareto trade-offs to represent competing objectives. PCNs encode these trade-offs in a desired-return command, while this paper adopts a return-only formulation to study command-space action counterfactuals.
- MORL foundations: MORL replaces scalar rewards with reward vectors, so policies produce vector-valued returns evaluated through Pareto dominance.A return vector dominates another when it is no worse in every objective and strictly better in at least one.
- MORL foundations: Scalarization maps vector returns to a scalar utility but requires specified preferences and can miss parts of the Pareto front.Multi-policy and conditioned-policy methods instead represent multiple trade-offs within one learned system.
- Pareto-Conditioned Networks: PCNs learn one policy network conditioned on state, desired return, and originally also desired horizon, reproducing actions associated with return–horizon commands.The paper uses a revised return-only variant because horizon conditioning was sometimes an unstable or unintuitive control signal, even for Pareto-optimal desired returns.
- Command-space counterfactuals: For a fixed state, command-space counterfactuals seek a small desired-return perturbation that changes the policy’s preferred action.The return command is therefore treated as an interpretable intervention point, avoiding counterfactual changes to both return and time.
- Related counterfactual explanations: Unlike visual, symbolic-state, or action-sequence counterfactuals, PCN explanations ask how the user’s requested objective trade-off must change for a different local action to become preferred.This framing is especially natural for MORL because its focus is on objectives.
3 Command-Space Counterfactual Search
CF-ZOO explains a fixed-state PCN decision by changing only the desired-return command until a foil action becomes greedy. It combines a targeted black-box margin objective with Pareto-front directional seeding and ZOO refinement to seek small feasible command changes.
- Counterfactual formulation: A counterfactual keeps the state and policy fixed while changing the desired-return command so the foil action replaces the original greedy action.The policy is queried as a black box through action log-probabilities for candidate commands.
- Black-box objective: The targeted margin M_f(R) compares the foil’s log-probability with the strongest competing valid action, with M_f(R) ≥ κ enforcing a nonzero winning margin.The foil is greedy when M_f(R) ≥0; κ ≥0 specifies the desired strictness of the flip.
- Black-box objective: The objective penalizes command distance and targeted flip failure while clipping commands to the feasible reward box.The coefficient c > 0 controls the flip-penalty strength, while the first term favors small command changes.
- Directional boundary seeding: Directional boundary seeding addresses flat or non-convex command-action landscapes by testing feasible directional endpoints and binary-refining successful perturbations toward the original command.The seed supplies a global directional guess before local refinement; unsuccessful directions are discarded.
- Directional boundary seeding: Pareto-front solutions provide behavioral direction heuristics, and the closest successful ray candidate seeds subsequent ZOO coordinate refinement; if none succeeds, ZOO starts from the original command.ZOO estimates coordinate derivatives with finite-difference queries at the fixed state and optimizes the command perturbation using ZOO-ADAM.
4 Worked Examples
Worked examples in Branch-Path, Collect-Two, and Minecart show that command-space counterfactuals produce action flips that can be communicated as intuitive, preference-based explanations. The examples span discrete grids and a continuous-state benchmark, including a sparse one-objective intervention and a learned ore-trade-off.
- Qualitative inspection: Across the three environments, the examples demonstrate both action flips and human-facing explanations of how command preferences determine agent behavior.The environments are Branch-Path, Collect-Two, and Minecart, with the latter testing interpretability beyond small discrete grids.
- Branch-Path: Branch-Path changes the greedy action from up to down when the command asks for much less objective A and somewhat more objective B.The counterfactual command is Rcf = (0.2953, 1.464), with δR = (−2.7047, 0.464).
- Collect-Two: Collect-Two changes the greedy action from down to right by reducing only the desired return for objective D toward objective C.The counterfactual changes D by δR_D = −0.1563, yielding Rcf = (0, 0, 0.8, 0.8437).
- Minecart: Minecart changes the greedy action from strongly left to right when the user requests slightly more of the first ore objective while leaving the second ore and fuel commands unchanged.This interpretable counterfactual makes the policy stop steering toward a nearby mine containing only ore 1 and steer right instead.
5 Experimental Results
CF-ZOO is evaluated on diagnostic environments and standard MORL benchmarks, comparing boundary-seeded zeroth-order search with a white-box Carlini–Wagner baseline. It achieves higher overall success and remains effective in a broad stress test, while dense-grid analysis separates search failures from apparently infeasible foils.
- Experimental setup: Experiments use custom diagnostic environments and standard MORL benchmarks, including Deep Sea Treasure and Breakable-Bottles.DST trades treasure value against time, while Breakable-Bottles accounts for delivery time and environmental side effects.
- White-box comparison: CF-ZOO succeeds in 637/797 cases (79.9%), compared with 322/797 (40.4%) for WHITE-CW.WHITE-CW adapts the Carlini–Wagner targeted attack with white-box gradient access, whereas CF-ZOO uses boundary-seeded directional search before zeroth-order refinement.
- Stress test: Across 3,195 stress-test cases, CF-ZOO succeeds in 2,422 (75.8%).It succeeds on all Collect-Two and Fruit-Tree cases and performs highly on Minecart and Reward-Line, while DST and Resource-Gathering have lower success rates.
- Failure diagnosis: Dense-grid diagnosis reports 78.7% of all cases solved and 16.6% with grid-infeasible foils.A foil is treated as grid-infeasible when no evaluated grid command flips the action, providing heuristic evidence that the policy may not have learned that behaviour.
6 Discussion and Scope
The discussion frames PCN counterfactuals as local command interventions whose validity and usefulness depend on policy response, optimizer success, and command-space criteria. It also distinguishes this question from approaches that search for action sequences leading to counterfactual states.
- Interpretation and limitations: A successful counterfactual means that, at state st, the same trained PCN would prefer foil action af under command Rcf, not that Rcf is Pareto-optimal.Failure may instead reflect optimizer failure, an unsupported foil, or local policy insensitivity to the desired-return command.
- Relation to reinforcement-learning counterfactuals: Unlike RACCER, which searches for an action sequence reaching a counterfactual state, this work asks how the command can change so the policy chooses the foil at the fixed state.The distinction addresses the sequential and temporally embedded nature of reinforcement-learning decisions.
- Counterfactual desiderata: Validity requires the margin condition Mf(Rcf) ≥κ, meaning the foil becomes preferred by the PCN.This specializes the validity desideratum to the command-intervention setting.
- Counterfactual desiderata: Proximity uses the scaled command distance ∥D−1(Rcf−Rt)∥2, while actionability means that users can directly issue the counterfactual command.Sparsity concerns changing few objective dimensions, although the current objective emphasizes small scaled distance rather than an explicit ℓ0 penalty.
- Counterfactual desiderata: In practice, explanations are usually sparse, changing 1 or 2 command entries.This practical sparsity is reported alongside the distance-based objective rather than an explicit ℓ0 penalty.
7 Conclusion and Future Work
The paper introduces desired-return counterfactual explanations for command-conditioned MORL policies and proposes CF-ZOO to compute trade-off changes that induce specified foil actions. It identifies infeasible foils and optimizer failure as limitations and calls for human studies of explanation understandability and usefulness.
- Contributions and scope: CF-ZOO computes desired-return counterfactuals that identify how a requested trade-off must change to induce a specified foil action for a fixed policy and state.The approach is currently PCN-specific but could extend to other agents with user-controllable preference, goal, or utility-conditioning inputs.
- Limitations: Failures can result from infeasible foil actions or optimizer failure.These limitations are discussed in Section 5.3.
- Future work: Human studies with participants outside MORL and RL are needed to test whether command counterfactual explanations are understandable and useful.Related work has examined whether non-experts can identify flawed agents and whether counterfactual visualizations improve understanding of agent preferences.
A Appendix
The appendix covers the return-only PCN adjustment and the construction of landscapes and grids used in the analysis.
- Return-Only PCN Adjustment: The appendix includes a return-only PCN adjustment.
- Landscape and Grid Construction: It also describes landscape and grid construction.
A.1 Implementation Details · Scaled Command Coordinates
The implementation uses scaled command coordinates because raw perturbation distances are incomparable across reward objectives with different ranges and units. Commands are handled relative to feasible bounds, with invalid or near-zero ranges replaced by 1 and commands projected to the feasible box.
- Scaled Command Coordinates: Scaled command coordinates address incomparable raw perturbation distances across reward objectives with different ranges and units.The scaling is introduced to make perturbation distances comparable across objectives.
- Scaled Command Coordinates: Feasible command bounds are defined using Rmin and Rmax.These bounds establish the feasible range for each command coordinate.
- Scaled Command Coordinates: The scaled-coordinate formulation uses feasible command bounds when representing perturbations.The passage introduces Rmin and Rmax as the relevant bounds for this representation.
- Scaled Command Coordinates: Invalid or near-zero command ranges are replaced by 1.This replacement avoids using invalid or nearly degenerate ranges in the coordinate scaling.
- Scaled Command Coordinates: Commands are projected to the feasible box.Projection keeps command values within the feasible command region.
- Scaled Command Coordinates: Optimization is evaluated using the scaled command representation.The passage states that optimization evaluates the scaled formulation, although the supplied excerpt does not include the full expression.
Binary Refinement Along Successful Rays … A.5 Runtime Details
The appendix specifies CF-ZOO’s ray refinement, coordinate-wise ZOO-ADAM optimization, return-only PCN design, command-landscape construction, hyperparameter reporting, and runtime guidance. It also documents visualization limits and the trade-off between computational cost and counterfactual quality.
- Binary Refinement Along Successful Rays: Binary refinement searches each successful ray for its smallest successful step, defining αf(u) as the infimum step meeting Mf(Rt + αDu) ≥κ.The refinement targets the nearest successful endpoint rather than accepting a potentially farther one.
- Binary Refinement Along Successful Rays: 16 refinement steps reduce the remaining ray interval to at most 2^-16 of its original width.Each binary-search step halves the remaining interval.
- ZOO-ADAM Coordinate Optimizer: CF-ZOO re-implements ZOO-ADAM with uniformly sampled coordinate updates, finite-difference derivative estimates using h = 10^-3, and two policy queries per estimate.Coordinate-wise ADAM uses β1 = 0.9, β2 = 0.999, and ϵ = 10^-8.
- A.2 CF-ZOO Hyperparameters: CF-ZOO settings are reported in Table 4, whose white-box comparison matches the stress-test settings except for random-direction and query budgets.The passage identifies the reported hyperparameter source and the stated comparison difference.
- A.3 Return-Only PCN Adjustment: The return-only PCN removes horizon conditioning because it could add variation to the command-action mapping, while retaining trajectory filtering and relabelling effects.The return-only and return–horizon variants had comparable evaluation results, so the simpler command space was selected.
- A.4 Landscape and Grid Construction: Each landscape panel varies two desired-return dimensions over 121 equally spaced values each, queries every pair, and colors the foil margin Mf(R) against validity threshold κ.Valid commands satisfy Mf(R) ≥κ, while unshown command dimensions, state, foil action, and policy remain fixed.
- A.4 Landscape and Grid Construction: Two-dimensional panels are slices through the full d-dimensional command space and should not be interpreted as complete visualizations.The feasibility audit instead varies all command dimensions using a Cartesian-product grid.
- A.5 Runtime Details: CF-ZOO is slower because it adds directional screening, refinement, and zeroth-order local search, while runtime grows steadily with query budget; 50k is recommended as the best compromise.Absolute runtimes depend on hardware and implementation and may improve with faster hardware or code optimization.