When Do Options Help? Policy Necrosis and Redundant Coverage in Option-Critic

1 INTRODUCTION

• The scaling curve cannot identify temporal abstraction as the cause because increasing K also changes lower-level capacity and optimization geometry, as well as the router’s decision burden.The paper varies termination, lower-level exploration, and K across transition kernels, reward definitions, and tabular and neural implementations.Source: §1 · p. 1
• The termination rule is vacuous when its test and option selector read the same values, and can incur Ω(T) regret when it vetoes selector exploration.Always terminating holds to O(log T) regret on every instance of the exhibited class; experiments find that forcing termination leaves the option-count dependence intact.Source: §1 · p. 2
• At temperature 10^-3 with no lower-level exploration, states can lock onto an initially positive-advantage action and stop updating, leaving roughly half of states necrotic in a typical option.Restoring exploration repairs the failure: one option improves from 306.7 to 55.4 expected steps and improves on all 350 seeds.Source: §1 · p. 2
• 59% to 4%: adding options leaves mean single-option necrosis near 60% but lowers the chance that every option is necrotic in the same state.Performance follows this joint failure quantity, not individual-option improvement.Source: §1 · p. 2
• The paper’s account is that multiplicity masks a repairable lower-level failure, while exploration repairs it and learned termination plays no part.It also proposes a state-level diagnostic and a frozen-checkpoint evaluation protocol with exploration disabled during testing.Source: §1 · p. 2

2 SETUP

• An option comprises an intra-option policy, an initiation set, and a termination function, while a policy over options selects the next option after termination.Option-critic learns the intra-option policy, termination function, and option policy from return.Source: §2 · p. 2
• FourRooms has 103 non-goal states and supports exact solution of the MDP, allowing the study to identify acceptable actions state by state.The constant-cost reward makes optimal actions those that reduce wall-respecting distance to the goal; the sparse-reward setting derives acceptable actions by value iteration.Source: §2 · p. 3
• Direct and Inertia test the mechanism with neural function approximation, using position or position-and-velocity observations and different action spaces.In Direct actions are unit displacements; Inertia uses accelerations and collisions zero the velocity.Source: §2 · p. 3
• Evaluation freezes checkpoints, disables lower-level exploration, marginalizes the retained option-selector exploration exactly, and computes expected capped steps by finite-horizon dynamic programming.Tabular termination and neural experiments instead report mean steps over the last 100 training episodes; training-seed contrasts are paired across 350 seeds.Source: §2 · p. 3

3 DETECTING POLICY NECROSIS

• An option is necrotic at a state when its policy has committed to an action outside the optimal policy’s acceptable-action set.Under constant cost, an option is viable when its highest-preference action reduces wall-respecting distance to the goal; under sparse reward, acceptable actions come from exact value iteration.Source: §3 · p. 3
• With K options, each state-seed pair is D when all options are necrotic, M when the router selects a necrotic option despite viable coverage, or R when it selects a viable option.D isolates states no router choice can help; M identifies missed usable coverage.Source: §3 · p. 4
• Comparing the probability all options are necrotic with mean single-option necrosis distinguishes redundant coverage from improvement to individual options.If extra options improve each option, both quantities fall; if they supply redundancy, only the joint-failure probability falls.Source: §3 · p. 4

4.1 WHAT THE TERMINATION GRADIENT IS HEADING FOR

• When the selector and termination test use consistent values, the learned rule terminates at every step and is identical to β≡1.The update raises termination for non-greedy options, heading toward greedy interruption.Source: §4.1 · p. 4
• With inconsistent values, learned termination can veto the exploring selector; the resulting worst-case separation is not pointwise and can favor either rule.The authors note an instance where the veto wins, but it has no uniform sublinear guarantee.Source: §4.1 · p. 4

4.2 WHAT THE EXPERIMENTS MEASURE

• Forcing β≡1 removes temporal extension, so the surviving option-count curve shows that temporal extent is not needed for the measured improvement.With termination at every step, the option is a latent index refreshed at each state.Source: §4.2 · pp. 4–5
• At K=8, learned termination loses all five tabular kernels by 11.1%–37.0%, while at K=2 it wins by 11.6%–23.8%.At K=4, it is within ±5% in all five kernels, with three intervals covering zero; the neural grid has no consistent direction.Source: Table 1 · p. 5, §4.2 · p. 5
• With β≡1, expected steps fell 963.2 →822.3 →384.4 →122.5 as K grew, matching the eightfold improvement under learned termination.The paired difference contained zero at every option count, using exact frozen dynamic programming.Source: §4.2 · p. 5

5 POLICY NECROSIS, AND WHAT REPAIRS IT

• At τ = 10^-3 and learning rate 0.25, an advantage is scaled by 250, driving a sampled action toward near-certain selection and silencing its score-function update.Under sparse reward, the first advantage can be nearly arbitrary, so the state may not recover through further training on that option.Source: §5 · p. 5
• The matched fork changes only executed actions: after an identical 500-episode prefix, rescue adds εlo = 0.1 exploration, and both branches are evaluated with εlo = 0.This isolates parameter changes written during training from randomness at test time.Source: §5 · p. 5
• At K=1, rescue reduces expected steps by 251.2 across 350 seeds; under constant-cost reward, a β ≡1 option falls from 963.2 to 36.5 expected steps.Its frozen success probability rises from 0.037 to 0.999.Source: §5 · p. 6, Figure 2 · p. 6
• The paired rescue reduction falls from 251.2 steps at K=1 to 85.7, 9.9, and −3.9 at K=8, as extra options leave less necrosis cost to repair.Under constant-cost reward, the exploring agent worsens from 36.5 to 85.0 expected steps as options are added.Source: §5 · p. 6, Figure 2 · p. 6
• With one option, neural ε-greedy reaches 299 and 250 mean steps, while entropy regularization reaches 207 and 156 across the two environments.Both mechanisms are compared with the base softmax under β ≡1 throughout.Source: §5 · p. 6, Figure 3 · p. 7
• Adding options helps the neural base softmax, helps ε-greedy less, and makes entropy-regularized performance monotonically worse in both environments.Inertia uses τ = 0.5, yet its one-option base softmax still fails.Source: §5 · pp. 6–7, Figure 3 · p. 7

6 WHAT EXTRA OPTIONS PROVIDE IS REDUNDANT COVERAGE

• The coverage analysis uses constant-cost FourRooms, where acceptable actions are exactly those that reduce distance; the sparse-reward version in Appendix D agrees.The authors describe the sparse-reward value spread as making a shortest-path yardstick coarser there.Source: §6 · p. 7
• At K=1 versus K=8, mean single-option necrosis stays near 59%, while joint necrosis falls from 59.0% to 3.9% and expected steps from 959.5 to 220.5.The added options fail in different places rather than becoming individually better.Source: §6 · pp. 7–8, Table 3 · p. 8
• At K=8, joint necrosis is 3.9% versus 1.5% under independent failures and 59% under identical failures; the effective independent-option count is 6.20.Shared trajectories and a shared critic make failures somewhat more correlated than chance predicts, so coverage depends on failure decorrelation.Source: §6 · p. 8
• As joint necrosis falls, the router picks a necrotic option despite a viable one in 0%, 17.6%, 32.9%, and 42.7% of states at K=1, 2, 4, and 8.At K=8, routing error is the dominant residual error rather than lack of coverage.Source: §6 · p. 8
• With εlo = 0.1, moving from one option to eight lowers joint necrosis by 35.5 points but raises single-option necrosis by 21.4 points and expected steps by 134.9.The passage attributes the cost to spreading a fixed budget over eight options and asking the router to distinguish eight policies.Source: §6 · p. 8, Table 3 · p. 8
• On the map, joint necrosis occupies 62 of 103 cells at K=1 and none at K=8; four of every five vacated cells become routing misses.For the explored agent, correctly routed cells fall from 66 to 50 as K rises to eight.Source: §6 · p. 8

7 RELATED WORK

• The options framework formalizes temporally extended actions, while option-critic learns intra-option policies and terminations from task return.Related directions include learned initiation, options from environment structure or intrinsic objectives, and hierarchy over goals.Source: §7 · p. 8
• Prior termination-collapse remedies add deliberation cost or use an option-model objective; the paper argues these hold β away from its boundary without removing the objective-driven drift.The cited work treats collapse as an empirical nuisance to engineer around.Source: §7 · p. 8
• Nachum et al. (2019) attribute most gains from goal-conditioned hierarchies to high-level exploration, whereas this paper locates its mechanism inside the option’s lower-level policy parameterization.Source: §7 · p. 9
• Policy necrosis is linked to softmax policy-gradient sensitivity and slow movement near simplex corners; prior work identifies exploration or regularization as necessary for global convergence.The paper’s distinction is that option-critic covers for the phenomenon with options.Source: §7 · p. 9
• The paper places its controlled hierarchical evaluation in a broader literature reporting that gains can weaken under stronger controls and larger seed counts.Source: §7 · p. 9

8 DISCUSSION

• A pool of options can improve aggregate performance without repairing any option: partly overlapping necrotic sets reduce joint failure, while individual failure remains.The discussion argues that redundancy should be measured by how decorrelated failures are, not by option count alone.Source: §8 · p. 9
• The same accounting applies to architectural priors: repair the base learner first, then test whether the prior still helps.The discussion connects deliberate ensemble redundancy with option-critic's incidental redundancy.Source: §8 · p. 9
• The necrosis diagnostic needs an acceptable-action set; outside exactly solvable settings, conclusions inherit the quality of a surrogate.The point at which multiplicity stops helping also depends on κ, which is set by how options were trained together.Source: §8 · p. 9
• Future work proposes evaluating diversity by its effect on κ at fixed K, extending the necrosis test beyond exact solutions, and studying termination objectives other than return.The discussion names deliberation costs and predictability-based termination as objectives that change the return-based analysis.Source: §8 · p. 9
• The release includes code, derived tables, figure-rebuilding tools, and a validator for file hashes and the figure contract.The paper also says the study uses no human subjects, personal data, or deployment, and reports AI use only for readability editing.Source: REPRODUCIBILITY STATEMENT · p. 10, ETHICS STATEMENT · p. 10, AI USE STATEMENT · p. 10

A EXPERIMENTAL SETTINGS

• Tabular FourRooms runs use 350 seeds per cell, 1,000 episodes, and a 1,000-step cap across five transition kernels.The settings specify a 13×13 grid with 104 free cells; matched-fork runs save checkpoints every 10 episodes and snapshot the random-number-generator state.Source: A EXPERIMENTAL SETTINGS · p. 13
• Neural experiments use Direct and Inertia environments, 50 seeds per configuration, and 1,000 episodes with a 1,000-step cap.The agents use two-hidden-layer tanh networks of width 64; exploration settings differ by environment and configuration.Source: A EXPERIMENTAL SETTINGS · p. 13

B STATISTICS

• Tabular confidence intervals are 95% paired-bootstrap intervals over seeds; neural intervals resample individual seeds, while frozen exact evaluation adds no sampling error.When kernels are pooled, the analysis averages within each seed and counts the shared p = 0 cell once.Source: B STATISTICS · p. 13

C TERMINATION RULES AND REGRET

• The termination analysis fixes a minimal two-level model to expose interactions between termination and exploration; its negative constructions rule out a general guarantee.The authors caution that saying both levels use UCB does not by itself specify an algorithm.Source: C TERMINATION RULES AND REGRET · p. 13

C.1 A TWO-LEVEL STOCHASTIC BANDIT

• The bandit model treats each option–action pair as a leaf, and its regret is decomposed into upper- and lower-level contributions.The setting has one recurring state and independent bounded rewards for leaves.Source: §C.1 · pp. 13–14

C.2 COMPATIBLE TWO-LEVEL UCB, AND THE TWO TERMINATION RULES

• Compatible two-level UCB scores each option by its highest leaf index, then selects the option with the highest such score.The appendix compares a rule that fixes β ≡1 with greedy interruption, which may continue the current option instead of switching to the selector's proposal.Source: §C.2 · p. 14

C.3 FIXED β ≡1: FINITE-TIME BOUNDS

• With β ≡1, the process is stepwise equivalent to UCB1 run directly on the M option–action leaves.The appendix also gives instance-dependent and distribution-free bounds for this fixed-termination process.Source: §C.3 · p. 14
• For fixed non-degenerate Bernoulli instances, consistent algorithms cannot improve the O(ln T) regret rate to o(ln T).The lower bound is attributed to Lai & Robbins (1985).Source: §C.3 · p. 14

C.4 TWO VALUE SETTINGS FOR THE RETURN-MAXIMISING RULE

• When selector and termination values use the same optimistic indices, the return-maximising rule terminates at every step and matches the fixed-termination algorithm pathwise.The proof establishes matching action sequences, and therefore matching regret, on every sample path.Source: §C.4 · p. 15
• With raw critic values for termination and confidence bonuses for selection, the selector may propose an option that the termination test refuses to enter.The mismatch occurs when the selector's proposed option has a higher score but a lower raw value than the current option.Source: §C.4 · p. 15

C.5 PROOF OF THEOREM 2

• On an event of probability 1/16, learned termination runs the bad option and incurs at least (T −2)/64 regret, while always terminating has regret at most 64 ln T + C0/2 on the same instance.The resulting worst-case regret is Θ(T) for learned termination versus O(log T) for always terminating.Source: §C.5 · p. 15
• Even with episode resets, the two-step counterexample yields learned-termination regret at least 1 + (T −8)/64, while always terminating selects among four leaves by UCB1.The lower bound follows because the fixed prefix restores option B at the second state in every episode.Source: §C.5 · p. 15

C.6 NO POINTWISE DOMINANCE

• At T = 11, always terminating has regret 3 versus 2 for learned termination, so the worst-case separation does not imply pointwise dominance.The termination veto can lock in an error when the current option is suboptimal, but can save exploration cost when it is optimal.Source: §C.6 · p. 16

C.7 RELATION TO THE INTERRUPTION THEOREM

• The fixed-policy interruption theorem does not cover this online process: its values are finite-sample estimates and its upper policy changes with the counts.In the counterexample, estimation error makes the termination rule continue with B while preventing the samples that could correct the error.Source: §C.7 · p. 16

C.8 EPISODIC TABULAR RL

• With β ≡1, hierarchical selection exactly matches optimistic maximisation over augmented actions; the regret guarantee requires the corresponding model-based Bellman recursion and confidence event.When option-action pairs duplicate one another, using D ≤ |O||A| can be loose unless statistics are pooled across options.Source: §C.8 · pp. 16–17

C.9 SUMMARY

• The paper concludes that return-maximising termination is redundant under value consistency and unguaranteed when the selector uses optimistic values but the termination test uses raw values.The two rules are not comparable instance by instance.Source: §C.9 · p. 17

• Table 4 compares learned termination with forced per-step reselection using exact frozen evaluation of expected capped transitions and success probability across 350 matched seeds.Differences are paired with 95% bootstrap intervals; at K = 1, termination cannot change which option runs, providing a seed-matching noise-floor row.Source: Table 4 · p. 17
• The termination comparison’s tabular grid reverses sign completely between K = 2 and K = 8, while the neural grid has no consistent direction.Figure 5 encodes learned-termination minus β ≡1 step counts, with blue favouring learned termination and red favouring forced termination.Source: Figure 5 · p. 17
• Expected steps fall 306.7 →152.9 →79.1 →56.4 as K increases, while joint necrosis falls 47.5 →31.8 →14.6 →3.3 percent and mean single-option necrosis stays near 50%.The router’s miss rate rises to 39.2%; effective independent option count is κ = 1.82, 2.85 and 4.14 at K = 2, 4, 8.Source: Figure 6 · p. 18
• In constant-cost FourRooms, the β ≡1 agent’s coverage maps are visually and numerically close to learned termination at every option count.The maps show episode-1000 checkpoint coverage under teleport p = 1/2; dark grey marks walls and yellow marks the goal.Source: Figure 7 · p. 18