SegBench-GC: Testing Segmentation Invariance in Multi-Step Offline Goal-Conditioned Reinforcement Learning

1 Introduction

• Administrative cuts may not be true terminations, and their sensitivity in offline GCRL had not been isolated while transitions, goal semantics, optimization, and evaluation were held fixed.Treating such cuts as absorbing removes continuation value from multi-step targets.Source: §1 · p. 1
• SegBench-GC varies artificial backup boundaries while fixing transition tuples, source trajectories, sampled-goal semantics, optimization settings, and evaluation.Its continuation-valid control preserves standard nonterminal bootstrapping at artificial cuts while genuine goal completion remains noncontinuing.Source: §1 · p. 2
• The matched-count PointMaze study tests three optimization seeds and three segmentation realizations, and reports a large penalty and segmentation sensitivity when artificial cuts are absorbing.Target- and critic-level diagnostics and a published n-step baseline provide additional mechanism and cross-learner evidence.Source: §1 · p. 2
• 20.0 percentage points: artificial-terminal handling reduced primary mean success versus CVT, while independent Puzzle-4x5 validation showed 0.27% naive success versus 58.5% with CVT.The reported results support a scoped claim about sensitivity to administrative segmentation, not a claim that CVT outperforms correctly uncut data.Source: §1 · p. 2

2 Related Work

• Pardo et al. (2018) establish bootstrapping at nonterminal truncations; SegBench-GC applies that principle to trajectory-aware multi-step offline GCRL rather than proposing a new Bellman correction.OGBench also distinguishes trajectory ends used for future-goal sampling from Bellman masks for task success.Source: §2 · p. 2
• Offline-GCRL methods explore multi-step propagation and temporal abstraction, but the cited evaluations do not use benign resegmentation as a counterfactual reliability intervention with transitions and future-goal semantics fixed.The discussion includes Horizon Reduction, OTA, multistep quasimetric learning, Decoupled Q-Chunking, and Transitive RL.Source: §2 · pp. 2–3
• Theory on fixed-length segments studies information-loss failures, whereas SegBench-GC keeps ordered transitions and source trajectories intact and changes only backup segmentation.The paper limits its empirical claims to two tested learners consuming trajectory prefixes.Source: §2 · p. 3
• The experiments pair seeds and segmentation configurations and aggregate segmentation variants within seed before aggregating across seeds, following recommendations for reporting deep-RL variability.The cited evaluation guidance favors paired seeds, uncertainty, and distributions over isolated point estimates.Source: §2 · p. 3

3.1 Controlled resegmentation

• The method adds artificial backup boundaries to ordered transitions while keeping source-trajectory boundaries fixed for future-goal sampling, so resegmentation does not change the underlying transitions or goal semantics.The Original control ignores artificial cuts; segmented conditions stop reward accumulation at a cut.Source: §3 · p. 3, §3.1 · p. 3

3.2 Continuation-valid targets

• At an artificial cut, CVT retains continuation from a valid stored successor, whereas naive handling zeros that continuation; both conditions disable bootstrap at genuine goal completion.For the primary learner's negative goal reward, dropping continuation shifts naive targets upward; CVT applies standard nonterminal truncation semantics, not a new Bellman rule.Source: §3.2 · pp. 3–4

3.3 Segmentation invariance

• Segmentation sensitivity is summarized using the standard deviation of learned-policy rollout performance across artificial segmentations.When multiple segmentations share an optimization seed, the method aggregates those variants within seed before aggregating across seeds.Source: §3.3 · p. 4

3.4 Test subjects and goal protocol

• The primary test subject is a goal-conditioned IQL-style learner with twin horizon-indexed critics and value heads for horizons {1, 2, 4, 8}.One-step GCIQL is an exact negative control because it does not consume artificial backup-boundary metadata.Source: §3.4 · p. 4
• Cross-implementation validation uses the published Decoupled Q-Chunking NS baseline with policy chunk size one, backup horizon 25, and no chunk critic.Primary PointMaze value goals mix current, future-trajectory, and random states; goal completion receives reward zero and disables bootstrap regardless of segmentation.Source: §3.4 · p. 4

4.1 Primary matched-count study

• The PointMaze study compares uncut data with three matched-count segmentations, using 35,000 artificial cuts and paired CVT and naive runs at each optimization/segmentation seed.All conditions use the same source-boundary semantics; CVT retains continuation at artificial cuts, while naive handling zeros it.Source: §4.1 · p. 5

4.2 Independent published 𝑛-step validation

• The independent published n-step validation uses the NS configuration with a 25-step backup, three optimization seeds, and one fixed segmentation realization.CVT and naive minibatches are identical except for the artificial-cut continuation mask; the validation is not a segmentation-dispersion study.Source: §4.2 · p. 5

4.3 Diagnostics and negative control

• Diagnostics compare CVT and naive targets from identical transitions and a shared CVT value function, then compare learned critics near artificial cuts; one-step GCIQL is the negative control.Paired replay checks require one-step GCIQL's sampled observations, goals, rewards, and masks to remain unchanged under resegmentation.Source: §4.3 · p. 5

4.4 Metrics and aggregation

• Primary estimates average segmentation realizations within each optimization seed before aggregation; independent-validation standard deviations span three seeds under one fixed segmentation, and uncertainty summaries are descriptive.The table reports final five-task success as mean ± sample SD over optimization seeds and includes the paired CVT-minus-naive difference.Source: §4.4 · pp. 5–6, Table 1 · p. 6

5.1 Artificial terminalization substantially degrades the primary learner

• 50.5 ± 15.4% uncut, 39.1 ± 5.1% CVT, and 19.1±0.2% naive success; the paired CVT-minus-naive difference is 20.0±5.4 percentage points.This contrast holds despite identical source-boundary semantics, transitions, goals, architecture, and optimization settings.Source: §5.1 · p. 6

5.2 The arbitrary segmentation realization materially changes naive performance

• Naive segmentation means are 20.5%, 31.9%, and 4.8% across the three realizations, while individual optimization/segmentation runs range from 0% to 37.6%.Transitions and source trajectories remain unchanged; only artificial backup-boundary locations vary.Source: §5.2 · p. 6

5.3 The target-level mechanism is exact and produces learned critic optimism

• Target identity residual is at most 3.8 × 10−6; naive targets shift upward on 99.8% of affected slots by a mean of 39.26.After training, naive-minus-CVT value and Q predictions are +15.54 ± 2.13 and +15.84 ± 1.62, while CVT-minus-Original predictions remain near zero in aggregate.Source: §5.3 · pp. 6–7

5.4 A published 𝑛-step baseline independently reproduces the failure

• 47.2 ± 14.3% uncut, 58.5 ± 19.5% CVT, and 0.27 ± 0.23% naive Puzzle-4x5 success in the published NS (𝑛= 25) baseline.The paired CVT-minus-naive difference is 58.3 ± 19.8 percentage points; this supports a cross-learner failure pattern, not a general CVT improvement claim.Source: §5.4 · p. 7

5.5 Negative control

• One-step GCIQL's replay samples remain identical under resegmentation, localizing the observed failure to multi-step target construction.Its sampled observations, goals, rewards, and masks are unchanged because it does not consume artificial backup-boundary metadata.Source: §5.5 · p. 7

• CVT requires an observed successor from the same continuing process and must not bootstrap across resets, missing-data discontinuities, environment termination, or goal completion.The benchmark applies only when boundary semantics can be identified well enough to distinguish continuation from true termination.Source: §6 · p. 7
• The evidence is limited to three optimization seeds and three segmentation realizations on PointMaze Medium; broader validation should cover more algorithms, manipulation domains, and naturally fragmented datasets.The reported standard deviations are descriptive, not well-powered inferential confidence intervals, and the results do not establish the same sensitivity magnitude for every multi-step offline RL algorithm.Source: §6 · p. 7
• At 100k updates, CVT reaches 39.1% versus 50.5% for uncut training, leaving an 11.4-percentage-point gap rather than perfect segmentation invariance.Approximation error, finite optimization, and horizon-indexed heads that need not satisfy an exact continuing fixed point may leave residual sensitivity.Source: §6 · p. 7

7 Conclusion

• Treating artificial nonterminal cuts as absorbing substantially reduces performance and makes the primary learner sensitive to the segmentation realization.Target- and critic-level diagnostics connect the degradation to optimistic pseudo-terminal targets, and a separate published n-step implementation reproduces the failure pattern outside the primary learner.Source: §7 · p. 8
• Multi-step offline RL should make boundary semantics explicit and test sensitivity to benign resegmentation when sequence boundaries enter target construction.CVT is the continuation-consistent control for this test, and SegBench-GC provides a controlled protocol to isolate and measure the failure mode.Source: §7 · p. 8

Ethics statement

• The main methodological risk is misclassifying administrative data boundaries as genuine terminations; the study uses public benchmark data and simulated environments, with no human subjects, personal data, or physical deployment.The authors address the risk by stating continuation assumptions, releasing aggregate artifacts and protocol tests, and limiting claims to tested learner families.Source: Ethics statement · p. 8

Reproducibility statement

• The appendix and supplementary source code provide target arguments, hyperparameters, seed-level outcomes, pinned revisions, configurations, runners, deterministic aggregate tables, tests, and build utilities.The matched protocols and aggregation rules appear in Section 4.Source: Reproducibility statement · p. 8

• For a nonterminal artificial cut after k < h valid transitions, the CVT target is introduced along a stored trajectory whose value satisfies the continuing Bellman equation.Source: A Continuation-valid target identity · p. 9
• The corresponding artificial-terminal target provides the comparison target for the same artificial cut.Source: A Continuation-valid target identity · p. 9
• Expanding a consistent continuing value over the remaining h−k transitions recovers the uncut backup only when the successor is part of the same continuing process.Genuine goal completion, environment termination, reset discontinuities, and missing successors remain noncontinuing; an explicitly finite-horizon value would require a residual-horizon quantity such as V(h−k).Source: A Continuation-valid target identity · p. 9

• The primary learner uses horizons {1, 2, 4, 8}, twin critics, horizon-indexed value and actor outputs, and a fixed actor horizon mixture weighted toward horizon 8.The architecture uses three 512-unit hidden layers, GELU activations, layer normalization, discount 0.99, expectile 0.9, batch size 1024, learning rate 3 × 10−4, target update coefficient 0.005, and DDPG+BC coefficient 0.003.Source: B Primary experimental details · p. 9
• PointMaze value goals mix current, future-trajectory, and random states, while actor goals mix future-trajectory and random states equally.The source adapter uses non-compacted OGBench transitions so continuing collections retain their stored successors.Source: B Primary experimental details · pp. 9–10
• The primary intervention samples exactly 35,000 artificial cuts for each of segmentation seeds 101, 102, and 103, with optimization seeds 0, 1, and 2.Source boundaries remain continuation-valid in all conditions; at artificial cuts, CVT bootstraps and naive handling does not, and final reevaluation uses 50 episodes for each of five fixed tasks.Source: B Primary experimental details · p. 10

C Independent NS validation details

• The Puzzle-4x5 Play validation uses the published NS configuration with backup horizon 25, two critics, and policy chunk size one.It also uses expectile distillation, quantile implicit backup, mean Q aggregation, κ_b=0.7, κ_d=0.5, and batch size 4096.Source: C Independent NS validation details · p. 10
• CVT and naive minibatches are paired by resetting NumPy sampling state, with all batch fields matching except the continuation mask on cut-affected backups.A 4096-sample verification batch contained 1846 cut-affected backups and passed the equality check.Source: C Independent NS validation details · p. 10
• Training runs for 250k updates, and final reevaluation uses 50 episodes per task.Artificial cuts are sampled at 3.5% of eligible successor-state locations using segmentation seed 101; optimization seeds are 100001, 200002, and 300003.Source: C Independent NS validation details · p. 10

D Seed-level results

• Table 2 reports seed-level final five-task success (%), averaging the three segmentation realizations for primary PointMaze CVT and naive entries, but using one fixed realization for Puzzle NS validation.The artifact protocol averages segmentation realizations within an optimization seed before aggregating optimization seeds.Source: Table 2 · p. 10
• Naive PointMaze success is 20.5%, 31.9%, and 4.8%; CVT success is 39.6%, 42.3%, and 35.5% for segmentation seeds 101, 102, and 103, respectively.These are primary segmentation-seed means; Table 2 reports seed-level final five-task success (%).Source: D Seed-level results · p. 10, Table 2 · p. 10

E Artifact organization

• Each run logs its resolved configuration, optimization seed, boundary counts and continuation assumptions, update step, held-out evaluation, rollout success, and checkpoint metadata.Final reevaluations, target-identity diagnostics, and boundary-local critic contrasts are stored separately; released manifests and generated tables provide provenance for reported summaries.Source: E Artifact organization · p. 10