PaperlayerView exact revision on arXiv

Source-linked AI summary

Completion-Aware Cross-Fidelity Offline-to-Online Reinforcement Learning for Multi-Line Bus Holding

Yifan Zhang, Qifan Zhang, Liang Zheng

arXiv:2609.39868v1cs.AIcs.LG

TL;DR

The paper studies whether offline-to-online bus-holding policies transfer across simulators while keeping passenger journeys complete. It combines a fixed H2O+ actor with a completion-safety reserve and finds improved passenger-time and completion outcomes against the actor’s direct parent in fresh SUMO episodes.

  • Problem

    Offline policies cannot acquire new experience, while simulator interaction may not transfer across differences in target transitions and event durations, and lower passenger time alone does not ensure completed journeys.

    Source: §1 · pp. 2–3

  • Method

    The controller combines a fixed H2O+ actor with a deterministic reserve that modifies actions in a bounded set of low-load tail-bus states.

    Source: §1 · pp. 3–4

  • Results

    92.12 s per departed passenger lower generalized passenger time, completion increased by 0.00466, and unfinished passengers decreased by 77.22 versus the direct parent.

    Source: §7 · p. 24

  • Takeaways & Limitations

    The reserve improves its fixed parent at the registered endpoint within the evaluated SUMO scenario family.

    Source: §7 · p. 24

    The dataset has millions of correlated transitions from only 30 environment cells, and Markov sufficiency of the centralized observation is an assumption, not a proved property.

    Source: §6.3 · pp. 22–23

Abstract

from arXiv · show

Exploratory reinforcement learning (RL) on an operating bus fleet is impractical,while policies trained only from historical data cannot acquire new experience. Hybrid Offline-and-Online (H2O) RL combines fixed target replay with simulator interaction, but the inexpensive online simulator can differ from the target in transition and event-duration dynamics. We study this cross-fidelity problem for multi-line bus holding and address a failure mode in which lower generalized passenger time coexists with incomplete passenger journeys.

1. Introduction

Because simulator interaction may not transfer reliably to target dynamics, bus-holding policies must be assessed for both passenger time and journey completion. The paper pairs an H2O+ actor with a completion-safety reserve and reports a preregistered comparison against its fixed parent.

1. Introduction

  • Simulator interaction can adapt an offline policy, but differences in target dynamics and event duration make transfer and service completion central concerns.Bus holding is asynchronous, and lower passenger-time objectives alone do not ensure journeys finish before the evaluation horizon.Source: §1 · pp. 2–3
  • The study separates offline learning, adaptive development, and fresh formal confirmation, then compares a reserve-augmented candidate with its identical direct parent.The candidate and parent share frozen actors and differ only in the final deterministic completion-safety transform.Source: §1 · pp. 3–4
  • 92.12 s per departed passenger lower generalized passenger time (3.71%), completion rose by 0.00466, and unfinished passengers fell by 77.22 on average.The primary endpoint improved and completion noninferiority passed in all ten independent blocks.Source: §1 · pp. 3–4

2.1. Offline and offline-to-online reinforcement learning

  • Prior work already combines offline and online learning for ride-pooling with public transit (Hu et al., 2025); this paper makes no broad transport-RL novelty or universal algorithm-ranking claim.Its focus is asynchronous fixed-route holding under a target-simulator dynamics gap and an explicit completion boundary.Source: §2.1 · p. 5

2.2. Transfer across dynamics

  • The event-duration classifier serves as a domain-shift diagnostic, while the final training profile uses unit simulator weights and formal testing isolates the reserve on a fixed actor.The classifier is not the confirmed mechanism.Source: §2.2 · p. 5

2.3. Bus holding and service completion

  • Asynchronous bus control is not itself new; the study distinguishes its evaluation through immutable target replay, separate-simulator interaction, and fresh target-simulator confirmation.A common passenger ledger and paired reserve comparison separate the work from a generic claim that a new learner beats existing holding algorithms.Source: §2.3 · p. 6
  • The reserve targets lightly loaded tail buses without followers when a leader exists and demand remains, suppressing hold in those states.The paired parent comparison changes only this reserve, testing its effect without retraining the actor or reselecting a checkpoint.Source: §2.3 · p. 6

3. Problem formulation

The formulation casts asynchronous multi-line bus holding as a finite-horizon SMDP with serialized event decisions and exact generalized passenger-time accounting. It evaluates policies across a target SUMO environment and a calibrated online simulator with shared transit contracts but different dynamics.

3.1. Chronological decision process

  • Bus-stop arrivals trigger asynchronous decisions, and simultaneous events are serialized in a deterministic batch of up to 16 events.Each token observes the fixed batch and previously selected actions; the environment advances after the final action.Source: §3.1 · p. 7
  • The centralized canonical state is 229-dimensional, packing normalized local event features, active-slot flags, prior actions, and a shared batch prefix.The raw packed state is 245-dimensional; the representation’s sufficiency as a Markov state is a modeling assumption.Source: §3.1 · p. 7
  • A single scalar action maps to physical hold duration, with exactly −1 denoting zero hold and all actions in (−1, 1] denoting positive holds.The operational duration is capped, and simultaneous intermediate tokens have zero elapsed time while the final token carries the interval duration.Source: §3.1 · p. 8

3.2. Exact generalized passenger-time reward

  • Generalized passenger-time cost integrates waiting and in-vehicle passenger counts over each physical interval, and the chronological ledger charges system cost once.Intermediate serialized tokens have zero duration and zero reward, preventing duplicated cost.Source: §3.2 · pp. 8–9
  • The study evaluates full-horizon generalized passenger time per departed passenger, while separately tracking completion and unfinished passengers.Time accrued by unfinished journeys remains in the primary endpoint; the denominator avoids conditioning on policy-dependent completion.Source: §3.2 · p. 9

3.3. Target and online-simulator environments

  • The target is a high-fidelity SUMO environment, while the calibrated event-driven simulator shares transit contracts but uses calibrated runtimes and dwell mechanics instead of a microscopic road model.The simulator was accepted after a preregistered 30-cell fidelity audit against matched SUMO cells.Source: §3.3 · pp. 9–10

3.4. Offline-to-online learning objective

  • Offline-to-online learning uses immutable replay from 240 complete episodes across eight behavior policies and 30 registered environment cells, then permits interaction only in the online simulator.No target-environment transition is queried online; the target-return objective is evaluated after training, and the completion measure remains a separate evaluation constraint.Source: §3.4 · p. 10

3.3. Target and online-simulator environments

  • The target and online simulator share service topology, stop sequences, timetables, capacities, and passenger journeys, but differ in traffic and event-propagation mechanisms.Policy control covers all 12 directional services.Source: Figure 1 · p. 11

4. Method

The method adapts an H2O+ actor to bus holding, continues learning in a calibrated simulator, and applies deterministic physical-action transforms at deployment. The final candidate differs from its direct parent only by a bounded completion-safety reserve, while density-ratio estimates remain diagnostic rather than part of training.

4. Method

  • The deployed controller combines a frozen transit-adapted H2O+ actor with a deterministic physical-action transform, with no target-SUMO parameter updates.The transform calibrates the actor’s holding proposal and adds a bounded, state-conditioned completion reserve.Source: §4 · p. 12

4.1. Structured hurdle actor

  • The structured hurdle actor represents exact no-hold as a point mass and positive holds with a continuous logistic-normal branch.At deterministic evaluation, it selects the positive-branch median when the positive-hold probability is at least 0.5; otherwise it returns no hold.Source: §4.1 · p. 12

4.2. Offline initialization and simulator continuation

  • Five actors began with 50,000 IQL updates on immutable target replay, then continued in the calibrated simulator after a 5,000-event frozen-policy warmup.Critic minibatches mix target and simulator data 50/50; only the 25,000-event endpoint was exported, without checkpoint selection by target-SUMO behavior.Source: §4.2 · pp. 13–14

4.3. Dynamics-ratio diagnostic

  • The joint/marginal domain classifier estimates a logged-target-to-simulator dynamics ratio, but its output remained diagnostic and did not weight the final training critic.The estimate concerns the logged runtime rather than directly estimating the formal-evaluation kernel, and requires generalization to evaluated states.Source: §4.3 · p. 14

4.4. Physical action calibration

  • Physical calibration combines the actor proposal with a clipped headway-based hold, then applies a reserve that sets the hold to 60 seconds in eligible low-load tail-bus states.The candidate and direct parent share the actor and inherited transforms and differ only through the final reserve transform.Source: §4.4 · pp. 15–16

4.5. Development and confirmation separation

  • The reserve transform was selected using adaptive development rounds, but the candidate, direct parent, formal blocks, endpoint, margin, and tests were frozen before formal outcomes were accessed.The formal stage introduced no new training run and allowed no candidate replacement.Source: §4.5 · p. 16

5. Experiments and Results

The experiments distinguish offline-data and simulator acceptance, adaptive development, and formal confirmation; only the final layer supports the confirmatory effect claim.

5.1. Target benchmark and offline data

  • The offline benchmark contains 240 complete episodes across eight behavior policies, and the online simulator passed preregistered zero-policy and intervention-response audits.The audits used matched transit contracts and evaluated 30 cells.Source: §5.1 · pp. 16–17

5.2. Formal confirmation matrix

  • Formal confirmation compared the fixed reserve candidate with its direct parent across five seeds, three demand levels, and ten fresh environment blocks.The evaluation comprised 1,350 SUMO episodes; seven additional controls supplied lineage context but did not expand the confirmatory claim.Source: §5.2 · pp. 17–18

5.3. Endpoints and preregistered inference

  • The preregistered intersection–union test required primary superiority and completion noninferiority, with at least nine of ten block wins for each test.Completion noninferiority used a 0.003 margin.Source: §5.3 · pp. 18–19

5.4. Confirmed candidate–parent effect

  • The reserve-augmented candidate reduced generalized passenger time by 92.12 s per departed passenger, increased completion by 0.00466, and passed both confirmatory tests in all ten blocks.Unfinished passengers decreased by 77.22 on average.Source: Table 2 · p. 19, §5.4 · p. 19

5.5. Supportive lineage controls

  • The candidate had lower generalized passenger time and higher completion than every listed control, including online-only SAC, but these comparisons were not the confirmatory claim.Its candidate-minus-parent primary effect was favorable in all ten blocks; Table 3 reports demand-stratum differences.Source: §5.5 · pp. 19–20, Table 4 · p. 20, Table 3 · p. 20

5.6. Data and environment diagnostics

  • Only 30 exogenous cells underlie the 3.28 million transitions, 80.2246% of offline actions were zero, and the ratio diagnostic generalized poorly.The 60-s reserve had sparse offline action support and is a bounded overlay; collection and formal evaluation also used different SUMO versions without a paired sensitivity test.Source: §5.6 · pp. 20–21

5.7. Execution and reproducibility

  • An independent implementation recomputed the result count, means, block wins, sign-test value, and preregistered gates from the frozen analysis.The synchronized artifact contains results, manifests, logs, and verification metadata, but not checkpoints, datasets, or optimizer state.Source: §5.7 · p. 21

6. Discussion

The reserve improved the fixed policy’s registered simulator endpoint, while the discussion limits that conclusion to the tested scenario family. Development results alone were insufficient because the process was adaptive and reused cells.

6.1. What the formal result establishes

  • Across ten fresh blocks, the reserve reduced generalized passenger time and improved completion versus its direct parent, with the same direction at every demand level and training seed.The proposed tail-service explanation is mechanistic rather than causal at the individual-passenger level because guard activations were not randomized within trajectories.Source: §6.1 · pp. 21–22
  • The benefit was smallest at demand 0.75 and largest at 1.25, where a tail bus without a follower has greater passenger consequences.The reserve can add local in-vehicle time; fewer unfinished passengers are consistent with preventing later demand from going unserved before the horizon.Source: §6.1 · p. 22

6.2. Why development evidence alone was insufficient

  • The development process reused nine cells across adaptive rounds, and earlier variants’ looser completion margins did not establish passage of the final 0.003 boundary.For these reasons, development improvements were not treated as a formal effect.Source: §6.2 · p. 22
  • Formal confirmation froze the candidate, parent, cells, endpoints, margin, and tests before outcomes, using cells disjoint from training and development.Candidate replacement was forbidden; this design supports the bounded confirmed claim.Source: §6.2 · p. 22

6.3. Limitations

  • Millions of correlated SMDP tokens came from only 30 environment cells, and the centralized state’s Markov sufficiency is an assumption, not a proved property.The five seeds represent optimizer variation on fixed data, not five new datasets; future collection should prioritize independent days, demand realizations, and disruptions.Source: §6.3 · pp. 22–23
  • Positive-hold support is sparse, so the 60-s completion reserve is a transparent rule rather than evidence that the offline actor learned reliable values in those states.Training uses total accrued cost while evaluation normalizes by departed passengers; within-cell ordering agrees, but cross-demand weighting can differ.Source: §6.3 · p. 23
  • Runtime mismatch and reduced-simulator approximations constrain transfer claims; the study does not test changed traveler preferences, fares, dispatch rules, or fleet composition.Formal evaluation used SUMO/libsumo 1.25.0, while training data used SUMO 1.27.1; learned-policy state distributions are not fully covered by the simulator audits.Source: §6.3 · p. 23
  • Ten-block sign tests show directional consistency, not a population-average confidence interval, and the endpoint-only study does not establish learning speed or compute efficiency.The reserve is not a complete deployment safety system; live evaluation requires authorization, operational safeguards, and held-out operational records.Source: §6.3 · pp. 23–24

7. Conclusion

The fixed reserve-augmented controller improved the registered endpoint relative to its direct parent in fresh SUMO confirmation. The conclusion is bounded to the evaluated scenario family, with operational-record validation left for future work.

7. Conclusion

  • Relative to the identical direct parent, the fixed candidate reduced generalized passenger time by 92.12 s per departed passenger, increased completion by 0.00466, and reduced unfinished passengers by 77.22.Paired effects were confirmed in 1,350 fresh formal SUMO episodes, with superiority and completion noninferiority passing in all ten independent blocks.Source: §7 · p. 24
  • The completion-aware reserve improves the fixed parent at the registered endpoint in the evaluated SUMO scenario family.The preregistered intersection–union test passed in all ten independent blocks.Source: §7 · p. 24
  • Future work should align runtimes, expand independent environment cells and positive-hold support, incorporate completion into learning, and validate against held-out operational records.Source: §7 · p. 25

Data and Code Availability

The reproducibility package includes frozen source, manifests, formal results, registered analysis, and independent verification, while some materials and sharing terms remain unsettled.

Data and Code Availability

  • The archive includes frozen training and evaluation source, manifests, 1,350 formal records, registered analysis, and independent verification, but excludes duplicated checkpoints and workspaces.Repository details and final sharing terms for operating data and derived SUMO inputs were pending confirmation before submission.Source: Data and Code Availability · p. 25

Declaration of Competing Interest

The competing-interest declaration remained to be confirmed by all authors before submission.

Declaration of Competing Interest

Funding

Funding and contributor details are withheld during double-anonymous review; the authors disclose AI-assisted preparation and review.

Funding

  • Funding information and author-identifying acknowledgements are withheld pending author confirmation before submission.Source: Funding · p. 25

Declaration of Generative Artificial Intelligence (AI) and AI-Assisted Technologies in the Manuscript Preparation Process

  • AI-assisted tools supported drafting, editing, reference verification, consistency checks, and code and log inspection; the authors reviewed the work and accept responsibility.No generative-AI-created figure, image, or artwork is included.Source: Declaration of Generative Artificial… · p. 25

Contributor Roles Taxonomy (CRediT) Authorship Contribution Statement

Appendix A. Accounting and action-distribution properties

The appendix supports the passenger-time accounting with a nonduplicating chronological ledger and characterizes the paper’s hurdle policy and deterministic action.

Define the pre-control cost

Appendix A.1. Chronological passenger-time accounting

  • Chronological batch intervals partition the controlled horizon, so summing their passenger-time costs yields full-horizon accounting without overlap.The pre-control cost and departed-passenger count are policy-invariant for a fixed exogenous realization under the registered demand contract.Source: Define the pre-control cost · p. 27, Appendix A.1 · p. 27

Appendix A.2. Hurdle density and deterministic action

  • The hurdle policy is a normalized probability measure whose positive-branch median is 2 sigmoid(µϕ(s)) −1, the paper’s deterministic action.The appendix notes that a nonlinear transformation does not generally preserve the conditional mean.Source: Appendix A.2 · p. 28

Appendix B. Implementation and provenance details

The appendix documents the structured actor architecture, frozen training choices, and why the domain-ratio classifier did not provide a usable correction.

Appendix B.1. Structured network implementation

  • A shared slot encoder and five pooled views are fused with the batch prefix into 256 state features for the actor and critic architecture.The actor produces a gate logit, latent normal mean, and log standard deviation; the critic has two heads after a shared structured encoder.Source: Appendix B.1 · pp. 28–29

Appendix B.2. Frozen training values

  • Training retains rewards in physical generalized passenger-seconds, applies numerical conditioning only inside the trainer, and continues sealed terminal-quarter branches without restarting chronology or optimizer state.Only the preregistered quarter-rate branch supplied the five final candidate actors.Source: Appendix B.2 · p. 29

Appendix B.3. Ratio diagnostic

  • The domain-ratio classifier did not provide a usable generalizing correction, and unit ratio weights kept its diagnostic failure from directly perturbing critic updates.Validation was split by complete environment blocks for offline data and complete trajectories for simulator data.Source: Appendix B.3 · p. 29

Appendix B.4. Dataset integrity and support

The appendix audits replay integrity and support, reports seed-wise formal-effect direction, and describes formal-result verification procedures.

Appendix B.4. Dataset integrity and support

  • The replay contains 3,284,718 rows, with exact zero hold in 80.2246% of actions; 3.91% of audited rows meet the final low-load guard’s physical-state conditions.Among those guard rows, behavior holds of at least 20, 30, and 40 s occur in 5.81%, 3.69%, and 2.17%, respectively.Source: Appendix B.4 · pp. 29, 31

Appendix B.5. Seed-specific formal effects

Loading 2609.39868v1…