Large Language Model-Enhanced Reinforcement Learning for Generic Bus Holding Control Strategies

1. Introduction

• Model-based holding methods require accurate bus propagation and passenger-arrival patterns, which are difficult to obtain amid fluctuating real-world traffic dynamics.This motivates growing interest in model-free approaches.Source: §1 · p. 1
• Existing RL bus-holding strategies face three challenges: designing dense rewards for sparse goals, generalizing across systems, and maintaining stable performance despite inefficient LLM outputs.Sparse objectives often require substantial expert input and manual adjustment to create stepwise learning signals.Source: §1 · p. 3
• An LLM-enhanced RL paradigm uses modules for reward initialization, modification, performance analysis, and refinement to iteratively improve rewards with performance feedback.Its iteration rules are intended to support stable improvements and reliable agent performance.Source: §1 · p. 3
• The workflow aims to reduce time-consuming manual reward tuning and diagnosis by having an integrated LLM refine reward functions using training and test feedback.A seemingly reasonable reward may still perform poorly because RL behavior is difficult to interpret and can be random.Source: §1 · p. 3
• The proposed strategy shows promising performance and robustness against vanilla RL, LLM-based, and model-based feedback controllers in synthetic and real-world bus systems.The study also verifies adaptiveness across generic scenarios, including single-line and multi-line systems.Source: §1 · p. 3

2.1. Bus operation control

• Bus holding includes fixed and dynamic control points and speed control, while existing strategies remain scenario-specific, motivating generic methods for single- and multi-line systems.Related work also studies holding at traffic signals and asynchronous control actions.Source: §2.1 · p. 4

2.2. Reward design of RL

• Reward design often requires explicit behavior definitions or expert demonstrations, whereas LLMs can generate and iteratively refine rewards using RL-agent performance as feedback.The proposed feedback loop aims to make reward design more autonomous and efficient.Source: §2.2 · pp. 4–5

2.3. LLM-based applications in transportation

• LLM controllers face latency and interpretability concerns; LLM–RL integration supports offline reward generation, but response reliability remains under-addressed, motivating filtering rules.The proposed rule filters inefficient LLM outputs to support reliable and stable strategy formulation.Source: §2.3 · p. 5

3.1. Problem statement

• Holding must balance headway regularity against onboard travel time, while passenger waiting and travel-time rewards arrive late and can be absent at stops.The study addresses asynchronous decisions through action-step discretization and knowledge sharing among agents.Source: §3.1 · pp. 5–6

3.2. Design of the RL agent

• Each stop agent observes space headways, onboard passenger count, and current holding duration, and chooses whether to hold for the next 5-second step.The state representation accommodates multi-line systems by including headways to buses across lines; branch stops not served by other lines assign those headways zero.Source: §3.2 · pp. 6–7
• The study uses one LLM-generated reward function across homogeneous agents to test generic control across heterogeneous bus-stop scenarios.The authors suggest a successful generic reward may improve strategy adaptiveness and transferability.Source: §3.2 · p. 7

3.3. Modeling dynamics of bus propagation

• The simulation tracks bus movement, boarding and alighting, passenger waiting and travel times, and dwell time; between stops, speed is derived from observed travel time.Boarding is constrained by bus capacity, and dwell time is the maximum of boarding and alighting times.Source: §3.3 · pp. 7–8

3.4. Action-value function approximation

• A DQN approximates the agents’ action-value function, with learning updates based on uniformly sampled experience from a shared replay memory.Agents in the same bus system share one set of Q-network parameters.Source: §3.4 · pp. 8–9

4.1. Overview of LLM-enhanced RL paradigm

• The four-module paradigm initializes executable rewards, trains and tests agents, analyzes performance, and modifies rewards using the resulting suggestions.The workflow ends at the iteration limit, performance convergence, or the expected test performance.Source: §4.1 · p. 10

4.2. Reward initializer and modifier

• Initializer and modifier prompts supply task, agent, reward-input, and environment information; the modifier also receives the current reward and improvement suggestions.Output instructions request executable code and explanations of its design to support interpretability.Source: §4.2 · pp. 10–11

4.3. Agent performance analyzer

• The analyzer uses training reward evolution, agent decision trajectories, and test results to assess performance and suggest reward improvements.Its analysis and reward-code generation are separated into distinct modules.Source: §4.3 · pp. 11–12

4.4. Reward refiner

• The reward modifier and performance analyzer form an iterative loop with the RL agents and environment, while the refiner filters reward functions when test performance deteriorates.Refinement continues until the test results meet the criterion; its slack can be adjusted to impose increasingly stringent requirements.Source: §4.4 · p. 12

4.5. Implementation details

• An example modification adds a penalty for increased headway variation and a reward for decreased headway variance, implementing analyzer suggestions in the reward function.The algorithm repeats reward updates, agent training, testing, and analysis across iterations.Source: §4.5 · pp. 12–13

• The evaluation covers a synthetic single-line system, a real-world two-line system with a partially shared corridor, and a real-world large-scale system with six lines.Source: §5 · p. 13
• RL agents are homogeneous across all scenarios and share a generic reward function.Source: §5 · p. 13
• The study reports that its code and data are available online.Source: §5 · p. 13

• Agents train for 70 episodes of 200 epochs each; action steps last 5 seconds and holding is capped at 90 seconds.Case 1 simulations last 4 hours per episode; case 2 training lasts 2 hours and testing lasts 4 hours.Source: §5.1 · p. 13
• Vanilla RL baselines use expert-designed rewards inspired by Wang and Sun (2020) and Yu et al. (2023a), providing a comparison with manually designed rewards.Source: §5.1 · p. 14
• Other baselines include an LLM that directly chooses holding actions, physics-based feedback controllers, and optimization-based controllers targeting passenger waiting time.The optimization approaches include robust and stochastic variants.Source: §5.1 · p. 15
• Evaluation measures headway standard deviation, average passenger travel and waiting times, and average bus holding time.Source: §5.1 · p. 15

5.2.1. Scenario description

• The synthetic scenario uses six buses and eight evenly spaced stops on a 10.66-km loop, with passenger arrival rates varying over a four-hour simulation.It assumes uniform speed and travel time, excludes randomness in bus travel, and counts boarding time—but not alighting time or capacity—in dwell time.Source: §5.2.1 · pp. 15–16, Figure 5 · p. 16

5.2.2. Comparisons with baselines

• Robust and stochastic optimization underperform the RL approaches in this test, where online interaction supports adaptation to random passenger arrivals and real-time conditions.The optimization methods are static, and particle swarm optimization may converge to a local rather than global optimum in this high-dimensional problem.Source: §5.2.2 · p. 17
• 3.73% lower average travel time and 12.29% lower average waiting time than the best vanilla RL strategy; warm-start LLM-enhanced RL also has the lowest SD of time headways.The comparison is reported across 30 random seeds.Source: §5.2.2 · p. 16, Table 1 · p. 17
• Feedback control and LLM-enhanced RL perform similarly in average waiting time and headway SD and outperform other strategies at each stop; LLM-enhanced RL reduces headway SD by 59.50% versus RL-local.Feedback control’s corresponding reduction versus RL-local is 53.48%.Source: §5.2.2 · p. 18

5.2.3. Sensitiveness on LLMs

• GPT-4 and GPT-4o improve more efficiently early and remain more stable later than the other tested LLMs; Claude-Opus does not meet the refiner criterion within ten iterations.Only nine Claude-Opus iterations are shown because it did not meet the criterion within a reasonable number of explorations in the tenth iteration.Source: §5.2.3 · p. 19

5.2.4. Comparisons with existing LLM-enhanced RL methods

• 5.59% lower average travel time and 5.30% lower average waiting time than the best existing LLM-enhanced RL baseline; the proposed method improves steadily across iterations.The critic variant has the lowest SD of time headways, but the proposed method has the lowest average travel and waiting times.Source: §5.2.4 · pp. 19–20, Table 3 · p. 21, Figure 10 · p. 22

5.2.5. Ablation studies

• Ablating the analyzer or reward refiner raises average travel time by 15.97% or 13.56%, respectively, and both ablations show large performance fluctuations.Warm-start performance is strongest overall; cold start performs similarly. The initializer does not affect final performance when an expert reward is available, but is needed without one.Source: §5.2.5 · p. 20, Table 4 · p. 22

5.3.1. Scenario description

• The two Beijing lines each have 27 stops and eight shared stops; buses depart every five minutes, and the test uses weekday morning-rush-hour passenger data.Simulation varies passenger arrivals and bus travel times, allows overtaking, and models bus capacity and dwell time.Source: §5.3.1 · pp. 24–25, §5.3 · p. 24

5.3.2. Comparisons with baselines

• The proposed paradigm achieves lower average travel time with fewer holding interventions than feedback control, which balances headways more evenly.The proposed reward balances headway consistency and holding delay; its trajectories show more bunching than feedback control but less than no-holding, and average travel time is slightly lower than feedback control.Source: §5.3.2 · p. 25
• Only 1.07% of passengers cannot complete their trip under the proposed paradigm, compared with 12.03% under no-holding.The paper attributes incomplete trips in the no-holding scenario to large headways and bus capacity limits.Source: §5.3.2 · p. 26

5.3.3. Sensitive analysis

• In most tested shared-passenger settings, “GPT-4 + RL” outperforms other strategies in average travel time, except when shared passengers make up 5%.The tested proportion ranges from 0% to 25%; total passenger numbers remain constant, and average travel time first decreases up to 15% shared passengers, then rises.Source: §5.3.3 · p. 26
• Across demand levels from 70% to 130% of status quo, the trained agents beat feedback control on average travel time; “GPT-4 + RL” beats all strategies from 90% to 110%.Average travel and waiting times rise as demand increases; broader demand training may be needed for optimal performance across a wider range.Source: §5.3.3 · p. 27

5.3.2. Comparisons with baselines

• Across 30 random seeds in Case 2, “GPT-4 + RL” achieved minimum average travel time for each bus line and overall, reducing overall average travel time by 3.14% versus no-holding.Average holding times were 6.52 seconds on Line 1 and 6.61 seconds on Line 52, 43.45% and 36.81% lower than feedback control, respectively.Source: §5.3.2 · p. 25, Table 5 · p. 28

5.4.1. Scenario description

• The Case 2 reward function and trained agents are tested unchanged in a real-world system of six bus lines that share some stops.The morning-rush-hour passenger demand is scaled up to simulate busy lines where holding is necessary.Source: §5.4 · p. 29, §5.4.1 · p. 29, Figure 18 · p. 29

5.4.2. Comparisons with baselines

• The proposed paradigm gives the lowest average travel time on most lines except Line 55, while feedback control has the lowest waiting time for most lines.Feedback control's smaller headway variation does not correspond to the minimum average travel time.Source: §5.4.2 · p. 29

5.4.3. Adaptiveness and generalization analysis

• A generic reward across stop configurations and a self-adjusting workflow support zero-shot use in unseen control scenarios with few adjustment iterations.The reward refiner enables self-supervision; the performance analyzer and reward modifier enable self-improvement.Source: §5.4.3 · p. 30
• In complex systems, balancing headways alone is insufficient to achieve large reductions in total passenger travel time.The paper contrasts this with Case 1, where balanced headways yield similarly low average travel times.Source: §5.4.3 · pp. 30–31

5.4.2. Comparisons with baselines

• The paradigm reduces overall average travel time by 13.41 seconds versus no-holding, 4.62 seconds versus feedback control, and 29.07 seconds versus RL-local.Each control method was evaluated using 30 random seeds.Source: §5.4.2 · p. 29, Table 6 · p. 32

5.5. Generic tests

• The 30 generated systems vary from 2 to 10 bus lines, 16 to 25 stops per line, and 100 to 1,100 departing passengers per stop over four hours.The same reward function is used as in Case Studies 2 and 3.Source: §5.5 · p. 33
• The paradigm outperforms feedback control in 90% of 30 scenarios and reduces average travel time by 4.51% versus no-holding on average.It reduces average travel time by 29.82 seconds versus feedback control overall, although feedback control slightly outperforms it in scenarios 3, 9, and 21.Source: §5.5 · p. 33, Table 7 · p. 34

6. Conclusions

• The paradigm automates reward initialization and improvement through multiple LLM modules, with performance checks using feedback from the RL environment.The workflow is intended to filter ineffective rewards and support stable improvement.Source: §6 · p. 35
• Across synthetic and real-world tests, the paradigm outperforms vanilla RL, an LLM-based controller, feedback controllers, and optimization-based controllers on average travel time.The paper also reports robustness across passenger compositions and demand levels, and effective performance improvement and convergence within a manageable number of iterations, such as 10.Source: §6 · p. 35
• The study focuses on LLM design and planning; fine-tuning for reward design and better explanation of generated rewards remain future directions.The paper also identifies other smart mobility applications, such as traffic signal control and variable speed limits, as possible extensions.Source: §6 · p. 35

• Yu is credited with methodology, conceptualization, visualization, and original drafting and review and editing.Source: CRediT authorship contribution statement · p. 35
• Wang is credited with writing review and editing.Source: CRediT authorship contribution statement · p. 35
• Ma is credited with conceptualization, supervision, writing review and editing, and funding acquisition.Source: CRediT authorship contribution statement · p. 35

Appendix A.1. Prompts of reward initializer

• The prompts define the objective as balancing time headways within and across two bus lines while minimizing passenger travel time, since excessive holding causes delays.Balancing time headways is treated as a proxy for balancing space headways.Source: Appendix A.1 · p. 41
• The initializer prompt specifies a binary hold-or-release action in 5-second steps and a state containing neighboring-bus headways, onboard passengers, and elapsed holding time.The state also represents headways to buses on another line at shared stops.Source: Appendix A.1 · p. 41
• Because the same reward formulation must serve stops on one or both lines, the initializer is instructed to make it generic and directly usable without tuned variables.The prompt requests Python 3.8.16 code with comments and defined dependencies.Source: Appendix A.1 · p. 42

Appendix A.2. Prompts of reward modifier

• The modifier prompt supplies the current reward and performance analysis, then asks the LLM to revise it, rebalance rewards and penalties, or replace it entirely.The changes are guided by training and test results and suggestions for improvement.Source: Appendix A.2 · pp. 42–43
• The modifier is instructed to return directly usable, checked Python code, without excessively large or small reward values or variables requiring tuning.The prompt also requires concise reasoning comments and defined imports or helper functions.Source: Appendix A.2 · p. 44

Appendix A.3. Prompts of analyer

• The analyzer prompt asks for possible causes of inefficiency and reward-improvement suggestions based on training-reward trajectories and test-time actions, rewards, and states.It frames inefficiency assessment around both training and test performance.Source: Appendix A.3 · p. 44
• The analyzer prompt defines a binary action at 5-second steps and a six-element state covering same-line and other-line headways, onboard passengers, and elapsed dwell holding time.Other-line headways are zero at stops that are not shared, and passenger counts remain unchanged during holding because boarding and alighting are disallowed.Source: Appendix A.3 · p. 45
• The analyzer’s simulation example uses two 27-stop lines with eight shared stops, a 90-second maximum hold, and exogenous Poisson passenger arrivals.Passengers cannot board or alight during holding.Source: Appendix A.3 · p. 45
• The analyzer receives training rewards and test trajectories, states, actions, rewards, and outcome metrics, then is asked for concise convergence and test analysis with reward-improvement suggestions.The provided test metrics include headway standard deviation, average passenger travel and waiting times, and average bus holding time.Source: Appendix A.3 · pp. 45–46

Appendix A.4. Prompts of reward refiner

• The reward refiner uses prior reward analyses and test feedback, while treating a modification that performed worse than its predecessor as a failure to avoid.It is asked to improve the previous function for a generic reward shared across stops serving one or two bus lines.Source: Appendix A.4 · pp. 47–48
• The refiner is instructed to modify or replace the reward, balance its values, avoid extreme rewards and tunable variables, and return checked Python 3.8.16 code only.The requested function uses current_state, action, and next_state and should include concise reasoning and implementation comments.Source: Appendix A.4 · p. 48

Appendix A.5. Prompts of LLM-based controller

• The direct controller prompt asks whether to hold each arriving bus and, if so, for how long within a 0–90-second range.Its prompt uses only three state parameters in case study 1: forward and backward space headways and onboard passenger count.Source: Appendix A.5 · pp. 48–49
• The prompt states that time and space headways are strongly correlated and consistently perform together, so balancing time headways can be treated as balancing space headways.Source: Appendix A.5 · p. 49
• The prompt represents input state as [s0, s1, s2]: forward and backward space headways in meters and the current bus’s onboard passenger count.It defines time headway as the interval between successive buses’ arrivals at a stop, space headway as the distance between successive buses, and passenger travel time as waiting plus onboard time.Source: Appendix A.5 · p. 49
• The prompt specifies a cyclic loop corridor with 8 stops and 6 buses, 10,656 m long, with 1,332 m between stops and a 90-second maximum hold.Buses travel at 5.55 m/s, take 4 minutes between adjacent stops excluding dwell and holding, and have a boarding time of 3.0 s/pax.Source: Appendix A.5 · p. 49
• Passenger arrivals follow a Poisson process and are exogenous to the controller; passengers cannot board or alight while holding control is applied.The 8 stop positions from the origin are 666, 1998, 3330, 4662, 5994, 7326, 8658, and 9990 meters.Source: Appendix A.5 · p. 49
• The prompt asks the controller first to choose whether to hold; if so, it must then select a holding duration from 0 to 90 seconds.A no-hold decision is encoded as 0 in step 1 and -1 in step 2; a hold decision is encoded as 1 followed by the duration.Source: Appendix A.5 · p. 49
• The controller is instructed to use the task description and input state, fill separate holding-decision and holding-duration fields, and output no analysis.The prompt requires strict adherence to the output format.Source: Appendix A.5 · p. 49

Appendix B. Final reward function

• The final reward combines headway-balance penalties, holding penalties, rewards for reduced headway variability, passenger holding-delay penalties, and an added penalty for long holding.The components are aggregated as a multi-objective reward balancing operational goals.Source: Appendix B · p. 50
• Headway-balance penalties use absolute differences between the bus’s forward and backward space headways and the ideal space headway.The component aims to bring space headways closer to the ideal.Source: Appendix B · p. 50
• The holding penalty is the square of the increase in holding time, discouraging excessive holding delays.Source: Appendix B · p. 50
• Headway-variability reduction earns reward when the next-state standard deviation across four headways is below the current-state value.The code calculates standard deviations over the four listed headways in each state and assigns a reduction reward only when the next value is lower.Source: Appendix B · pp. 50–51
• The passenger holding-delay penalty sums onboard passengers’ holding times and normalizes the total by average bus capacity.The implementation uses passengers_current divided by 50 as the normalization factor.Source: Appendix B · pp. 50–51
• Holding beyond 60 seconds triggers a 1.5-times holding-penalty multiplier to target prolonged holding.The reward subtracts the headway penalties, dynamic penalty, and passenger waiting-time penalty.Source: Appendix B · pp. 50–51