Offline Reinforcement Learning for Wind Farm Control: A Wind Tunnel Study under Dynamic Wind Directions

I. INTRODUCTION

• Greedy control maximizes individual turbine output by facing the incoming wind but ignores wake coupling, so it is generally suboptimal for farm-level power.Wake effects arise when upstream turbines reduce wind speed reaching downstream turbines.Source: §I · p. 1
• Model-based strategies depend on model accuracy, while direct model-free optimization needs long evaluation windows and must re-converge when inflow changes.These direct optimization methods provide point solutions for quasi-static conditions rather than feedback policies.Source: §I · p. 1
• Prior RL wind-farm studies largely used online training and often assumed a specific wind direction, whereas direction changes alter wake patterns and complicate learning.Online training requires frequent simulator interactions and can face a sim-to-real gap.Source: §I · p. 2
• Offline RL learns farm-level yaw control under dynamic wind directions from logged operation, requiring no wake model or simulator interaction for training.The logged data need sufficient variation in yaw settings but may come from any deployed controller, commissioning perturbations, or a mixture.Source: §I · p. 2
• MTD3-BC extends TD3-BC with action consistency regularization to suppress yaw chattering, alongside wind-direction conditioning and dataset-level state normalization.The consistency term accounts for the slow response of yaw actuators.Source: §I · p. 2
• Wind-tunnel tests matched LUT and online PPO performance, marginally exceeded the training dataset’s best behavior policy, and used approximately 5% of PPO’s training cost.Three deploy-and-retrain iterations also demonstrated the offline refinement workflow’s practicality.Source: §I · p. 2

II. PROBLEM FORMULATION

• Yaw-angle control changes total farm power, although the relationship between turbine power and operating conditions is too complex to derive accurately.The proposed RL method therefore leaves the governing function unknown and uses a data-driven, model-free approach.Source: §II · p. 3
• The control objective balances farm power generation against structural loads that large yaw misalignments may impose on turbines.The reward uses a user-defined weight k between power boosting and yaw-misalignment penalization.Source: §II · p. 3
• The objective is to maximize expected cumulative discounted reward, with a discount factor between zero and one and an upper bound on yaw angles.Source: §II · p. 3
• Changing wind direction changes turbine wake-interaction patterns, requiring the controller to adapt and making learning more difficult.Greedy yaw control may be suboptimal at farm level because it ignores wake coupling.Source: §II · p. 3

A. Offline Dataset

• Offline RL learns from a fixed dataset of single-step state, action, reward, and next-state transitions without interacting directly with the environment.The wind-farm dataset contains transitions collected under different control algorithms.Source: A. Offline Dataset · p. 3
• The state combines internal turbine operating conditions, including yaw angles, with external conditions such as measured wind direction, turbine-level wind speed, and turbulence intensity.Source: A. Offline Dataset · p. 3
• Each action represents the yaw references that the turbines should follow.Source: A. Offline Dataset · p. 3

B. Modified TD3-BC Algorithm

• TD3-BC adds behavior-cloning regularization to TD3’s policy update, favoring actions represented in the offline dataset and mitigating extrapolation error.The regularizer steers the policy toward behavior observed in the data.Source: B. Modified TD3-BC Algorithm · p. 3
• The TD3-style architecture uses two critics and target networks, with soft replacement to help stabilize learning.Critics are trained with temporal-difference error-based losses.Source: B. Modified TD3-BC Algorithm · pp. 3–4
• Small state variations can produce large action changes, while slow yaw actuators may not track rapid yaw references, degrading control performance.These sensitivities motivate adding action consistency to the policy objective.Source: B. Modified TD3-BC Algorithm · p. 4
• The policy objective combines Q-value maximization, behavior cloning, and an action-consistency penalty on actions generated for similar inputs.The consistency term is intended to mitigate chattering and promote smoother control actions.Source: B. Modified TD3-BC Algorithm · p. 4
• Training normalizes dataset states once, adds clipped noise to target-policy actions, and updates actor and target networks once per d critic updates.The dataset remains fixed throughout training.Source: B. Modified TD3-BC Algorithm · p. 4

A. Experimental Setting

• Three scaled turbines are spaced 5D apart on a 13-m turntable, with WT1 upstream, WT2 centered, and WT3 downstream.Each turbine has a 1.1-m rotor diameter and a hub height of 0.825 m.Source: A. Experimental Setting · pp. 4–5, Figure 2 · p. 5
• The test used 6% turbulence intensity, a 5.35 m/s inflow, and a maximum yaw misalignment of 30°.The rated wind speed was 5.7 m/s; torque control operated below rating and blade-pitch control above it.Source: A. Experimental Setting · p. 5
• A five-day, 1-Hz field wind-direction record was compressed into 90 minutes, with directional variation within ±15° and an 80× time speed-up.The rotating turntable simulates low-frequency, large-scale direction changes.Source: A. Experimental Setting · p. 5
• The 90-minute record was divided into nine 610-second profiles, with the first 10 seconds of each excluded and sensors zeroed before and after each segment.Source: A. Experimental Setting · p. 5

B. Dataset Creation, Network Architecture, and Hyperparameter Setting

• The state concatenates three turbine yaw angles relative to wind with measured global wind direction; actions are farm yaw-misalignment commands, sampled at 1 Hz.Rewards are computed according to equations (2) and (13).Source: B. Dataset Creation, Network… · p. 6
• The initial dataset came from 80 GB of wind-tunnel results using greedy control and LUTs based on lower-, medium-, and higher-fidelity models.LUTs were calculated over wind direction and included modeled direction uncertainty of 0°, 2°, 4°, or 6° across nine profiles.Source: B. Dataset Creation, Network… · p. 6
• The policy was deployed and retrained three times, adding measurements from each deployment to the offline dataset for subsequent fine-tuning.The iterations were labeled Offline RL 1, Offline RL 2, and Offline RL 3.Source: B. Dataset Creation, Network… · p. 6
• Feeding normalized wind direction separately from the normalized three-turbine yaw vector provides ancillary conditioning information and improves actor learning.The customized actor architecture is shown in Fig. 4.Source: B. Dataset Creation, Network… · p. 6

• Approximately 10% average total power increase over greedy was achieved by all three offline RL policies; MTD3-BC matched data-calibrated LUT performance without a wake model.Source: C. Experimental Results · p. 6, Figure 5 · p. 7
• Under controlled, repeatable wind-tunnel conditions with a good-quality data-calibrated wake model, LUT and offline RL performed similarly, making LUT a near-optimal benchmark.The authors note residual differences would probably not be measurable in full-scale field deployment.Source: C. Experimental Results · p. 7
• The offline RL policy maps measured conditions to continuous yaw commands, whereas LUT uses a discrete grid and requires interpolation between operating conditions.Source: C. Experimental Results · p. 7
• Across most wind directions, farm-level power gains were positive and highest around −6° to 6°; WT2 and WT3 gains exceeded 100% in some cases from −9° to 6°.WT1 power was lower than greedy in most directions, but downstream gains more than compensated at farm level.Source: C. Experimental Results · p. 7, Figure 6 · p. 8
• Farm-level gains were similar for MTD3-BC and LUT; LUT had a greater WT1 power drop near −3° to 3°, a higher WT2 peak gain, and comparable WT3 trends.Source: C. Experimental Results · p. 7, Figure 7 · p. 8
• MTD3-BC training and fine-tuning took approximately 5% of online PPO's cost, despite only marginally better performance.Source: C. Experimental Results · p. 8
• Mean yaw schedules nearly coincided; near −3° to 3°, MTD3-BC had less WT1 yaw-offset dispersion, while WT3's systematic offset stayed below 1° and warrants caution.Source: C. Experimental Results · p. 7, Figure 8 · p. 9
• A fourth 10-minute profile showed farm-level power over time alongside turntable rotation, demonstrating increased production under time-varying wind directions.Source: C. Experimental Results · p. 7, Figure 9 · p. 9

V. CONCLUSION

• MTD3-BC builds on the standard TD3 actor-critic structure and adds behavior-cloning and action-consistency terms to its policy-learning objective.Source: §V · p. 9
• MTD3-BC performed on par with a data-calibrated model-based wake-steering benchmark, with a marginal improvement over the best behavior policy in its training dataset, at approximately 5% of an online PPO controller’s training cost.The results come from wind-tunnel tests in dynamic wind environments.Source: §V · p. 9
• The work reports the first successful experimental validation of an offline RL-based control strategy for wind farms.The authors identify commercial wind-farm deployment as a subject for future verification.Source: §V · p. 9