Horizon: Facebook's Open Source Applied Reinforcement Learning Platform

• Existing RL implementations were mainly tailored to academic prototyping and evaluation on simulated benchmarks, while applied settings bring large, varied datasets and high-dimensional action spaces.Source: §1 · p. 1
• Horizon is an open-source, end-to-end applied RL platform developed at Facebook for real, industry-produced datasets.It uses PyTorch for modeling and training and Caffe2 for serving.Source: §1 · p. 1
• Horizon’s workflows include data preprocessing, feature normalization, deep RL algorithms, distributed training, counterfactual policy evaluation, and optimized model serving.The platform implements algorithms for discrete, large discrete, and continuous action spaces.Source: §1 · pp. 1–2
• A model-based data understanding tool checks problem formulation before RL training, helping practitioners examine whether real-world problems fit an MDP framework.The paper says it accelerated data-engineering iterations and provided explainable insights.Source: §1 · p. 2
• Horizon tests its RL algorithms with unit and integration tests in custom and standard environments on every pull request.Source: §1 · p. 2
• Horizon-trained RL models outperformed supervised-learning and heuristic policies in examples involving notifications and video streaming at Facebook.The paper describes real examples of these models replacing existing systems.Source: §1 · p. 2

• The Timeline Spark pipeline transforms logged production data into the row format needed to train RL models on sequential state-action examples.Source: §2 · p. 2
• Each training row can include an MDP identifier, sequence number, state features, action, action probability, metrics, and possible actions.The format also includes next-state features, next action, sequence ordinal, time difference, and possible next actions.Source: §2 · pp. 2–3
• A metrics map replaces an explicit reward scalar, allowing reward shaping and counterfactual policy evaluation over individual metrics.This supports granular estimates of a newly trained policy’s performance.Source: §2 · p. 3
• Users can change reward formulas by adjusting metric weights without regenerating the data tables.The scalar training reward is computed as the dot product of metric values and user-provided weights.Source: §2 · p. 3

• Horizon analyzes each feature and selects a transformation function and normalization parameters, which developers can override using prior knowledge.The workflow identifies binary, probability, continuous, enum, quantile, and boxcox feature types.Source: §3 · p. 3
• Feature normalization addresses sparse, noisy, and arbitrarily distributed recommender data, where large features can destabilize RL recurrence.Source: §3 · p. 3
• Transformations are precomputed but applied to raw features during the training network’s forward pass, enabling iteration without regenerating the dataset.Source: §3 · p. 3

• Problem formulation matters because ill-defined MDP data can lead to online metric regressions or engineering effort spent tuning irrelevant training factors.Source: §4 · p. 3
• The tool combines a data-driven, model-based method with heuristics to check whether problem-formulation properties conform to the MDP framework.Source: §4 · pp. 3–4
• The tool learns a probabilistic environment model from the same dataset intended for RL training, using current state and action to predict next states and rewards.A Gaussian mixture output represents possible stochasticity rather than producing only point estimates.Source: §4 · p. 4
• Feature importance is estimated by the increase in model loss when a feature is masked by setting it to its mean value.The authors propose using this measure to select features for RL training and reduce noise.Source: §4 · p. 4
• The tool checks whether actions and state features predict transitions, and whether action-dependent state features predict rewards relevant to those transitions.Its checks flag formulations without sequential structure or with rewards reducible to estimating each action’s return.Source: §4 · p. 4
• The tool has helped users understand problem formulation early in RL training and has caught many improperly defined problems, though its heuristics do not cover all invalid formulations.Source: §4 · p. 4

5. Model Implementations

• Horizon supports algorithms for tractable discrete, very large discrete, and continuous action spaces.The platform also supplies default configurations for test domains.Source: §5 · p. 4

5.1. Discrete-Action Deep Q-Network (Discrete DQN)

• DQN implementations include double Q-learning, dueling architecture, and multi-step learning improvements for discrete actions.DQN is included for its relative simplicity and use as a building block for improvements.Source: §5.1 · p. 4

5.2. Parametric-Action Deep-Q Network (Parametric DQN)

• Parametric-Action DQN scores concatenated state-action feature pairs, addressing action spaces with more than millions of possible actions.The platform also adapts double Q-learning and dueling architecture to this variant.Source: §5.2 · p. 5

5.3. Deep Deterministic Policy Gradients (DDPG) and Soft Actor-Critic (SAC)

• For continuous-action domains, Horizon implements DDPG and SAC.DDPG was selected for simplicity and familiarity, while SAC was selected for demonstrated state-of-the-art performance on continuous-action domains.Source: §5.3 · p. 5
• Support for additional deep RL algorithms remains a continued focus.Source: §5.3 · p. 5

6. Training

• Horizon supports CPU, GPU, and multi-GPU training across multiple machines through PyTorch's multi-GPU functionality.Source: §6 · p. 5
• Hundreds-to-thousands-feature RL models train on tens of millions of examples in a few hours using GPU or multi-GPU training, with normalization and CPE on every batch.Source: §6 · p. 5
• As the policy moves from off-policy data to on-policy data, A/B test results improve; Figure 1 shows the optimized metric starting neutral before the RL model exceeds control.After deployment to a fraction of production traffic, subsequent training uses data generated by the RL policy.Source: §6 · p. 5, Figure 1 · p. 5
• Daily retraining from prior weights and optimizer state uses self-generated data to make learning more on-policy and mitigate extrapolation error.The paper describes extrapolation error as unrealistic value estimates for unseen state-action pairs when learning from fixed batches generated by another policy.Source: §6 · p. 5

7.1. Training: TD-loss & MC-Loss

• TD-loss tracks function-approximation error and is used mainly to monitor optimization stability because RL labels shift with the model.An unbounded increase indicates an overly aggressive optimization step, such as a learning rate that is too high or minibatch size that is too small.Source: §7.1 · p. 6
• MC-loss compares predicted Q-values with logged discounted rewards, but is limited as a real-world performance measure when the logged policy is not optimal.It is monitored for extreme values or unbounded increase.Source: §7.1 · p. 6

7.2. Evaluation: Counterfactual Policy Evaluation

• CPE predicts a learned policy's performance without online deployment, enabling offline estimates alongside real-world testing instead of relying only on costly, time-consuming A/B searches.The paper describes CPE estimates as a basis for automated hyperparameter sweeps.Source: §7.2 · p. 6
• Horizon runs six CPE estimators as part of training, combining methods designed for contextual bandits with sequential methods for longer horizons.The step-wise direct method, importance sampling, and doubly robust estimators were designed for one-step horizons; sequential doubly robust, sequential weighted doubly robust, and MAGIC address longer horizons.Source: §7.2 · p. 6
• CPE estimates are available at each epoch, share feature-normalization improvements with training, and add minimal time overhead to the pipeline.The workflow restores sampled training data to its original sequence for sequential estimators and batches Q-value queries at epoch end.Source: §7.2 · pp. 6–7
• On a real Facebook dataset, CPE estimates indicate roughly 1.5x–1.8x as much cumulative reward as the logged system, with estimates improving over training epochs.Figure 2 plots relative value estimates against training time; a score of 1.0 indicates parity with the logged policy.Source: Figure 2 · p. 7

7.3. TensorboardX

• TensorboardX exports training metrics for viewing in Tensorboard; its example CPE plots show estimates by training epoch for MAGIC and Weighted Doubly Robust.The dashboard can also display additional CPE methods and loss plots.Source: §7.3 · p. 7, Figure 3 · p. 7

8. Model Serving

• ONNX tracing converts an RL model's forward pass, including feature transformation and policy outputs, into a serializable, portable, efficient Caffe2 network for deployment to thousands of machines.Source: §8 · p. 7
• At serving time, teams log possible actions, their propensities, the selected action, and its reward, which may arrive hours or days later.A unique key can join samples to external reward data sources.Source: §8 · p. 7
• Joined serving data is fed back into Horizon for incremental updates; closed-loop training matters because off-policy algorithms remain limited by observed policies and changing data distributions.Source: §8 · p. 7

9.1. Push Notifications

• For user notifications, a Discrete-Action DQN compared send and drop action values to optimize long-term value and capture incremental effects.The difference between the send and drop Q-values was passed through a sigmoid to form the notification policy.Source: §9.1 · p. 8
• A PID controller tuned the user-notification policy threshold to keep its action distribution in line with the previous production policy.The threshold was used in the RL policy’s sigmoid-based action rule.Source: §9.1 · p. 8
• The user-notification model was retrained daily on data from exposed people, with action exploration during serving and batches of tens of millions of transitions.The passage describes incremental retraining using newly collected experience.Source: §9.1 · p. 8
• Horizon-based notification policies significantly improved activity and meaningful interactions, replacing the prior supervised-learning system.The previous system used supervised predictions of click-through rate and meaningful interactions to score notifications, but did not capture long-term or incremental value.Source: §9.1 · p. 8

9.2. Page Administrator Notifications

• Page-administrator DQN improved daily, weekly, and monthly metrics without sacrificing notification quality.Training spanned multiple weeks and used accumulated discounted rewards to capture administrators’ longer-term responses and intent to remain active.Source: §9.2 · p. 8

9.3. More Applications of Horizon

• In 360-degree video, Horizon reduced bitrate consumption without harming people’s watching experience.The reported improvement was attributed to more intelligent video buffering and pre-fetching.Source: §9.3 · p. 8

10. Future Work

• Horizon plans incremental improvements to current models and continued additions of the best-performing algorithms from the research community.The paper also welcomes community pull requests, suggestions, and feedback.Source: §10 · p. 8