Source-linked AI summary
ALIGN-HOLD: Experience Alignment for Real-Time Hold Control in Large-Scale Ride-Hailing Matching at DiDi
Zuhao Zhang, Xu Liu, Kai Wan, Zihao Lu, Li Ma, Shuai Li
TL;DR
ALIGN-HOLD addresses the difficulty of optimizing ride-hailing hold decisions with handcrafted rewards when marketplace preferences and behavioral feedback are heterogeneous and noisy. It learns an experience Reward Model from multi-view implicit preferences and uses it to guide policy learning with dense rewards and low-identifiability filtering. Online experiments and deployment in Brazil show improved passenger–driver experience and marketplace efficiency.
Problem
Handcrafted hold rewards are difficult to generalize because marketplace preferences vary, while observed behavioral feedback is sparse, noisy, and weakly attributable.
Method
ALIGN-HOLD constructs order-, driver-, and market-context preference pairs, trains a Reward Model with balanced and hard-pair sampling, and uses the frozen model for dense rewards and filtering during policy learning.
Results
ALIGN-HOLD improves trip completion and matching efficiency while reducing experience-degrading cancellations in large-scale online experiments, and it has been fully deployed in Brazil.
Takeaways & Limitations
Learning experience alignment from marketplace behavior provides a production approach for more precise hold decisions under heterogeneous experience preferences.
Takeaways & Limitations
Ride-hailing feedback remains difficult to interpret because realized outcomes can reflect external factors beyond intrinsic driver–order compatibility.
Abstract
from arXiv · showhide
Real-time hold control is a high-leverage mechanism in large-scale ride-hailing systems: by selectively deferring driver-order pairs, the platform can wait for better matching opportunities and improve end-to-end passenger-driver experience. Existing production systems such as EXHOLD learn bandit-based hold policies from handcrafted combinations of trip completion, cancellations, waiting time, and driver effort. However, designing such rewards becomes increasingly difficult as marketplace preferences are heterogeneous and observed passenger-driver behavior can be sparse, noisy, and affected by dynamic supply-demand conditions. We present ALIGN-HOLD, a production-scale experience alignment framework that learns hold policy from implicit marketplace preferences. ALIGN-HOLD constructs complementary preference pairs from order trajectories, driver trajectories, and contemporaneous local matching graphs, and trains an experience Reward Model (RM) using balanced multi-view sampling and model-adaptive hard preference sampling. During simulator-based policy learning, the frozen RM provides a dense, context-dependent reward and supports label-free filtering of low-identifiability interactions whose behavioral feedback is difficult to attribute to matching quality. We deploy ALIGN-HOLD on DiDi's ride-hailing platform and evaluate it in a 28-day randomized A/B experiment, covering approximately 100,000 passenger requests per day. Compared with the deployed production policy, ALIGN-HOLD achieves statistically significant improvements in trip completion rate and driver income, while significantly reducing passenger cancellations before and after driver acceptance. Complementary ablations, RM diagnostics, and behavioral analyses validate the contributions of the proposed components. ALIGN-HOLD has been fully ramped up and is currently serving DiDi's Brazil marketplace.
1 Introduction
ALIGN-HOLD replaces manually engineered hold rewards with experience alignment learned from heterogeneous marketplace behavior. It addresses sparse, noisy, weakly attributable feedback through multi-view preferences and a reward-model-guided policy, achieving production improvements in Brazilian ride-hailing.
- Motivation: Handcrafted rewards become difficult to maintain because passenger and driver preferences vary across regions and time periods.A globally shared reward may require frequent recalibration and still miss the marketplace’s latent experience objective.
- Motivation: Marketplace feedback is sparse, noisy, and weakly attributable because many outcomes reflect non-response or external circumstances rather than match quality.The paper notes that driver non-response dominates many broadcast sequences and explicit cancellations are relatively rare.
- Approach: ALIGN-HOLD constructs relative preference pairs from order, driver, and local matching-graph trajectories instead of relying on isolated absolute outcomes.These comparisons transform production trajectories into scalable implicit preference supervision.
- Approach: The framework trains an experience Reward Model with model-adaptive hard preference sampling, then uses it for dense rewards and low-identifiability filtering during policy learning.The resulting policy retains the production execution layer that converts experience tiers into hold times.
- Real-world deployment and key findings: ALIGN-HOLD improves trip completion and matching efficiency while reducing experience-degrading cancellations in large-scale online experiments.The system was evaluated across multiple Brazilian cities and fully ramped up for production traffic.
2 Problem Formulation
The paper formulates hold control as choosing whether each driver–order pair enters matching immediately or is deferred. Because latent passenger–driver utility is unobserved, policy learning relies on downstream behavioral outcomes and a learned ranking surrogate.
- 2 Problem Formulation: Hold control decides whether each candidate driver–order pair enters downstream matching immediately or is temporarily held.The objective is to improve trip success and passenger–driver experience without unnecessarily holding promising matches.
- 2.1 Real-Time Hold Decisions: The decision state combines the passenger order, available driver, and marketplace context, including trip attributes and local supply–demand conditions.The policy assigns a discrete experience tier from the observed state.
- 2.1 Real-Time Hold Decisions: Lower experience tiers favor immediate exposure, whereas higher tiers indicate stronger preference for deferral.The selected tier is converted into a hold duration by the guardrail-constrained execution layer inherited from EXHOLD.
- 2.2 Implicit Experience Feedback: The ideal objective depends on latent passenger–driver utility, but that utility is not directly observable.The platform instead records downstream outcomes such as trip completion, non-response, and passenger or driver cancellations.
- 2.2 Implicit Experience Feedback: Observed behavioral outcomes provide only imperfect evidence about the quality of a matching decision.Only one interaction is realized for each marketplace state, and outcomes may be affected by factors beyond pair compatibility.
- 2.2 Implicit Experience Feedback: EXHOLD manually combines behavioral outcomes into a reward, whereas ALIGN-HOLD learns a ranking surrogate from implicit relative preferences.The learned surrogate provides dense, context-dependent supervision but must handle sparse, confounded behavior and imperfect reward learning.
3 Method
ALIGN-HOLD learns an experience Reward Model from complementary implicit preference views and uses the frozen model to guide contextual-bandit hold-policy learning. The pipeline combines balanced sampling, hard-pair selection, dense rewards, and filtering of low-identifiability interactions.
- Method: ALIGN-HOLD has two learning stages: multi-view preference-based Reward Model training, followed by simulator-based policy learning with the frozen model.The learned policy outputs experience tiers that the production execution layer converts into hold times.
- 3.1 Multi-View Preference Construction: Completed interactions are preferred to related non-completed interactions, while different failure outcomes are not manually ordered.This shared completion-based rule is applied across the order, driver, and market-context views.
- 3.1 Multi-View Preference Construction: The three preference views compare related interactions sharing an order, driver, or contemporaneous local matching context.Order sharing controls request-level variation, driver sharing controls persistent driver variation, and market snapshots capture local supply–demand alternatives.
- 3.2 Experience Reward Model: The Reward Model assigns a scalar score to each state–action decision and learns relative ranking rather than calibrated satisfaction.Its MLP–Transformer scorer models cross-feature interactions before producing the final score.
- 3.2 Experience Reward Model: Preference pools are balanced during Reward Model training, with model-adaptive hard sampling focused on difficult pairs from the mixed pool.The regular portion samples the three views at a 1:1:1 ratio, while hard pairs are selected using current loss.
- 3.3 RM-Guided Policy Alignment: After training, the frozen Reward Model provides a continuous, context-dependent reward while filtering feedback with weak outcome evidence.Normalization preserves score ordering and supplies a stable reward scale for policy learning.
- 3.3 RM-Guided Policy Alignment: The policy interacts with a production-calibrated matching simulator, and only retained interactions contribute normalized Reward Model scores to contextual-bandit updates.The simulator and execution architecture are inherited from the existing production pipeline.
4 Experiments
ALIGN-HOLD is evaluated against the deployed EXHOLD policy through online experiments, ablations, RM diagnostics, and behavioral analyses. It improves marketplace outcomes and reallocates holds more selectively while maintaining nearly unchanged overall hold intensity.
- 4.2 RQ1: Overall Online Evaluation: TC increases by 0.57% and DI increases by 0.64% versus the deployed EXHOLD baseline in the online A/B experiment.The comparison uses the same policy architecture and hold-time execution layer, isolating the learned alignment signal and resulting policy learning.
- 4.2 RQ1: Overall Online Evaluation: Passenger cancellations fall by 1.85% before acceptance and 2.07% after acceptance, while overall hold ratio changes by only 0.04%.Driver acceptance rises by 0.51%, and the results are consistent with more precise allocation rather than globally increased intervention.
- 4.2 RQ1: Overall Online Evaluation: ALIGN-HOLD improves TC, DI, PCAA, and CAT directionally across all five cities and remains beneficial during both peak and off-peak periods.Peak-hour gains are larger across the four reported metrics, while no individual market determines the aggregate direction.
- 4.3 RQ2: Ablation Study: Removing any major component weakens the online results: single-view preferences reduce gains, hard-sampling removal reverses DI improvement, and retaining ambiguous feedback lowers driver income.The full combination of multi-view supervision, model-adaptive hard sampling, and low-identifiability filtering yields the strongest balanced marketplace gains.
- 4.4 RQ3: Alignment-Signal Reliability and Low-Identifiability Filtering: Low-identifiability bands concentrate among non-completed interactions with shorter observable pickup, waiting, and travel costs, making their outcomes ambiguous supervision.These diagnostics support excluding such interactions from simulator policy updates through a per-slice, label-free detector.
- 4.5 RQ4: Behavioral Analysis: At the 90th percentile, held pairs reach approximately 2.3× pickup time and 2.9× pickup distance relative to non-held medians, while hold rates shift across cohorts without changing globally.Holding increases for long-pickup, long-wait, and low-historical-acceptance cohorts and decreases for favorable counterparts; the largest shifts are −0.21 and +0.28 percentage points.
5 Related Work
Prior work studies real-time ride-hailing dispatch, matching, timing, experience-aware hold control, and pairwise preference learning. ALIGN-HOLD retains EXHOLD’s production architecture while replacing its globally handcrafted reward with context-dependent alignment from implicit behavior.
- Ride-hailing dispatch, matching, and timing control: Prior ride-hailing systems combine learning with planning for dispatch, model pair-level matching quality, and optimize matching intervals or pickup-time targets.These approaches balance matching quality, waiting, and cancellation risk under dynamic marketplace conditions.
- Experience-aware hold control and EXHOLD: EXHOLD learns interpretable experience tiers with a contextual bandit and maps them to hold times through guardrail-constrained optimization.Its reward combines completion, cancellation, waiting, and driver-effort outcomes.
- Experience-aware hold control and EXHOLD: ALIGN-HOLD retains EXHOLD’s policy architecture and execution guardrails while learning a context-dependent experience objective instead of a globally handcrafted reward.This addresses context-dependent passenger and driver responses to pickup time, trip attributes, and marketplace conditions.
- Pairwise preference learning and reward modeling: Pairwise preference learning uses relative comparisons or trajectory rankings to supervise reward models rather than relying on absolute utility labels.The paradigm has been applied to implicit-feedback ranking, trajectory comparison, and language-model alignment.
6 Conclusion
ALIGN-HOLD replaces handcrafted reward engineering with experience alignment learned from implicit marketplace behavior for production-scale real-time hold control.
- ALIGN-HOLD learns hold policies from implicit marketplace behavior rather than handcrafted reward engineering.
- The framework combines multi-view preferences, balanced and model-adaptive sampling, dense Reward Model rewards, and low-identifiability feedback filtering.
- ALIGN-HOLD was evaluated through large-scale online experiments and fully deployed in DiDi’s Brazil market.
A.1 Behavioral Analysis
The behavioral analysis examines whether ALIGN-HOLD adapts intervention strategies to temporal dynamics and local supply–demand environments.
- The analysis extends pair-level decision patterns by studying policy adaptation across temporal and local marketplace conditions.
- It evaluates whether hold decisions change with temporal dynamics rather than remaining fixed across marketplace conditions.
- It also evaluates adaptation to local supply–demand environments beyond static pair-level decisions.
A.1.1 Temporal Adaptation.
ALIGN-HOLD adapts holding to changing matching uncertainty, using more intervention when alternatives are limited and less when immediate matching is promising.
- ALIGN-HOLD reallocates holds toward periods with higher matching uncertainty instead of uniformly changing intervention intensity relative to EXHOLD.
- During late-night and early-morning periods, ALIGN-HOLD holds more, while during dense daytime periods it avoids unnecessary delays.
- The policy responds to the contextual value of waiting, not only static driver–order compatibility.
- ALIGN-HOLD applies more holding in low-connectivity regions and becomes less conservative as supply availability and matching density increase.
A.1.2 Supply–Demand Regime Adaptation.
ALIGN-HOLD adapts hold decisions to marketplace context by combining interaction-level experience assessment with supply–demand opportunity costs. Its reward model preserves meaningful relative experience ordering despite noisy behavioral feedback.
- A.1.2 Supply–Demand Regime Adaptation.: ALIGN-HOLD considers the opportunity cost of waiting, so holding depends on marketplace alternatives as well as current driver–order quality.The resulting strategy combines individual experience assessment with marketplace-level adaptation.
- A.2 Reward Model Validation: The RM achieves 0.845 pairwise preference accuracy on held-out production-derived preference pairs.This diagnostic evaluates whether the model recovers relative experience ordering from noisy, naturally occurring marketplace trajectories.
- A.2.2 RM score as an experience ordering signal.: Higher RM-score deciles achieve higher trip completion and driver acceptance rates while reducing passenger and driver cancellation rates.The aggregate outcome AUC is 0.852, and the highest–lowest decile gap is substantial across positive and cancellation-related outcomes.
- A.2.2 RM score as an experience ordering signal.: RM discrimination is stronger for active cancellations than driver non-response because non-response is influenced by availability, attention, and external constraints.The model nevertheless maintains meaningful discrimination under noisy feedback by aggregating multiple preference views.
- Training and deployment workflow: The workflow constructs multi-view preference pairs, trains a reward model with balanced and adaptive sampling, and uses calibrated RM scores during simulator policy learning.The deployment path uses a tier policy and calibrated lookup table rather than online RM inference or filtering.
- B.1 Multi-View Preference Construction Details: Preference supervision treats a completed interaction as preferred over a non-completed interaction only for contextually related pairs.Comparisons between two completed or two non-completed interactions are excluded, and post-decision information is used only for outcome labeling.
B.2 Reward Model Architecture and Training Settings
The reward-model and policy pipeline uses a calibrated frozen RM to provide dense rewards and filter low-identifiability simulator interactions. Ordered experience tiers are then converted into executable hold times through a constrained, auditable calibration stage.
- B.2 Reward Model Architecture and Training Settings: The reward model uses two pre-norm Transformer layers and a scalar preference-score head after numerical and categorical feature tokenization.Numerical features become 64-dimensional tokens, while categorical features use hashed 16-dimensional embeddings projected into the same token space.
- B.2 Reward Model Architecture and Training Settings: Training balances order-, driver-, and market-context preference pools at 1:1:1 for 80% of each batch and samples hard pairs from the 70th–90th loss percentiles for the remaining 20%.The adaptive component is selected by current pairwise loss, and optimization uses only the pairwise Bradley–Terry objective.
- B.3 Low-Identifiability Filtering and Tier-Policy Settings: The frozen RM supplies normalized contextual-bandit rewards, while detected score-spike interactions are excluded from policy updates.RM inference and low-identifiability filtering remain offline; the RM score is not added to the policy context or serving path.
- B.3 Low-Identifiability Filtering and Tier-Policy Settings: The retained interactions train a four-action Transformer–LinUCB policy covering immediate release and three increasing hold tiers.After training, the policy outputs ordered tiers that are translated into executable durations by the inherited calibration module.
- B.4 Guardrail-Constrained Hold-Time Calibration: ALIGN-HOLD changes how experience tiers are learned but reuses EXHOLD’s production-tested hold-time execution module and runtime guardrails.This separation preserves existing operational properties while isolating the learned alignment policy’s contribution.
- B.4 Guardrail-Constrained Hold-Time Calibration: Hold durations are selected from empirical quantile grids by dynamic programming under trip-completion guardrails and monotonicity constraints.The resulting lookup table can be refreshed, audited, versioned, and rolled back independently of learned-model retraining.
B.5 Simulator Fidelity Evaluation
The matching simulator closely reproduces the deployed EXHOLD policy’s production behavior over a held-out seven-day period. Agreement is strong across city–hour cohorts and preserves behavioral ordering.
- B.5 Simulator Fidelity Evaluation: A 0.94 Pearson correlation links simulated and online hold rates across city–hour cohorts.The simulator also matches experience-tier shares within a 0.46 percentage-point mean absolute difference.