HiLO: Towards Product-Level Graceful Degradation with a Quality-Aware, Load-Aware Control Framework

1 Introduction

• Persistent traffic growth, transient capacity losses, and GPU-intensive LLM demand make availability vulnerable to overload that can outpace human intervention.Overprovisioning is expensive and inefficient at scale, while specialized capacity constraints and difficult-to-predict spikes limit its effectiveness.Source: §1 · p. 1
• Local degradation can preserve individual services yet leave product quality unpredictable when independent decisions interact across interdependent services.Manual feature knobs also create a large interaction space that requires continual maintenance.Source: §1 · p. 1
• HiLO maps aggregate service load to a finite lattice of prevalidated product states, each deterministically assigning coordinated service levels.Dependencies are resolved at design time, enabling autonomous control without runtime dependency discovery across heterogeneous services, including GPU-intensive models and LLMs.Source: §1 · pp. 1–2

2.1 Five Metrics to Watch

• The evaluation metrics cover availability, product quality, stability, recovery velocity, and capacity savings.Quality includes OneDCG, error signals, freshness, and engagement; stability measures consistent responses across repeated queries under identical observable context.Source: §2.1 · p. 2

2.2 Evolution from Basic Graceful Degradation to HiLO - Quality Awareness

• Earlier service-local protections maintained availability above 99.95%, but made product state and baseline quality difficult to reason about and complicated product-level testing.The incident illustrates how an available service can still contribute to upstream latency, cascading errors, and prolonged quality loss when degradation is uncoordinated.Source: §2.2 · pp. 2–3
• OneDCG fell 1.11 points after callers continued sending demand to a locally degraded shared model-serving service; product stability dropped from ~90% to ~63% and took over four hours to recover.The triggering load spike lasted under 10 minutes, and OneDCG fluctuated for another two days when load increased before returning to baseline after a fix reduced dependency calls.Source: Figure 1 · p. 2, §2.2 · p. 3, Figure 2 · p. 3

2.3 Related Work

• HiLO targets tightly interdependent systems with heterogeneous capacity that need continuous autonomous coordination rather than manual intervention or binary feature toggles.The related approaches discussed include feature degradation, service-level admission control, and priority-based shedding.Source: §2.3 · pp. 3–4

3.1 Load-Factor and Product State Lattice

• HiLO treats normalized service load factors as ordinal congestion signals and maps them to a small lattice of coordinated states, trading fine-grained optimality for predictability and testability.Load factors can reflect CPU or GPU use, queue depth, tail latency, or dependency backpressure; HiLO assumes only that higher values indicate greater overload risk.Source: §3.1 · pp. 4–5

3.2 Deep Model & LLM Specific Strategies

• For GPU-intensive models and LLM serving, HiLO applies lower-cost variants uniformly only under overload, while progressively reducing queries and documents before deeper model-layer changes.The degradation sequence is determined and validated at design time.Source: §3.2 · pp. 5–6

3.3 Product and Service-Level Construction

• Service owners validate local degradation levels, callers and callees jointly resolve dependencies, and product owners compose and end-to-end test candidate states.Candidates that exceed user-metric targets or cause unintended experiences are adjusted or discarded.Source: §3.3 · p. 6

3.4 Further Controls in HiLO

• Route contexts support distinct concurrent product states, while request-priority metadata lets services degrade lower-priority traffic before higher-value requests.The frontend assigns route context and priority metadata for downstream services.Source: §3.4 · pp. 6–7

3.5 Monitoring, Testing & Experimentation

• Monitoring logs aggregated load factors, selected states, and assigned service levels; test and experiment hooks validate candidate states before gradual user-traffic rollout.Test traffic supports measuring OneDCG, latency, and capacity utilization; experiments track business metrics as traffic exposure increases.Source: §3.5 · p. 7

3.6 Safety and Operational Properties

• HiLO transitions monotonically toward more restrictive states as load rises, damps recovery transitions, and lets services preserve local protection if HiLO becomes unavailable.Each service enforces the more restrictive of its HiLO-assigned level and its locally determined level.Source: §3.6 · p. 7

• The evaluation examines HiLO’s operational response to real overload conditions.Source: §4 · p. 7
• The evaluation compares performance with and without HiLO.Source: §4 · p. 7
• The evaluation measures load-factor and capacity improvements during degradation for services and GPU-intensive models.Source: §4 · p. 7
• The evaluation assesses user-facing impact through A/B experiments and quality metrics.Source: §4 · p. 7

4.1.1 HiLO’s Response to Overload

• 0.14 OneDCG points of impact occurred in an APAC model-induced overload, versus a 1.11-point drop and severity-2 escalation in the earlier incident.The event lasted about six hours daily over two days; another North America surge reached 1.93× baseline peak and remained in overload for three hours.Source: §4.1.1 · pp. 7–8

4.1.2 HiLO’s One Month Operational Snapshot

• HiLO degraded only while load-factor exceeded its threshold, so active degradation was shorter than overload duration and recovery followed promptly.Similar peak traffic could produce different escalation depths when proportional surges and load-factor dynamics differed.Source: §4.1.2 · pp. 8–9
• 0.425 OneDCG points at 2.38× peak load was the worst monthly drop, below the 0.5-point target; all datacenters maintained availability above 99.95%.HiLO managed 9 user-traffic surges and 16 planned capacity tests during January 2026.Source: §4.1.2 · p. 8
• Metric recovery measurements show recovery after overload ends, with OneDCG recovery defined by return within 0.05 of its pre-event baseline and stability recovery by reaching 90%.The paper reports that metric recovery confirms underlying services remained healthy and recovered rapidly once load subsided.Source: §4.1.2 · p. 8, Figure 8 · p. 8, Figure 9 · p. 9

4.1.3 Comparative Performance: Capacity Tests

• At approximately 2× baseline peak in short, gradual capacity tests, HiLO showed no OneDCG impact and reduced no-result errors versus uncoordinated service-local degradation.The paper cautions that these tests are less representative of production-overload impact than the monthly surge results.Source: §4.1.3 · p. 9, Table 3 · p. 9, Figure 12 · p. 10

4.1.4 30+ Hours, 3 Datacenters and Autonomous Overload Management

• During a 30+ hour outage of DC-A, DC-B spent 83% of its 16.7 degradation hours at L1–L2, while DC-C’s L5 spikes lasted 0.4 hours.Availability remained above target; OneDCG drop stayed within target at a maximum of 0.38 points, but DC-C oscillated during a GPU-model capacity deficit until additional capacity was provisioned.Source: §4.1.4 · p. 10, Figure 13 · p. 10, Figure 14 · p. 10

4.2.1 Deep Models and Services Evaluation

• Upstream Multi-Queries and document reduction cut ranking-model inference calls by up to 29% at L5, while leaving the model unchanged.At L4–L5, model-layer reduction separately reduces per-inference latency and compute cost; this mode was validated but pending production deployment.Source: §4.2.1 · p. 11, Table 4 · p. 11
• For the ranking model, throttling or skipping it entirely produces OneDCG impact far above the Severe threshold, supporting graduated degradation over binary disabling.Call-volume reduction acts on demand, whereas model-layer reduction acts on per-call supply cost.Source: §4.2.1 · p. 11, Table 4 · p. 11
• OneDCG is the primary business-critical quality metric, but content freshness and engagement can worsen at deeper states even when OneDCG is held.Service owners manage OneDCG drops across degradation levels to keep them within target.Source: §4.2.1 · p. 11

4.2.2 Case Study: HiLO for Copilot Search

• Copilot Search uses five product states that progressively disable features triggering model calls, from tab repositioning at L1 to hiding the tab at L4.Its degradation is enforced at the frontend, unlike Web Search’s backend-enforced degradation, yet HiLO’s framework applies unchanged across both execution regimes.Source: §4.2.2 · p. 11, Figure 15 · p. 11

4.3.1 HiLO Operational Overhead

• HiLO operates outside the request path and adds no execution-path latency.Table 6 summarizes its operational footprint.Source: §4.3.1 · p. 11, Table 6 · p. 11

4.3.2 User-Facing Impact of Product States

• A/B experiments compare candidate product states with baseline L0 using business metrics, including revenue, DAU, and session engagement, to measure state-level quality tradeoffs.Candidate states are activated on a controlled fraction of production traffic.Source: §4.3.2 · pp. 11–12
• Retention was unaffected, while content freshness degraded monotonically and DSQ reformulation increased by 0.14% to 3.49% across surfaces and states.The mobile L1 DSQ increase of 3.49% was statistically significant; tail latency improved on both surfaces.Source: §4.3.2 · p. 12, Table 7 · p. 12

Coarse-grained control

• Bing's six product states simplify validation and operational reasoning but limit fine-grained, per-feature adjustments.The authors report that finer granularity neither meaningfully increases capacity reclaimed per step nor accelerates reclamation.Source: §5 · p. 12

Load signals are service-specific

• Services select normalized load-factor indicators or blends that reflect their bottlenecks, such as queue length, CPU utilization, or GPU utilization.The appropriate indicator can shift as a service's workload evolves.Source: §5 · p. 12

Design-time burden

• Defining product states requires services to identify degradation actions, quantify capacity impact, and coordinate across teams, often taking weeks to months.For a service with 90k+ sub-workflows, analysis took three months; full automation of construction remains open.Source: §5 · p. 12

Staged adoption

• HiLO adoption proceeds in stages, beginning with load-factor signals and coordinated degradation for critical, capacity-constrained services before expanding coverage.Regular validation, A/B experiments, and recalibration inform further model coverage and degraded-mode design.Source: §5 · pp. 12–13

Applicability to decoupled architectures

• In decoupled architectures, HiLO coordinates service levels off the request path while services enforce assigned levels in their own request paths.A scenario in Bing coordinates roughly a dozen major services and models, with design cost typically scaling with bottleneck services rather than total service count.Source: §5 · pp. 12–13

Quality vs. stability tension

• Under 2× load, aggregate stability drops while OneDCG is preserved, reflecting a policy that prioritizes business metrics over stability as an end in itself.Recovery stability is configurable through a damping factor, and the policy sometimes accepts reduced stability during degradation.Source: §5 · p. 13

Metric latency and tuning risk

• Business-critical metrics such as revenue and daily active users arrive hours or a day later, so fast-loop signals alone may miss a misconfigured state's impact.The authors say real-time signals and A/B experimentation are both needed; degradation changes are carefully validated and rolled out to reduce live-incident risk.Source: §5 · pp. 12–13

Load-spreading first, degradation next

• HiLO is scoped per datacenter because CDN load-spreading is tried first, and its interaction with degradation thresholds can create feedback if misaligned.Bing uses damping factors for failaways and datacenter-level degradation to prevent rapid oscillations; the design assumes parity across datacenters.Source: §5 · p. 13

Conclusion

• HiLO makes product quality a controlled outcome by resolving dependencies at design time and operating over a lattice of pre-validated product states.The system emerged from six years of Bing's graceful-degradation experience.Source: §6 · p. 13
• Evaluation reports preserved availability, quality loss within operational targets, and monotonically ordered user-facing impact validated through A/B experiments.HiLO manages overload events across all Bing datacenters, including Web Search and LLM pipelines.Source: §6 · p. 13

Future Work

• Future directions include product-level awareness in cross-region traffic management and learned policies that discover states from historical load and quality outcomes.Source: §6 · p. 13