From Offline Proxies to Online Decisions: A Layered Engagement Evaluation Framework for Conversational AI

1 Introduction

• Online experiments are capacity constrained by traffic, seven-day engagement readouts, and limited candidate slots, motivating measured offline prioritization rather than assumed proxy validity.A typical experiment in the studied program resolves only a handful of the candidates teams want to compare.Source: §1 · p. 1
• The framework treats proxy validity as three linked alignments: training label to product outcome, classifier to candidate behavior, and suite plus aggregation to between-arm effects.The checklist makes the frozen composite the evaluation unit and orders diagnosis because later stages cannot repair earlier misalignment.Source: §1 · pp. 1–2
• The composite combines a session-engagement classifier, a fixed suite scoring each arm, and calibration from score differences to engagement deltas with an offline bootstrap interval.The main audit uses a fixed static suite for a multiturn assistant; simulated continuations are treated as a separate suite-design extension.Source: §1 · p. 1
• 81.1% contrast-micro F1 versus 34.3% for raw EC, with zero versus 31 wrong-direction calls, on 113 post-freeze replication contrasts.The 113 contrasts came from eight experiments; offline predictions predated exposure, although the analysis plan and model-tier split were not fixed in advance.Source: §1 · p. 2
• The evaluation supports prioritization within completed experiments, not full-funnel screening or traffic-savings claims; randomized experiments remain launch evidence.The frozen composite and its offline outputs predated treatment exposure, but the results were assembled after outcomes were known.Source: §1 · p. 2

2 Related Work

• Unlike early outcomes measured on exposed users, this proxy is computed from candidate behavior before exposure and tested against later signed A/B effects.Surrogate-outcome research motivates empirical validation against delayed targets [31] [34].Source: §2 · p. 2
• Logged-data policy evaluation corrects exposure bias, whereas these candidates have no treatment-arm logs and are scored on a fixed suite before later randomized readouts.The related recommender literature also tests whether offline metrics predict online performance [7] [13] [18].Source: §2 · p. 2
• Unlike preference-trained reward models or general dialogue-quality judges, the classifier measures agreement with randomized engagement effects, motivating decision and ranking metrics.The cited work covers reward models [2] [5] [24], overoptimization [10], and dialogue metrics or LLM judges [4] [11] [12] [20] [21] [23] [35].Source: §2 · p. 2
• The paper makes no standalone simulator-validity claim; rollout depth and scoring are assessed only by whether they improve offline–online agreement.Prior work uses simulators for dialogue-policy evaluation [28] and tests LLM user agents against real behavior [15] [22].Source: §2 · p. 2
• Its calibration regresses composite offline signals onto online deltas, while bootstrap intervals propagate offline-scoring variation rather than predicting future-effect uncertainty.The paper frames its distinction as an end-to-end, pre-exposure-computable audit target, not a new classifier, simulator, or calibration algorithm.Source: §2 · p. 2

3 Methodology

• The methodology positions completed-experiment agreement as evidence for prioritization, not as proof of full-funnel screening utility.Randomized experiments remain the launch evidence.Source: §3 · p. 2, §3.1 · p. 3

3.1 Layered Alignment

• The framework diagnoses three alignments—behavioral label to product outcome, scorer to candidate behavior, and suite plus calibration to experiment effects—and audits the frozen composite end to end.The layers are ordered by diagnostic dependency, not execution; later calibration cannot repair an earlier label or scorer mismatch, and the direct audit does not independently validate Layers 1–2.Source: §3.1 · p. 3

3.2 Evaluation Method: Interval-Aware Decision Agreement

• The evaluation protocol combines interval-aware significant-mover decisions, using online confidence intervals and offline scoring-stability intervals, with within-experiment ranking by top- and bottom-tail precision.The stability interval is not a prediction interval for a future online effect; ranking is evaluated on completed experiments.Source: §3.2 · pp. 3–4

3.3 Engagement Target and Scorer

• The scorer predicts whether a production user returns to a new prompt session within six hours, while candidate-response scores are aggregated by arm and calibrated against online engagement deltas.At evaluation time the return label is unobserved, and the score is comparative rather than a calibrated probability, causal effect, or online lift.Source: §3 · p. 2, §3.3 · p. 4

3.4 Evaluation-Suite Design

• The headline audit uses 4,832 fixed examples per arm/profile, selected at critical turns; simulated multi-turn rollout is evaluated separately as a suite extension.Critical-turn selection uses a judge that sees the later logged trajectory, so selection may favor events legible in hindsight.Source: §3.4 · pp. 4–5

3.5 Offline-to-Online Calibration

• Calibration maps pre-exposure offline features to online deltas using a no-intercept weighted composite learned from historical paired rows, with robust-H90 as its development-time selection criterion.The retained map uses four features; robust-H90 is a tail-sensitive development criterion, not a predictive interval or an F1 objective.Source: §3.5 · p. 5
• The scoring-stability interval resamples retained classifier scores through the frozen map, excluding generation and scorer randomness, calibration-fit uncertainty, and future online residual variation.The frozen export is on a three-day scale joined to seven-day outcomes; sign calls and ranking are invariant to common positive rescaling, but magnitude is not.Source: §3.5 · p. 5

4.1 Evidence Map and Protocol Boundaries

• The 286-contrast overlap and its 87.3% experiment-disjoint holdout support consistency, not primary replication; resampling ranges are not future-experiment coverage intervals.The primary discovery and replication cohorts are chronologically disjoint from each other and from the earlier overlap; results condition on completed evaluations.Source: §4.1 · pp. 5–6

4.2 Primary Test: Sequential Post-Freeze Discovery and Replication

• 81.1% contrast-micro F1 versus 34.3% raw, with zero versus 31 wrong-direction calls, across 113 post-freeze replication contrasts.The frozen composite also reaches 63.0% experiment-macro F1 versus 16.0% raw; the same-tier gain is not established, while cross-tier results repeat the correction.Source: §4.2 · p. 6

4.3 Comparison Against Naive Rules, Deployed Evaluators, and Reduced Variants

• Across alternatives, the composite outperforms raw EC and deployed evaluators, while component refits support complementary signals rather than EC-only or metadata-only sufficiency.On the completed overlap, the composite reaches 82.9% F1 versus 71.5% for raw EC; on a separate 157-contrast slice, calibrated EC exceeds the strongest deployed alternative by 25.7 points. On the frozen holdout, the full export reaches 87.3% versus 63.9% for EC-only.Source: §4.3 · pp. 6–7

4.4 Sensitivity to Horizon, Confidence Level, and Clustering

• Directional agreement persists across readout horizons and confidence thresholds; interval abstention removes wrong-direction calls, while clustered gains vary across experiments.Online-CI sign categories agree for 87.8% of 286 contrasts across horizons. Forcing raw-score calls raises wrong-direction calls from eight to 18 on the completed overlap; clustered F1 gains are not uniform.Source: §4.4 · p. 7

4.5.1 Study 1: Multi-turn rollout depth.

• On the 130-contrast suite-design slice, top-1K F1 rises from 68.2% at turn 0 to 81.9% at three simulated turns, while deeper rollout degrades.Four turns maximize P@20% at 94.4%; rollout choices were selected by F1 on the same slice, and a broader matched-slice gain remains unresolved.Source: §4.5.1 · pp. 8–9

4.5 Exploratory Evaluation-Suite Design Studies

• Critical-density selection improves F1 over same-size random samples most at smaller budgets, but gains narrow and disappear when both strategies score the full pool.Reported gains are 33.9 points at K=100, 20.7 at K=250, and 15.3 at K=500; the comparison is within the ranked, loadable pool.Source: §4.5.1 · p. 9

4.5.3 Study 3: Critical-turn versus last-turn suite bundles.

• The critical-turn bundle reaches 66.2% F1 versus 65.3% for last turn, but changes both suite membership and size.The exploratory result guides suite construction rather than independently validating the frozen proxy.Source: §4.5.3 · p. 9

4.6 Secondary Descriptive Tail Ranking for Human Review

• Tail ranking is evaluated as an input to human review, not as evidence for automatic launch approval, full-funnel screening, or traffic savings.On the completed overlap, the most confident half of offline-significant calls covers 32.9% of contrasts at 100.0% observed precision with no wrong-direction calls; the operating curve does not estimate full-funnel performance.Source: §4.6 · p. 9

• 81.1% contrast-micro and 63.0% experiment-macro F1, versus 34.3% and 16.0% for raw EC, with zero versus 31 observed wrong-direction calls in replication.The result covers 113 contrasts from eight post-freeze experiments.Source: §5 · p. 9
• On 78 same-tier rows, the composite shows moderate observed agreement, but its incremental improvement over raw EC is unresolved.The strongest effect is cross-tier, which drives the headline gap.Source: §5 · p. 9
• In deployment, the signal prioritizes which training checkpoints merit experiment slots and which system-prompt variants to promote, mainly in same-tier comparisons.The operational case rests on moderate observed same-tier agreement and zero wrong-direction calls, not the cross-tier correction.Source: §5 · p. 9
• Analysis choices, including the tier split, followed outcome availability; full-funnel screening, traffic savings, future coverage, and subgroup generalization were not measured.Engagement is not user welfare, so the proxy should inform human review rather than automatic launch decisions.Source: §5 · p. 9

• Engagement is a product outcome, not a measure of whether an interaction was good for the person having it.The six-hour return label does not distinguish a useful return from one prompted by an unhelpful interaction.Source: §6 · p. 9
• A system tuned toward engagement can reward unnecessarily prolonged interaction.The proxy's target is session engagement, including whether a user returns within six hours.Source: §6 · p. 9
• The proxy is for prioritizing experiment candidates, not a launch criterion or welfare measure; randomized experiments and human review remain the basis for launch decisions.Source: §6 · pp. 9–10
• The audit covers completed experiments, not full-funnel screening or subgroup generalization; the corpus is not stratified by language, locale, or user population.Reported data are aggregate statistics, not per-user measurements.Source: §6 · p. 10
• Evidence supports frozen-composite prioritization on one assistant in one product setting, not calibrated future coverage, traffic savings, or generalization to other products or engagement definitions.Source: §6 · p. 10

A Evidence Accounting

• Table 5 indexes the analyses by size and allowable conclusion; only the two post-freeze cohorts form the primary sequential test.Source: A Evidence Accounting · p. 12

B.1 Later Live-Snapshot Replication Cohort

• The later cohort includes all 113 eligible contrasts from eight experiments; frozen artifacts were applied unchanged, and bootstrap and LOEO ranges describe observed-cluster sensitivity, not future-experiment coverage.The analysis and tier strata were not fixed in advance.Source: §B.1 · p. 12
• 75.0–89.3% overall, 47.6–83.3% same-tier, and 96.2–100.0% cross-tier are frozen LOEO F1 ranges; same-tier improvement over raw EC remains unresolved.The paired same-tier range includes zero.Source: §B.1 · p. 12

B.2 Evaluation-versus-Materialization Provenance

• All offline evaluations and the frozen map predated treatment exposure, but snapshot extraction and model-tier splits were chosen after outcomes were known.Source: §B.2 · p. 12

B.3 Descriptive Pool Across Both Post-Freeze Cohorts

• The two post-freeze cohorts are not pooled for replication claims; their descriptive combined total is 203 rows in Table 8.Source: §B.3 · p. 12

B.4 Initial Discovery Cohort: Construction and Provenance

• The frozen specification preceded all five discovery-cohort experiments, covering all 90 contrasts without selecting or refitting any analysis component using their outcomes.Source: §B.4 · p. 12
• 90/90 discovery-cohort predictions matched an independent manual reconstruction exactly, although the analysis and its cross-/same-tier strata were not fixed in advance.The recovered features came from immutable pre-exposure outputs; this is failure localization on evidence analyzed after outcomes were known, not a forward-looking test of a predeclared plan.Source: §B.4 · p. 12
• The 90-row discovery cohort is chronologically disjoint from the earlier 286-row overlap, while other analyzed slices overlap those rows and are not independent replications.The 84-row frozen holdout and 211-row same-tier stratum are subsets of the 286 rows; the 254-row reduced-feature universe shares 236 contrasts and adds 18 rows.Source: §B.4 · p. 12

B.5 Per-Experiment Results

• 91.2% experiment-macro F1 for the frozen composite versus 0.0% for raw EC; zero wrong-direction calls describes observed calls, not zero error or a future guarantee.The reported 77.2% pooled F1 uses one fixed decision rule, and the scoring-stability widths omit calibration-fit, tier-feature, and future online residual uncertainty.Source: §B.5 · p. 12

B.6 Directional Decomposition Under Inherited Widths

• Tier direction completely accounts for recovered cross-tier direction in this cohort, while EC adds same-tier calls without detecting the lone same-tier mover.The earlier frozen/reduced component analysis found EC improvement over metadata-only variants, indicating cohort-dependent component value.Source: §B.6 · p. 13
• 97.8% tier-only F1 exceeds frozen-full F1 of 77.2%, but the tier-only row inherits frozen-full widths rather than independently estimated uncertainty.The five cluster sizes are severely imbalanced, and the experiment-level results do not attach a five-cluster uncertainty interval.Source: §B.5 · p. 12, §B.6 · p. 13

B.7 Prior-Data-Only Tier-Band Sensitivity

• A prior-data-only tier-only band with H90 = 3.1937 crosses zero for every discovery row, yielding zero calls despite nonzero tier contributions of magnitude 2.6261.The sensitivity changes actionability: decisive calls come from the full pipeline’s uncertainty construction, not tier direction alone.Source: §B.7 · p. 13

B.8 Support Shift Toward a Known Failure Tail

• All 22 discovery cross-tier raw points exceed the earlier cross-tier raw-point 90th percentile, placing discovery in a previously observed positive-tail failure regime.Earlier calibration improved cross-tier F1 by 16.0 points but same-tier F1 by only 1.0 point; evidence does not show tier dominance in arbitrary future or same-tier cohorts.Source: §B.8 · p. 14

B.9 Alternate Suite Profiles on the Discovery Cohort

• The critical top-1K static profile reduces observed wrong-direction calls from 22 to 3 chiefly by abstaining, with 9 calls versus 32, and makes zero correct decisive calls.These are raw-profile intervals; the frozen composite was not reconstructed for alternate profiles.Source: §B.9 · p. 14

C Supporting Evaluation-Suite Evidence

• The supporting evaluation-suite studies are design evidence, not the paper’s core end-to-end lifecycle evidence.The section also identifies a matched prompt-suite lifecycle cross-check and lessons learned as supporting evidence.Source: C Supporting Evaluation-Suite Evidence · p. 14

C.1 Exploratory Evaluation-Suite Studies: Multi-Turn

• At top-500, critical-density ranking improves interval-aware F1 by 15.3 points over random samples from the same pool, while lowering calibrated H90 by 0.356 and MAE by 0.210.The random-sampling entries average 20 draws, without across-draw or experiment-clustered intervals, so the lifts are descriptive.Source: §C.1 · p. 15, Table 14 · p. 16
• 81.9% versus 68.2% interval-aware F1 on the top-1K suite when moving from static scoring to three-turn rollout; deeper rollout eventually degrades.Four turns yield the best top-tail ranking precision on this slice, so the multi-objective operating region is three to four turns rather than the maximum horizon.Source: §C.1 · p. 15, Table 15 · p. 16
• At top-1K and turn 3, final-turn scoring raises F1 from 81.9% to 83.5% versus averaging through turns, but by turn 8 its F1 drops to 57.5%.Averaging is the more robust default because it preserves earlier evidence as depth grows.Source: §C.1 · p. 16, Table 17 · p. 17
• The chosen three-to-four-turn setting is tuned on the same 130-contrast design slice, without experiment-clustered uncertainty; the 96-contrast lifecycle comparison is descriptive.The slice covers 23 target configurations, 5,000 ranked cases per configuration, up to eight simulated turns, and six experiments.Source: §C.1 · pp. 14, 16
• Across 96 contrasts from four experiments, the matched top-1K three-turn rollout comparison raises interval-aware F1 from 71.6% to 81.7%, while top-2K rollout scores 77.5%.With only four experiments and no clustered interval, the cross-check supports directional consistency rather than a resolved rollout effect.Source: §C.1 · p. 16

C.2 Historical Critical-Turn Versus Last-Turn Bundle

• The critical-turn bundle improves F1 by 0.9 points and narrows H90 by 0.31 using roughly 5,000 rather than 7,000 cases, but does not isolate a causal cutoff effect.The comparison changes both suite membership and size.Source: §C.2 · p. 17, Table 19 · p. 17

C.3 Earlier Broad and Frozen-Holdout Details

• On the earlier holdout, calibration improves contrast-micro F1 by 18.7 points, while composite-proxy macro F1 is 93.1%; these are sensitivity results, not nominal uncertainty claims for four future clusters.The completed-overlap offline score in its native 3-day scale has descriptive experiment-balanced association r = 0.961 with realized 7-day online engagement delta.Source: §C.3 · p. 17

C.4 Historical Calibration Feature-Family Diagnostic

• The pooled calibration family trades a wider H80 half-width of 1.299 for narrower H90 and H95 half-widths of 1.735 and 1.922, versus 0.958, 2.087, and 2.909 for classifier-only.This is a historical tuning diagnostic, not a held-out ablation.Source: §C.4 · p. 17

C.5 Post Hoc Frozen Component Details

• The component analysis was post hoc, and its 63.9% EC-only result is not directly comparable with a separate 78.1% legacy single-signal baseline using a different protocol.The 63.9% row uses no-intercept weighted least squares and a newly selected earlier-data H90; the legacy baseline uses a weighted linear map with an intercept.Source: §C.5 · pp. 17–18
• 82.6% F1 for the reduced composite compares with 63.9% for EC-only and 48.2% for tier-only and auxiliary-only; EC+tier reaches 77.7%.These diagnostic refits support complementarity rather than an EC-only or metadata-only explanation.Source: §C.5 · p. 18

C.6 Post Hoc Fully Nested Reduced-Family Robustness Analysis

• 82.8/64.9/72.8% pooled precision/recall/F1 across 12 origins and 254 contrasts, with 14 wrong-direction calls, in the post hoc reduced-family analysis.Per-origin F1 had a median of 85.7% and ranged from 35.3% to 100.0%; the candidate universe was not prospectively specified.Source: §C.6 · p. 18
• Reduced full was selected at nine origins, auxiliary only at two, and tier only at one; the other four families were never selected.Selection used the prior-only robust-H90 criterion; the counts describe the post hoc selection path, not an uncertainty analysis.Source: §C.6 · p. 18, Table 25 · p. 20
• 79.2% F1 for a globally committed reduced-full family versus 72.8% for prior-only family selection; the separate production-family reference reached 82.0%.The 6.4-point difference is not a pure estimate of selection bias, and the 82.0% reference differs in features and implementation; the snapshot cannot reconstruct the original 32 specifications.Source: §C.6 · p. 18

C.7 Power-Aware Precision

• 94.0% power-aware precision on the completed overlap and 96.6% on the frozen later holdout, versus hard precision of 82.9% and 85.9%.The roughly eleven-point gap reflects decisive calls whose online interval moves in the predicted direction without reaching significance; power-aware precision is a sensitivity analysis, not a probability of correctness.Source: §C.7 · p. 18

C.8 Head-to-Head Comparison Against Other Offline Evaluators

• The head-to-head comparison covers 157 pairwise contrasts across seven experiments and supports relative evaluator ranking, not an independent estimate of absolute predictive validity.It uses deployed per-slice deltas rather than the stability-interval calibrated export; evaluators A and B use approximate OCR-read screenshot net-win-rates.Source: §C.8 · p. 18

C.9 Lessons Learned

• Suite construction can materially change what the engagement classifier observes, while interval-aware calibration changes recall and wrong-direction behavior relative to raw-score intervals.For multi-turn assistants, the suite choices named are rollout construction, critical-turn selection, and token-fit handling.Source: §C.9 · pp. 18–19
• Ranking metrics are more actionable than correlation alone when choosing candidates for scarce online experiment slots.Source: §C.9 · p. 19