PaperlayerView exact revision on arXiv

Source-linked AI summary

The Predictive--Optimization Gap in GuanDan Self-Play

Qicheng Zou, Biao Yuan, Siyu Song, Wentao Liu, Xiaokun Wang

arXiv:2610.29411v1cs.LGstat.ML

TL;DR

Outcome-predictive signals may describe progress without providing a reliable optimization direction. This study tests that distinction in GuanDan by adding changes in a frozen match-win predictor to round-level self-play supervision, finding predictive value but no consistent method-level gain.

  • Problem

    Predictive validity and downstream optimization utility are distinct, but intermediate outcome-predictive supervision is often treated as if one implied the other.

    Source: §I · p. 1

  • Method

    The authors test held-out state–gain predictive value and add changes in a frozen public-state match-win predictor to a matched DanZero-style round-level self-play target.

    Source: §I · pp. 1–2, B. Frozen Predictor · p. 3

  • Results

    State–gain interactions improve held-out log loss, yet progress augmentation shows no reliable method-level effect across ten seeds despite a successful selected checkpoint.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    Under the studied self-play protocol, predictive meaning does not imply optimization reliability; identifying mechanisms for reproducible gains remains an open question.

    Source: §VII · p. 10

    The experiment covers one game and one round-broadcast, executed-action DMC learner, and ten seed pairs are a modest sample for characterizing training-run variation.

    Source: B. Scope and Limitations · p. 10

Abstract

from arXiv · show

Outcome-predictive signals are often treated as promising optimization targets in reinforcement learning, but predictive validity and optimization utility are different claims. We expose this distinction in GuanDan, a multi-round imperfect-information game in which the implication of a round outcome depends on the public match state. Held-out analysis shows that state--gain interactions provide incremental predictive value beyond a constant signed-gain effect, establishing that match progress is predictively meaningful. We then use changes in a frozen match-win predictor, $\Delta\Phi$, as a controlled round-level supervision intervention in a matched DanZero-style Monte Carlo self-play protocol. The intervention can succeed in an individual lineage: a selected progress-augmented checkpoint substantially outperforms its matched Baseline on a locked evaluation. Yet the method-level result reverses this impression: none of five preregistered replication seeds is positive, the pooled ten-seed interval includes zero, and across-iteration effects repeatedly change sign without a common late-stage advantage. These results identify a predictive--optimization gap: under the studied learner, a signal can describe outcome-relevant progress while failing to provide a reliably beneficial optimization direction.

I. INTRODUCTION

This study separates whether match context makes round gains predictive of victory from whether adding that context produces reliable policy improvement. In a matched self-play intervention, predictive validity and selected-checkpoint success coexist with unsuccessful method-level replication.

I. INTRODUCTION

  • In GuanDan, a round gain's implication for match victory can depend on public match state because rounds accumulate toward a match winner.The game is a four-player, team-based, imperfect-information card game with an evolving public team-level state.Source: §I · p. 1
  • The predictive test asks whether pre-round public state changes the held-out relationship between a fixed signed round gain and eventual victory.The study compares state–gain interactions against a constant-gain specification.Source: §I · p. 1
  • The intervention adds changes in a frozen public-state match-win predictor, ΔΦ, to a Baseline round label in a DanZero-style Monte Carlo protocol.Both arms share initialization, architecture, learner, budget, and evaluation protocol; β controls the match-progress contribution.Source: §I · p. 1
  • Held-out state–gain interactions improve log loss over a constant-gain specification, establishing predictive value without guaranteeing optimization reliability.The paper frames predictor validity and across-run optimization evidence as distinct empirical obligations.Source: §I · pp. 1–2
  • 0.143 MeanDelta: the selected seed-1 Progress-10 checkpoint beat its matched Baseline, but none of five preregistered seeds was positive, the pooled interval included zero, and trajectories showed no common late-stage advantage.The comparison used a locked 1,000-pair evaluation.Source: §I · p. 2
  • The study tests predictor validity, selected-checkpoint efficacy, and optimization reliability across independent training runs as separate questions.A successful checkpoint demonstrates possibility, but cannot substitute for replication over training randomness.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

The related work situates this study within self-play evaluation, prediction-versus-decision objectives, and intermediate supervision. It motivates evaluating predictor-derived feedback separately from its downstream policy effects.

A. Self-Play in Card Games and Training Reliability

  • Card-game self-play systems address policy learning and game-specific challenges, while reliable evaluation requires uncertainty across independent training runs.Self-play can produce lineage-dependent trajectories; the authors do not claim strategic cycles are identified in GuanDan.Source: A. Self-Play in Card Games and… · p. 2

B. Predictive Validity and Decision Utility

  • Predictive loss need not track downstream decision quality, and adaptive policies can change the data distribution on which later learning depends.The paper tests this distinction when a validated predictor-derived signal enters an online self-play regression target.Source: B. Predictive Validity and Decision… · p. 2

C. Outcome-Potential and Intermediate Supervision

  • Temporal credit-assignment methods redistribute delayed outcomes, while outcome-potential methods convert predictions of eventual success into intermediate feedback.This study applies that conversion at completed-round boundaries and evaluates predictive content separately from optimization reliability.Source: C. Outcome-Potential and Intermediate… · pp. 2–3

D. Policy Guarantees and Causal Credit

The paper separates predictive progress from causal credit and policy guarantees, then uses a matched intervention to test whether round-level progress supervision helps self-play optimization. Its targets broadcast a shared label to executed actions within each round, without attempting action-level attribution.

D. Policy Guarantees and Causal Credit

III. DIAGNOSTIC INTERVENTION

  • The diagnostic intervention matches initialization, architecture, learner, budget, and evaluation protocol while changing the supervision target supplied to the learner.The design tests whether a predictively meaningful state–gain signal becomes optimization-reliable under the specified round-broadcast, executed-action regression mechanism.Source: §III · p. 3

A. Problem and Round-Level Targets

  • GuanDan's action → round → match hierarchy motivates assigning eventual match-outcome information to rounds, rather than attempting within-round action-level attribution.Every executed action in a round shares its round-level label, and the final match outcome is broadcast to all rounds after the match ends.Source: A. Problem and Round-Level Targets · p. 3
  • For a fixed signed gain and final match outcome, the baseline label is state-invariant, while the progress-augmented target adds a frozen predictor's change in estimated win probability across the round.The added term is intended to expose state-dependent outcome relevance as a round-local teacher, not to supply missing state access or causal attribution.Source: A. Problem and Round-Level Targets · p. 3

B. Frozen Predictor

The paper fits a public-state match-win predictor independently of policy training, calibrates it, and freezes it as a policy-distribution-conditional win-probability teacher.

B. Frozen Predictor

  • The predictor estimates match-winning probability from information publicly available at a round boundary and is not a policy-training value function or online critic.It is trained on complete-match trajectories, calibrated, and frozen before reward-training experiments.Source: B. Frozen Predictor · p. 3
  • Freezing the predictor separates match-progress estimation from policy optimization and prevents reward annotations from moving with the self-play policy.It is not updated from subsequent self-play trajectories.Source: B. Frozen Predictor · pp. 3–4
  • The predictor uses a 27-dimensional public-boundary representation and an ℓ2-regularized logistic model with scalar Platt calibration on validation data.Partitions are made at the complete-match level; metrics are evaluated once on untouched test matches, with no separate calibration partition.Source: B. Frozen Predictor · p. 4

C. Self-Play Optimization

Self-play proceeds through repeated collect–annotate–train iterations, using a frozen predictor to provide round-level progress labels for executed-action regression. The study also describes a complete-match dataset collected for both predictor and reward experiments.

C. Self-Play Optimization

  • The progress term shifts within-round residuals according to public match context, changing residual structure and teacher locality rather than merely rescaling the objective.The authors do not claim that this necessarily improves signal-to-noise ratio in every stratum.Source: C. Self-Play Optimization · p. 4
  • Each iteration collects new complete matches, annotates round boundaries with frozen-predictor progress, broadcasts each round label to its executed actions, and trains the next policy for one epoch.The predictor remains frozen, and optimizer state is reset between iterations.Source: Figure 1 · p. 4, C. Self-Play Optimization · p. 4

A. Complete-Match Dataset

  • The formal dataset contains 2,000 complete matches with zero aborted matches, 24,636 rounds, and 2,398,084 executed decisions.Matches average 12.3 rounds, with a 95th percentile of 17 and a maximum of 24.Source: A. Complete-Match Dataset · p. 4
  • Dataset partitions are defined by match_id, so no match contributes boundaries to more than one partition.Each boundary record includes public states before and after the round, signed level gain, final match outcome, and team-relative match information.Source: A. Complete-Match Dataset · pp. 4–5

B. Stage I: Testing Predictive Meaning of State–Gain Interactions

The predictive test compares a constant signed-gain effect with a model that adds preregistered interactions between gain and pre-round public state. The frozen predictor is developed separately, with its performance assessed on untouched test matches.

B. Stage I: Testing Predictive Meaning of State–Gain Interactions

  • The state-dependent model adds preregistered interactions between signed round gain and pre-round match-state variables to a shared constant-gain effect.Only pre-round variables enter the model, excluding variables determined by the round outcome to avoid post-treatment leakage.Source: B. Stage I: Testing Predictive Meaning… · p. 5
  • The nested models are evaluated on held-out complete matches using log loss and Brier score as primary metrics, with AUROC and calibration diagnostics secondary.Confidence intervals use match-level resampling, keeping team-relative views from each match in the same cluster.Source: B. Stage I: Testing Predictive Meaning… · p. 5

C. Constructing the Frozen Predictive Probe

  • The predictor is trained on the complete-match training split, calibrated on validation, and evaluated on untouched test matches before being frozen.Predictor development and calibration use validation; the test matches are not accessed for training or calibration.Source: C. Constructing the Frozen Predictive… · p. 5

D. Stage II: Matched Self-Play Optimization Probe

The matched self-play probe compares Baseline and progress-augmented supervision under shared training conditions, then evaluates playing strength with paired team swaps. Its design separates scale selection and checkpoint confirmation from the locked final evaluation.

D. Stage II: Matched Self-Play Optimization Probe

  • All arms share a frozen Parent, network, learner, exploration schedule, budget, duration, and evaluation seeds, while progress scales target approximately 10% or 20% of reference target RMS.The Parent descends from human-data supervised initialization and pre-experiment self-play; human replay is disabled during formal training.Source: D. Stage II: Matched Self-Play… · p. 5
  • Ten Baseline–Progress-10 seed pairs cover 50,000 completed training matches; Progress-20 remains limited to the initial five-seed scale-selection stage.Each lineage trains for eight iterations of 250 matches; the initial experiment includes five seeds per arm.Source: D. Stage II: Matched Self-Play… · p. 5
  • Progress-10 was selected after the initial five-seed comparison; its scientific checkpoint was confirmed on disjoint paired seeds before a locked 1,000-pair final holdout.Progress-20 did not receive a final-holdout evaluation.Source: D. Stage II: Matched Self-Play… · p. 5

E. Paired Team-Swap Evaluation

  • Each evaluation seed pairs two matches with team assignments swapped, using the candidate’s outcomes in both assignments.The paired outcome is +1 for a double win and −1 for a double loss.Source: E. Paired Team-Swap Evaluation · p. 6
  • When individual matches are binary, MeanDelta equals 2 WinRate − 1, and inference uses paired seeds because the team-swapped matches are coupled.Source: E. Paired Team-Swap Evaluation · p. 6

F. Criterion for Optimization Reliability

Optimization reliability is assessed across independent training runs, not just games played by one selected checkpoint. The section also specifies replication evaluation and read-only audits of replication, selection stability, and training trajectories.

F. Criterion for Optimization Reliability

  • Optimization reliability is evaluated across independent training runs because more games for one fixed checkpoint do not measure variability from retraining.The policy and data distribution co-evolve during self-play, making lineage-level effects the relevant object.Source: F. Criterion for Optimization… · p. 6
  • For seeds 6–10, a fresh 1,000-pair pool supplies the primary replication inference; pooling all ten seeds is a prespecified secondary summary.The report includes per-seed effects, mean, median, positive-seed fraction, and a training-seed uncertainty interval.Source: F. Criterion for Optimization… · p. 6
  • Read-only audits re-evaluate five matched checkpoints on 500 pairs and assess selection stability among confirmation-eligible candidates on a separate 500-pair pool.The audits do not replace frozen scientific checkpoints.Source: F. Criterion for Optimization… · p. 6
  • A post-hoc diagnostic tracks direct effects across iterations for ten lineages, summarizing mean trajectories, late contrasts, and per-seed slopes.Each seed-iteration estimate uses a fresh 200-pair pool; the analysis does not select or replace checkpoints.Source: F. Criterion for Optimization… · p. 6
  • A five-seed scale/shuffle ablation compares Baseline, scale-matched Baseline, shuffled progress, and true Progress-10 as exploratory mechanism diagnostics, not equivalence tests.Source: F. Criterion for Optimization… · p. 6

G. Diagnostics and Statistical Reporting

The analyses test whether state–gain interactions add held-out predictive value, check the frozen predictor, and compare progress scales before final-holdout evaluation. Diagnostic measures supplement rather than replace complete-match paired evaluation.

G. Diagnostics and Statistical Reporting

  • Optimization-health checks include Q-values, action-score dispersion, target scale, parent divergence, illegal actions, and predictor out-of-support rates.These diagnostics do not replace complete-match paired evaluation as the primary measure of playing strength.Source: G. Diagnostics and Statistical Reporting · p. 6

A. Predictive Meaning: State–Gain Interactions Improve Held-Out Fit

  • −0.00476 M1-minus-M0 held-out log loss change (95% CI [−0.00766, −0.00201]) supports added predictive value from state–gain interactions.The comparison covers 9,568 team-relative boundaries from 390 held-out matches; Brier and AUROC show no corresponding reliable improvement. Figure 2 further shows state-dependent progress, strongest at larger gains; Near-A is an overlapping slice.Source: A. Predictive Meaning: State–Gain… · pp. 6–7, Figure 2 · p. 7

B. Measurement Check: The Frozen Predictor Tracks Match Outcomes

C. Optimization Probe: Initial Scale Selection

  • +0.034 Progress-minus-Baseline aggregate MeanDelta for Progress-10 exceeded Progress-20’s +0.004; Progress-10 was positive for three of five seeds and selected before holdout access.Parent-relative aggregates were 0.120 for Baseline and 0.154 for Progress-10.Source: Table IV · p. 7, C. Optimization Probe: Initial Scale… · p. 7

D. Optimization Reliability Breaks Across Training Seeds

Across training seeds, Progress-10 does not show a reliable positive method-level effect, despite positive effects in some confirmation lineages. Replication and selection audits preserve substantial lineage variation.

D. Optimization Reliability Breaks Across Training Seeds

  • Three of five original confirmation effects were positive, but their mean was 0.034 (95% bootstrap interval [−0.080, 0.147]), showing seed heterogeneity.This interval does not establish a positive expected method-level effect.Source: D. Optimization Reliability Breaks… · p. 8
  • None of five preregistered replication seeds is positive (mean −0.056, 95% CI [−0.083, −0.030]); the pooled ten-seed mean is −0.030 (95% CI [−0.084, 0.025]).The pooled interval includes zero and does not support a reliable positive method-level gain.Source: D. Optimization Reliability Breaks… · p. 8
  • A fresh 500-pair replication preserved the original sign pattern for four of five seeds, and cross-fitted selection retained negative effects for seeds 3 and 5.Table VI reports seed-level effects from confirmation, independent replication, and the cross-fitted checkpoint-selection audit.Source: D. Optimization Reliability Breaks… · p. 8, Table VI · p. 8
  • Most checkpoint selections were stable, although Progress-10 seeds 3–4 and Baseline seed 2 were less stable.Five-fold cross-fitting nevertheless preserved the negative Progress-10 effects for seeds 3 and 5.Source: D. Optimization Reliability Breaks… · p. 8

E. A Successful Checkpoint Does Not Establish Reliability

A selected Progress-10 checkpoint substantially outperforms its matched Baseline, but diagnostics across seeds do not establish a reliable Progress-specific advantage. Selection, training horizon, and the exploratory scale/shuffle ablation do not resolve the heterogeneous effects.

E. A Successful Checkpoint Does Not Establish Reliability

  • Progress-10 achieved MeanDelta 0.143 (95% CI [0.103, 0.183]) against its matched Baseline on the locked 1,000-pair evaluation, but this is a selected-lineage result.The checkpoint also achieved MeanDelta 0.254 (95% CI [0.214, 0.294]) against the common DMC Parent.Source: E. A Successful Checkpoint Does Not… · p. 8

F. The Gap Is Not Explained by Selection or Training Horizon

  • Post-hoc audits found no simple measured mechanism separating positive and negative lineages, while leaving unmeasured multi-step self-play interactions possible.Measured predictor scores, reward scale, action-ranking statistics, and one-step rollout comparisons did not individually explain the seed pattern.Source: F. The Gap Is Not Explained by… · p. 8
  • Across ten matched seed pairs, Progress-minus-Baseline trajectories repeatedly crossed zero, and their peaks occurred at different iterations.The mean iteration-4-to-7 contrast was −0.0175 (95% CI [−0.1160, 0.0925]); the simultaneous band over eight inspected horizons included zero. Figure 4 shows the ten-seed trajectories and the direct treatment-effect comparison in Panel C.Source: Figure 4 · p. 9, F. The Gap Is Not Explained by… · p. 9
  • Both Progress-10 and Baseline improved relative to Parent, but their direct comparison showed no corresponding late-stage Progress-specific improvement.Longer training may increase absolute policy strength without resolving the absence of a reliable relative treatment effect.Source: F. The Gap Is Not Explained by… · p. 9
  • The five-seed scale/shuffle ablation found no stable semantic, scale, or noise advantage, and its wide intervals do not establish equivalence.The true-versus-shuffled contrast was −0.032 (95% interval [−0.182, 0.117]).Source: F. The Gap Is Not Explained by… · p. 9

A. Predictive Meaning Is Not Optimization Reliability

State–gain interactions improve held-out predictive value, but the studied self-play protocol does not produce a reproducibly positive optimization effect. The authors bound this conclusion to their GuanDan, round-broadcast, executed-action DMC construction.

A. Predictive Meaning Is Not Optimization Reliability

  • State–gain interactions improve held-out log loss, while the preregistered replication mean is negative and pooled ten-seed uncertainty includes zero.Relative training trajectories repeatedly change sign, so the signal does not meet the study’s criterion for a reproducibly positive optimization effect under its DMC protocol.Source: A. Predictive Meaning Is Not… · pp. 9–10
  • The seed-1 checkpoint demonstrates that progress augmentation can yield a substantially stronger policy in one lineage, not an unconditional expected effect.Selected-checkpoint efficacy and method-level reliability answer different questions: possibility conditional on one path versus whether a new run should benefit.Source: A. Predictive Meaning Is Not… · p. 10
  • The authors recommend validating predictive information and independently replicating training seeds to test whether a learner converts it into reliable policy gains.Neither predictor metrics nor one selected checkpoint’s locked evaluation replaces seed replication.Source: A. Predictive Meaning Is Not… · p. 10

B. Scope and Limitations

  • The conclusion is bounded to one game and one round-broadcast, executed-action DMC learner; it does not establish divergence under other constructions.Ten seed pairs remain a modest sample, and the diagnostics do not identify which random source determines lineage benefit.Source: B. Scope and Limitations · p. 10

VII. CONCLUSION

  • The paper concludes that predictive meaning does not imply optimization reliability under the studied self-play protocol.It leaves identifying mechanisms that convert valid predictive information into reproducible gains as an open question.Source: §VII · p. 10

Related papers

Loading 2610.29411v1…