Source-linked AI summary
Measuring the Value of World-Model Updates: A Counterfactual Utility Protocol for Continual Adaptation
Anqi Peter Li, Kaden Kim
TL;DR
Continual world models lack a direct measure of whether an individual update improves control relative to holding the deployed model fixed. The fork ledger supplies that matched counterfactual comparison, and the study finds that one fixed update mechanism lowers return across three simulated tasks while enabling triggers to be judged by the updates they select.
Problem
Continual adaptation methods specify when to update but do not reveal whether an individual update improves control relative to holding the same deployed model fixed.
Method
The fork ledger branches deployment streams at pre-registered decision points into matched UPDATE and HOLD continuations under common random numbers, then scores pre-fork triggers against ΔR labels.
Results
A fixed update mechanism lowers return on CartPole and Walker and is unresolved on Cheetah, with the effect varying by task and drift direction.
Takeaways & Limitations
The ledger makes counterfactual utility observable for a fixed mechanism, allowing triggers to be evaluated by the updates they select rather than detection alone.
Takeaways & Limitations
The corrected design deviates from the registered threshold procedure, and trigger rankings are not shown to transfer across tasks.
Abstract
from arXiv · showhide
Continual world models must decide whether new data justify changing the model. Fixed replay schedules and prediction-error triggers specify when to update, but neither reveals the value of an individual update: one deployment run cannot show how the same model would have performed at that moment had it held its parameters. We introduce the fork ledger, which branches a deployment stream at pre-registered decision points into matched update and hold continuations under common random numbers. It evaluates both continuations on the same episodes and records $ΔR = R_{\mathrm{update}} - R_{\mathrm{hold}}$. Always applying one fixed update mechanism lowers return on all three simulated control tasks: CartPole ($-144.0$; checkpoint-bootstrap $95\%$ CI $[-185.4,-116.1]$, against a converged return near $650$), Walker ($-82.8$; $[-101.1,-61.7]$) and Cheetah ($-18.6$; $[-29.0,-6.6]$). Divergence is an outcome of applying the update, so the estimand counts every attempted fork; restricted to the $693$ of $720$ that did not collapse, CartPole and Walker are unchanged in sign ($-113.4$ and $-82.1$) and Cheetah becomes unresolved ($-3.9$; $[-17.5,+13.0]$). The task is the unit of inference: each contributes $240$ attempted forks over five pretrained checkpoints crossed with two drift directions. The ledger makes counterfactual utility observable for a fixed mechanism, allowing triggers to be judged by the updates they select rather than by surprise detection alone.
1 Introduction
The fork ledger addresses the missing counterfactual value of an individual model update by comparing matched update and hold continuations. In the study, one fixed update mechanism lowers return on CartPole and Walker, while Cheetah is unresolved across drift directions.
- 1 Introduction: The fork ledger compares return after updating with the same deployed model’s counterfactual return had it held its parameters.It branches deployment streams at pre-registered decision points into matched UPDATE and HOLD continuations.
- 1 Introduction: A fixed update mechanism lowers return on CartPole and Walker, while Cheetah depends on drift direction and remains unresolved.The study makes a measurement claim and a task-bounded empirical claim rather than proposing a universal update rule.
- 1 Introduction: The protocol assigns a realized value to each individual update and evaluates triggers against those matched update-value labels.This shifts evaluation from detecting surprising data to judging the updates selected by a trigger.
2 Related Work
Related work studies when models are unreliable or should adapt using novelty, residual, change-point, and retraining signals. These approaches generally evaluate detection or aggregate behavior rather than matched per-update control value.
- 2 Related Work: Existing methods use novelty, residual, consistency, model-shift, change-point, and retraining signals to decide when models should adapt.The cited lines of work address model reliability, event-triggered learning, concept drift, and retraining decisions.
- 2 Related Work: Prior evaluations emphasize detection quality, aggregate downstream behavior, or forecasted model quality instead of matched per-update control contrasts.The fork ledger is positioned as a direct comparison against these evaluation targets.
3 Method
The method deploys a pretrained world model on controlled drift streams, freezes a reference stream, and forks it into matched update and hold evaluations. It scores a fixed intervention using common-random-number counterfactual labels and evaluates pre-fork trigger signals under a corrected design.
- 3 Method: The deployment stream distinguishes volatility in physics parameters from stochasticity in executed-action noise.Volatility is an environmental change an update can track in principle, whereas stochasticity raises transition uncertainty without changing the environment.
- 3 Method: Control uses latent-space MPC with CEM and a terminal value bootstrap, so updated world-model behavior is evaluated without actor retraining.MPC consults the current model at decision time rather than relying on an actor trained in the old latent space.
- 3 Method: The fixed update applies N gradient steps to the most recent W transitions and rehearses no pre-deployment data.The resulting labels price no-rehearsal fine-tuning of a converged model.
- 3 Method: At each pre-registered decision point, the protocol checkpoints a deployment stream and evaluates UPDATE and HOLD branches on matched episodes.Both branches share initial states, injected action noise, and planner sampling noise, while the physics perturbation is fixed at the fork.
- 3 Method: The score ΔR is the causal effect of applying the entire fixed intervention now versus not applying it, including timing, dose, window, and induced model change.Triggers use only pre-fork error-stream features and are scored against fixed ΔR labels; post-fork HOLD return is excluded.
- 3 Method: Corrected cells use 20,000-step drift periods, forks every 250 control steps, five checkpoint clusters, and m = 30 matched evaluation episodes per branch.Signal comparisons aggregate by checkpoint-cluster bootstrap rather than pooling raw forks; stochasticity is not varied in the corrected arm.
- 3 Method: The study uses DeepMind Control Suite CartPole, Walker, and Cheetah environments with a compact DreamerV3-style recurrent state-space model.The model is trained from scratch on clean dynamics before deployment on the perturbed streams.
4 Results
Across the corrected multi-task evaluation, the fixed update mechanism lowers return on CartPole and Walker, while Cheetah is unresolved after excluding divergence. Update value varies with task, drift direction, and frozen-branch competence, and trigger rankings do not reliably transfer across tasks.
- 4.1 A fixed update rule loses return on all three tasks: −144.0 on CartPole, −82.8 on Walker, and −18.6 on Cheetah are the all-attempted mean effects of the fixed update mechanism.The Cheetah retained estimate becomes unresolved after excluding diverged forks, whereas CartPole and Walker remain negative.
- 4.1 A fixed update rule loses return on all three tasks: −208.6 to −253.4 in the lowest drift quartile versus −8.4 to +53.0 in the highest shows CartPole harm varies with drift level.The two drift directions traverse the range in opposite time order, so the pattern is not explained by stream position.
- 4.3 Why the update’s value varies: a registered mechanism, falsified: +55 below RHOLD 350 becomes −73, −149, and −240 as frozen performance improves, so the update is harmful across most observed CartPole decisions.The exploratory fit has slope 0.27 and crosses the identity at RHOLD = 397.
- 4.3 Why the update’s value varies: a registered mechanism, falsified: −82.1 on Walker and −3.9 on Cheetah show that the crossing is task-dependent rather than a transferable constant.Walker is negative under both drift directions, while Cheetah changes sign with drift direction; the per-task crossing analysis was post hoc.
- 4.4 No fixed-rate schedule beats never updating on CartPole: −82.7 to −114.1 across fixed update doses shows no monotone improvement with N, so no fixed-rate CartPole schedule beats never updating.The apparent decline in loss per gradient step is attributed to the increasing denominator, not to better updates.
- 4.5 Trigger rankings do not demonstrably transfer across tasks: CartPole trigger policies select less damaging moments, but Walker rankings are undefined and Cheetah’s apparent gains disappear when divergence exclusions are removed.The evidence supports a protocol for testing triggers and a single-task demonstration, not a universal or cross-task ranking.
5 Discussion
The fork ledger measures realized value by comparing matched update and hold branches, exposing whether a trigger selects beneficial updates rather than merely detecting change. Its scope is bounded: results cover fixed mechanisms and task-level contrasts, not continually adapting deployment policies or population-wide estimates.
- 5 Discussion: The ledger compares policies by return rather than detection rate, assigning realized value to each selected update.
- 5 Discussion: The sign of update value varies with task and drift direction, so Walker and Cheetah test whether CartPole trigger ordering transfers.
- 5 Discussion: The evaluation instrument uses HOLD as the continuing reference and does not measure committed continual adaptation.A committing variant is identified as the natural successor instrument.
- 5 Discussion: −7,128 versus +2,198 return units separates a rate-matched random rule from the residual trigger under the same 14,000-gradient-step intervention.Matched counterfactual branches isolate whether the rule selected better update moments.
- 5 Discussion: Table 3 compares pre-fork trigger signals across corrected task sweeps with checkpoint-clustered AUCs and task-specific, non-pooled policy totals.Walker brackets are min–max ranges over only two checkpoint clusters containing positive forks.
A.1 Primary protocol
The primary protocol fixes one update response and evaluates matched branches across pretrained checkpoints, drift directions, and controlled stream randomness.
- A.1 Primary protocol: The corrected primary arm applies N = 200 gradient steps to W = 10,000 recent transitions and evaluates m = 30 matched episodes per branch.
- A.1 Primary protocol: The grid crosses five pretrained checkpoints with two drift directions, producing 10 cells and 24 attempted forks per cell.The drift period is 20,000 steps.
- A.1 Primary protocol: Each paired continuation pins the initial state, injected-action-noise stream, and recurrent-model sampler.The model is pretrained only on clean dynamics.
A.2 Deviations from the registered design
The corrected evaluation diverged from the registered design in its stochasticity ramp, trigger set, thresholding procedure, and primary endpoint reporting.
- A.2 Deviations from the registered design: The registered episode ramp was not executed: every corrected-arm ledger uses m = 30 at σa = 0, leaving the stochasticity axis unexercised.
- A.2 Deviations from the registered design: The corrected arm scores four registered trigger families plus an unregistered pre-fork deployment-return signal and the ΔR oracle ceiling.
- A.2 Deviations from the registered design: The registered causal running-quantile threshold rule was not used; corrected thresholds were fit leave-one-checkpoint-out on pre-fork signals.
- A.2 Deviations from the registered design: The registered primary endpoint is not reported, while periodic schedules have results only on quarantined legacy ledgers.
A.3 Policy and trigger totals
On CartPole, always updating is harmful, whereas residual and return-deficit triggers produce positive task-level totals that remain below the unattainable ΔR oracle.
- A.3 Policy and trigger totals: −26,441 return units is the always-update total relative to never updating on 228 retained CartPole forks.
- A.3 Policy and trigger totals: +2,198 and +2,211 are the leave-one-checkpoint-out totals for residual and return-deficit triggers, respectively.The ΔR oracle totals +3,692.
- A.3 Policy and trigger totals: Residual and return-deficit AUCs are nearly tied on CartPole at .919 and .915, respectively.
- A.3 Policy and trigger totals: Across tasks, AUCs are defined on only two of five Walker checkpoint clusters and overlap on Cheetah.Walker values are min–max ranges rather than intervals.
A.4 Endpoint and mechanism robustness
The corrected CartPole endpoint remains negative across three update doses, while the dose arm was not run on Walker or Cheetah.
- −88.4 at N = 50, −113.4 at N = 200, and −110.1 at N = 400; CartPole remains negative across all tested update doses.The reported checkpoint-bootstrap 95% intervals exclude zero for each dose.
A.5 Measurement error, plasticity, and the dose interval
Robustness checks support the negative CartPole and Walker findings while leaving Cheetah unresolved. The analyses also bound measurement and pinning concerns, though bootstrap coverage is optimistic and some sensitivities remain limited in scope.
- 0.98 versus 0.47 and 1.00 versus 0.44; split-episode controls do not reproduce the observed contraction or variance reduction.The null diagnostic uses independent episode halves, whereas the observed ratios come from divergence-retained forks.
- [−146.7, −80.0] and [−111.7, −52.5] preserve negative CartPole and Walker verdicts, while Cheetah remains unresolved under Student-t intervals.The corresponding percentile-bootstrap verdicts have the same sign pattern.