Source-linked AI summary
When Validation Stops Learning: Auditing Update Admission for Continual Embodied Agents
Qinzhen Ma, Ruihai Wu
TL;DR
Continual-learning evaluation must balance error control with the useful updates that survive admission at a stated interaction budget. The paper develops an admission-audit protocol with paired confidence checks, certified reference promotion, and an opportunity metric; in a synthetic pushing diagnostic, paired checks preserved updates that a range-based gate rejected, although replay learned better in closed-loop runs. Physical-robot and VLA validation remain open.
Problem
Independent evaluation can reject harmful policy updates but can also prevent useful continual learning, so admission must assess both errors and retained learning opportunities.
Method
The paper combines rule-specific feasibility analysis, paired-binomial admission bounds, certified historical-reference promotion, and a round-level missed-opportunity audit.
Results
At B = 2,000, paired checks admitted 81/256 updates while Hoeffding admitted none, but replay obtained higher final success in closed-loop runs.
Takeaways & Limitations
Fewer observed violations do not necessarily indicate a better continual learner; admission quality must include useful update opportunities and evidence cost.
Takeaways & Limitations
Physical resets, distribution drift, affordable broad task coverage, and physical-robot and VLA validation remain unresolved.
Abstract
from arXiv · showhide
Independent evaluation can reject harmful policy updates yet also prevent useful continual learning. We argue that update admission must be assessed through both error control and retained learning opportunities at a stated interaction budget. We identify a concrete failure: a range-based confidence gate cannot certify unchanged old-task behavior within otherwise substantial budgets. A standard paired-binomial construction reduces this burden when outcome disagreements are rare. We also specify certified historical-reference promotion and a round-level missed-opportunity metric. In a constructed one-step pushing diagnostic with 32 seeds, fresh paired checks admit 31.6% of a common update stream at 2,000 episodes per stage, versus zero for the range-based gate; unconditional replay nevertheless learns better in closed-loop runs. A separate learned-dynamics stress test distinguishes model bias from feedback-selection error. The contribution is an admission-audit protocol with analytical and synthetic evidence; physical-robot and VLA validation remain open.
1 Position: audit opportunities as well as accepted updates
Continual-learning audits should measure both harmful admissions and useful update opportunities, because a gate that accepts nothing can hide stalled learning. The paper frames admission as a process distinct from proposal and screening.
- A gate that accepts nothing may report no harmful admissions while preventing continual-learning progress.
- The proposed audit evaluates retained learning opportunities alongside errors and the full cost of obtaining evidence.
- The unit of analysis is a policy-update pool with historical obligations, extending risk–coverage analysis beyond ordinary selective prediction.
- Admission follows proposal and screening, using fresh evidence while keeping all-candidate audit outcomes outside the learning loop.
2 An admission contract that can be checked within budget
The admission contract compares candidate and reference policies on fixed tasks using fresh paired outcomes and explicit error budgets. Paired-binomial bounds can certify unchanged behavior with far fewer samples than a range-based Hoeffding gate, while references are promoted only after certified nondegradation.
- 2 An admission contract that can be checked within budget: With m = 3, t = 1, δ = ϵ = 0.05, the range-based gate requires 4,229 pairs per comparison and 25,374 episodes, exceeding a 20,000-episode cap.This is a limitation of the rule, not an information-theoretic lower bound.
- 2 An admission contract that can be checked within budget: Paired Clopper–Pearson bounds reduce unchanged-skill retention to n ≥ 117 pairs in the example when both disagreement counts are zero.The rule accepts only when all lower bounds meet thresholds, rejects when any upper bound fails, and otherwise defers.
- 2 An admission contract that can be checked within budget: The contract assumes i.i.d. within-task pairs, frozen policies, fixed-size batches, no optional stopping, and fresh evidence for retries.Its assurance excludes future distribution drift, unprotected tasks, and physical hazards.
- 2 An admission contract that can be checked within budget: Historical references are promoted only when their valid lower bound is nonnegative, protecting certified improvements rather than an unknown historical optimum.Active-reference eviction ends protection, and the guarantee concerns the certified sequence on fixed task distributions.
- 2 An admission contract that can be checked within budget: Each attempt commits candidates, references, required tasks, counts, and error allocation before drawing fresh pairs; insufficient evidence causes deferral.Acceptance requires all lower bounds to meet thresholds, while rejection requires an upper bound to fail.
3 Executed diagnostics and an opportunity-level audit
Synthetic diagnostics show that interval choice determines whether valid updates survive admission, but closed-loop replay can still learn better. A separate stress test shows that fresh proxy draws do not remove selection-score optimism caused by model bias.
- 3 Executed diagnostics and an opportunity-level audit: The missed-opportunity metric counts rounds with at least one valid candidate that no candidate is adopted, avoiding a misleading candidate-level rate.A round with four valid candidates and one adoption has zero misses under this definition.
- 3 Executed diagnostics and an opportunity-level audit: The diagnostic uses matched candidate streams, fresh audit samples, 32 fixed evaluation seeds, and exhaustive finite-population labels after the stream.Audit labels do not train or select policies, and the study is presented as an admission diagnostic rather than resource-optimal learning.
- 3 Executed diagnostics and an opportunity-level audit: At B = 2,000, paired checks admit 81/256 updates while Hoeffding admits none; at B = 20,000, they admit 229/256 versus none for Hoeffding.No observed contract violations occurred for either gate; all gates defer at B = 200.
- 3 Executed diagnostics and an opportunity-level audit: At B = 2,000, paired checks miss 60.4% of available opportunities, while replay obtains higher final success in closed-loop runs.Replay’s 5.9% contract violations reflect insufficient current-task gains rather than retention losses.
- 3 Executed diagnostics and an opportunity-level audit: At c = 0.35 and 256 adaptive queries, held-out proxy success exceeds true success by 40.8 percentage points with reused draws and 41.9 with fresh draws.At c = 0, the corresponding gaps are 0.15 and 0.32 points, showing that fresh draws do not eliminate selection-score optimism.
4 Implications and limits
The evidence supports auditing gate power, certified references, and opportunity counts, but not superiority over replay. Practical validation under physical resets, distribution drift, broad task coverage, VLA updates, and real robots remains open.
- The study supports diagnostic requirements for useful gate power, certified acquired skills, and opportunity-aware auditing rather than superiority over replay.Its simulator has known task identity, easy local adapters, and exact finite-population audits.
- Physical resets, distribution drift, affordable broad task coverage, and validation on LIBERO, VLA updates, and physical tasks remain unresolved.The paper proposes transplanting the same opportunity audit and matched candidate streams to those settings.
A Statistical construction and reference promotion
The paper combines paired-binomial admission intervals with certified historical-reference promotion, while showing that feasibility depends sharply on the interval choice and operational assumptions.
- Coverage construction: Binomial-tail inversion with a union bound provides simultaneous coverage at least 1 − α_t, so acceptance implies every committed contract on the coverage event.The construction composes standard intervals rather than introducing a new concentration result.
- Reference promotion: After acceptance, an old reference is promoted only when its lower bound is nonnegative, yielding a nondecreasing reference sequence on a fixed task distribution.This protects certified improvements but not an unknown historical optimum, and eviction ends protection.
- Rule-specific feasibility: 117 paired samples versus 4,229 Hoeffding samples suffice for a zero-disagreement old-task check when m = 3 and t = 1.With equal allocation at (m, t) = (10, 8), the corresponding episode costs are 4,440 and 170,380.
- Rule-specific feasibility: Neither interval construction is claimed uniformly best, and both can outgrow a fixed budget as protected tasks or verification attempts increase.Their admission coverage also differs from exact binomial inversion.
- Operational conditions: The guarantee requires i.i.d. paired data, frozen policies and scoring, fixed-size batches, no optional stopping, and fresh evidence for retries.It excludes future distribution drift, unprotected tasks, and physical hazards; changing a task distribution requires a new logged contract.
B Executable diagnostic specification
The executable diagnostic uses a synthetic ten-task scalar environment, fixed candidate streams, matched episode caps, and exact finite-bank audits to compare admission procedures.
- Implementation and scope: All demonstrations, dynamics observations, policy updates, and evaluations are synthetic, with no physical-robot trials, pretrained VLAs, or LIBERO runs.The preprint also omits the experimental implementation, configuration files, source hashes, commands, raw decisions, and pilot runs from its LaTeX source package.
- Environment and data: The environment permutes ten mobilities from 0.55 to 1.6 and evaluates one clipped action against perturbed displacement with a 0.06 success tolerance.The scalar model omits contact geometry, inertia, perception, and multi-step planning.
- Candidate generation and replay: Four supplied candidates include regularized current-task fits, a replay mixture, and a task-local update that leaves previous task actions unchanged.The easy local candidate deliberately creates a low-disagreement control, limiting claims about general policy capacity or state-of-the-art robot learning.
- Selection and admission: Common-stream diagnostics hold the source learner fixed while closed-loop runs let each learner branch and update its own references after acceptance.All compared rules receive the same four candidates and matched episode caps.
- Audit and uncertainty: Exhaustive finite-bank evaluation makes contract labels exact for committed candidates, while coverage, violations, missed opportunities, final success, and forgetting are reported separately.The audit is charged separately because it costs more than the largest checking cap.
- Development and evaluation separation: Pilot seeds 0–3 preceded evaluation on seeds 100–131 after expanding the audit bank and correcting proposal, regression, and proxy-cost accounting.The evaluated runs therefore use explicit ridge fits and separated harmful regressions from minimum-gain violations.
C Factorial proxy-feedback stress test
A separate one-step stress test varies dynamics curvature, search strategy, feedback reuse, and query budget to separate model bias from selection optimism.
- Stress-test design: The diagnostic uses true displacement x′ = a − ca^2, a linear-through-origin proxy, and success tolerance 0.06 in a separate one-step environment.It is not a closed-loop continual-learning benchmark.
- Stress-test design: Random and adaptive searches are compared at K ∈ {4, 16, 64, 256} under reused or newly drawn target offsets, across a full factorial design on 32 seeds.The design produces 1,024 recorded searches.
- Results: At curvature 0.35 and 256 adaptive queries, model gaps are 40.8 points with reused feedback and 41.9 points with fresh feedback.At zero curvature, the corresponding gaps are 0.15 and 0.32 points.
- Results: Fresh per-query feedback does not remove selection optimism because adaptive search still favors favorable estimates.Selection gaps at curvature 0.35 are 11.5 and 19.3 points for reused and fresh feedback, respectively.
D Counts, uncertainty, and resource reporting
The diagnostic reports admission coverage, violations, missed opportunities, uncertainty, and resource use across checking rules and budgets. Results show paired checking admits updates where range-based gates do not, while reported costs remain specific to scalar simulation.
- Uncertainty: The evaluation used seed-cluster bootstrap intervals at B = 2,000, while violation counts were exact on finite audit populations and zero counts did not imply zero risk.The uncertainty statement distinguishes interval estimates from finite-population audit counts.
- Closed-loop outcomes: Fresh paired checking achieved 37.1% coverage and 59.6% final success with 0.0 percentage points of forgetting, while fresh Hoeffding achieved 0.0% coverage and 31.4% success.The paired, no-proxy variant was similar, with 37.5% coverage, 59.8% success, and 0.0 percentage points of forgetting.
- Audit outcomes: Paired checking recorded zero old-task violations across 95 accepted rounds, whereas direct fine-tuning recorded 251 old violations across 256 rounds.Replay update recorded zero old violations but 15 contract violations across 256 rounds.
- Common-stream results: At 2,000 episodes, paired checking accepted 81 common-stream rounds versus zero for Hoeffding and empirical Bernstein, with 152 missed opportunities.At 20,000 episodes, paired checking accepted 229 rounds with four missed opportunities; empirical Bernstein accepted 195.
- Resource reporting: The complete recorded evaluation contained 1,536 settings and 12,288 decisions, ran for 50.9 seconds on CPU, and cannot estimate robot or VLA costs.Candidate audits excluded 368,640 additional checkpoint outcomes per stream, and replay never replaced fresh checks.