Admission Without Answers: Label-Free Certification and Experience Learning for LLM-Based Optimization Modeling

Junbo Jacob Lian; Huiling Chen; Hanzhang Qin; Chung-Piaw Teo

Source-linked AI summary

Admission Without Answers: Label-Free Certification and Experience Learning for LLM-Based Optimization Modeling

Junbo Jacob Lian, Huiling Chen, Hanzhang Qin, Chung-Piaw Teo

arXiv:2608.15565v1Aug 16, 2026cs.AI

TL;DR

Real ticket streams lack answer keys for admitting reusable optimization-modeling skills. AdmitOR uses calibrated cross-family behavioral agreement over resampled instances, achieving 0.927 admission precision and the strongest downstream library results with the fewest items.

Problem
Real ticket streams lack answer keys, making answer-based admission of reusable optimization-modeling skills unavailable.
Method
AdmitOR calibrates a cross-family clique of value-function agreement over resampled instances to a false-discovery target with abstention.
Results
0.927 admission precision is achieved, with the highest downstream accuracy and fewest library items among the compared judges.
Takeaways & Limitations
Label-free admission can support downstream skill learning without making answer keys a precondition for performance.
Takeaways & Limitations
If every model family shares the same incorrect interpretation, resampling cannot detect the resulting misinterpretation.

Abstract

from arXiv · show

Experience-learning agents for optimization modeling improve by storing verified skills, but existing learners admit knowledge by checking against known answers, which real ticket streams do not provide. The natural label-free alternatives are unreliable: on a 300-problem label-blind stream, admitting every executable model poisons roughly one admission in four, while single-instance agreement accepts models that match at one value but differ elsewhere. We propose AdmitOR, an admission gate built on calibrated external behavioral evidence. Candidates from three model families, prompting strategies, and solver stacks are run on instances resampled from an extracted parameter domain; agreement across the resulting value-function traces is summarized by a cross-family clique, and a calibrated threshold returns accept, abstain, or escalate. The preregistered false-discovery criterion holds on calibration data but not on the wild stream. We report this negative result in full and trace most failures to benchmark texts that do not faithfully encode their labeled instances. Comparing four admission judges on one collection of logs inside a state-of-the-art skill learner, AdmitOR raises admission precision to 0.927, against 0.871 for majority vote and 0.726 for execution success, yielding 3.1x and 8.0x fewer poisoned admissions. Its library is the smallest and attains the highest macro accuracy across five public benchmarks, 58.4 against 54.8 for majority vote and 53.9 for the ground-truth-labeled library. The 3.5-point gain over majority vote is supported by a paired bootstrap and survives correction for a host-side anomaly. To our knowledge, AdmitOR is the first label-free admission mechanism designed around an explicitly calibrated false-discovery target. The transfer failure identifies a necessary condition for extending it to wild streams.

1 INTRODUCTION

Label-free admission is the central barrier to experience-learning agents because execution-only and single-instance agreement can admit poisoned or inequivalent models. AdmitOR addresses this problem with statistically calibrated external behavioral evidence from cross-family, multi-instance value-function traces, targeting false-discovery control rather than per-instance certainty.

The admission problem: 878 candidate models were admitted by execution-only judging on a 300-problem label-blind stream, including 241 that disagreed with withheld answers.The result is presented as evidence of poisoned memory from self-assessment and execution-based admission.
The admission problem: 960 is returned by both the correct model and a model omitting a slack floor constraint, so single-point agreement accepts both inequivalent models.The models separate on resampled instances even though they agree at the original instance.
AdmitOR: ADMITOR evaluates candidates from three model families, prompting strategies, and solver stacks on resampled instances, comparing multi-instance value-function traces with a statistically calibrated cross-family clique.Under Proposition 1, inequivalent models disagree on a positive-measure set, while the probability of missing that disagreement decays geometrically in the number of resampled instances.
Contributions: The paper defines label-free admission as controlling the false-discovery rate on the admitted stream rather than achieving unattainable per-instance certainty.Its mechanism combines a cross-family panel, resampled value-function traces, an identifiability proposition, and finite-sample calibrated admission under a stated transfer assumption.
Contributions: Text-faithful systems have a measurable false-discovery-rate lower bound on the evaluated stream, and the authors release the code, verdicts, run ledger, and complete case packets behind it.The contribution frames this lower bound as a limitation on label-free certification when benchmark texts do not faithfully encode labeled instances.

2 RELATED WORK

Prior experience-learning, self-verification, ensemble, and optimization-verification methods rely on answers, single-instance comparisons, or individual-generation checks without calibrated admission for persistent libraries. ADMITOR instead combines label-free cross-family behavioral evidence with finite-sample calibration and explicitly identifies the assumptions limiting transfer.

Experience learning with labeled admission: Existing experience-learning systems admit reusable knowledge using known optima, ground-truth-labeled trajectories, or expert exemplars.ADMITOR replaces this supervision signal and is evaluated inside one such system using its native ground-truth oracle as reference.
Self-verification and its limits: Self-verification can degrade reasoning, miss errors, favor self-generated outputs, and requires external tools for effective critique.Prior work also establishes system-level failure and limits on sound fixed-threshold perturbation testing.
Ensembles, voting, and juries: Vote-, jury-, routing-, and debate-based methods compare answers at single instances and provide no error budget for accepted outputs.The shared-error floor motivates ADMITOR’s cross-family panel construction.
Verifiers for optimization modeling: Recent verifiers filter individual generations but do not define persistent-library admission semantics, target calibrated FDR, or close the experience-learning loop.ADMITOR adopts Lian et al.’s instrument as a diagnostic layer.
Conformal selection: ADMITOR is presented as the first label-free admission mechanism addressing label-free, behavioral, beyond-one-point, and budgeted evaluation properties.Its finite-sample calibrated selection uses split-conformal Benjamini–Hochberg, cross-family agreement over resampled value functions, and deployment-matched escalation; the calibrated budget does not transfer unchanged when calibration assumptions fail.

3 THE ADMITOR GATE

AdmitOR certifies optimization-model candidates by resampling a base-anchored parameter domain and requiring cross-family value-function agreement, then calibrates an admission threshold for false-discovery control. Accepted trajectories enter distillation, while abstention and escalation manage insufficient evidence; the gate detects behavioral agreement rather than shared misinterpretation.

Gate objective: AdmitOR returns ACCEPT with a certified value, ABSTAIN, or ESCALATE after evaluating executable candidates over a parameter domain.The gate’s objective is to control incorrect certifications among accepted admissions at nominal level α.
Panel construction: Three model families, prompting strategies, and solver stacks form a decorrelated panel intended to reduce shared errors.Separate backends lower correlated failure sources, while within-family agreement has a higher shared-error floor.
Evidence generation: A maximum clique spanning at least two families certifies the common value at the stated instance when candidates agree across all informative resampled instances.The parameter domain is anchored so the base instance exactly matches the ticket, while guardrails preserve structural dimensions and handle conflicting or degenerate ranges.
Calibrated admission: 10% is the pilot’s finite-sample certified false-discovery budget, distinct from the 5% nominal selection target.With four attainable score values and δ = 0.05 per threshold, the selected-threshold certificate is read at 95% per threshold and 80% simultaneously over the grid.
Distillation and limitations: Only ACCEPT verdicts enter distillation, whereas ABSTAIN and ESCALATE tickets are excluded; shared misinterpretations remain invisible to resampling.The gate targets the false-discovery rate of the admitted stream rather than correctness of every admission, and higher precision costs abstention.

4 EXPERIMENTS

Across five public benchmarks and a 300-problem label-blind stream, AdmitOR improved downstream accuracy over majority vote, but its preregistered false-discovery criterion failed because benchmark texts sometimes misencoded their labeled instances. The audit shows that consensus can certify a shared extracted specification rather than the intended problem.

Downstream performance: AdmitOR matched or exceeded majority vote on all five benchmarks, adding 3.5 macro points while using the smallest library.Every label-free arm exceeded 70% of ground-truth macro accuracy, and all three exceeded the observed ground-truth result.
Downstream performance: +3.53 points was AdmitOR’s bootstrap macro gain against majority vote, with a 95% interval of [+0.87, +6.75].Against the ground-truth arm, the gain was +4.46 points with a 95% interval of [+2.37, +6.64].
False-discovery criterion: 15.9% was the realized false-discovery rate among 138 admitted certificates, with a 95% upper bound of 22.0%, so the preregistered criterion failed.The stricter admission-time rule found 26 disagreements, or 18.8%.
Failure analysis: Cross-family consensus certified the extracted instance when the benchmark text omitted information, because all families were conditioned on the same faulty base specification.Resampling from that specification cannot validate the input it is conditioned on.

5 LIMITATIONS

AdmitOR certifies agreement among independently derived behaviors rather than intended problem meaning, so shared interpretation and extraction errors can evade detection. Higher precision also reduces coverage, while the guarantee depends on its assumption and calibrated escalation, and the host evaluation does not test backbone diversity.

Observation channel: Shared incorrect interpretations can agree across all sampled instances, because the gate observes behavior rather than the problem’s intended meaning.This limitation concerns the observation channel, motivating an admitted-stream FDR objective rather than a per-admission guarantee.
Coverage trade-off: Higher admission precision reduces coverage, with the gate emitting nothing on a substantial fraction of the stream, mostly for lack of informative instances.Escalation reserves human attention when every ticket requires an answer, while false admissions can poison a persistent library.
Conditional guarantee: False certifications with complete clique evidence survive thresholds based only on clique geometry, and shared extracted specifications make extraction errors undetectable by consensus.The guarantee is conditional on Assumption 1 and weakens under escalation policies other than the calibrated one.
Host evaluation: One generation backbone in the host isolates admission-judge effects without testing backbone diversity, although the cross-family panel extends to multi-backbone hosts.The limitation applies to the host evaluation rather than to the panel’s stated extension.

6 CONCLUSION

ADMITOR replaces answer-based admission with calibrated external behavioral evidence and explicit abstention, achieving the highest precision and downstream accuracy among the compared libraries. Its calibrated target does not transfer to wild streams because benchmark texts may fail to encode their labeled instances faithfully.

Contribution: ADMITOR replaces answer-based admission with externally generated behavioral evidence, using resampled value-function traces and cross-family agreement mapped to a false-discovery target.The protocol includes explicit abstention and evaluates three model families.
Contribution: The certified library achieves the highest precision and downstream accuracy because the collect-once, replay-many protocol changes only the judge.This design attributes library differences to the admission judge.
Limitation: The calibrated target fails to transfer to the wild stream, where benchmark texts often do not faithfully encode their labeled instances.This violates the stated transfer assumption and identifies a data-layer requirement for wild-stream certification.

A PROOFS · A.1 IDENTIFIABILITY BY RESAMPLING

Under compact, full-dimensional resampling and solver-exact parametric-model assumptions, candidate value functions have finite piecewise-polynomial structure. This yields an almost-everywhere identifiability dichotomy: resampling detects any positive-measure disagreement with increasing power, while null-set differences are certified as equivalent.

A.1 IDENTIFIABILITY BY RESAMPLING: Under (A1)–(A3), each candidate’s value function is piecewise polynomial over finitely many full-dimensional cells, with degree at most two.The function is affine when perturbations affect only the right-hand side or only the objective coefficients.
A.1 IDENTIFIABILITY BY RESAMPLING: Parametric linear programming proves this structure because fixed integer patterns yield finitely many critical regions with stable optimal bases.Within each region, the basis solution is affine in parameters, and multiplying it by affine objective coefficients produces a degree-at-most-two polynomial.
A.1 IDENTIFIABILITY BY RESAMPLING: If two candidates disagree on a positive-measure set D, the resampling probability satisfies p := P(D) ≥ ρ λ(D) > 0.If λ(D) = 0 instead, they agree P-almost surely and certify the same value on almost every instance.
A.1 IDENTIFIABILITY BY RESAMPLING: The same identifiability bound applies to tolerance-based disagreement Dδ = {θ : |∆(θ)| > δ} whenever λ(Dδ) > 0.This extends certification beyond exact equality to disagreements exceeding a specified tolerance δ > 0.
A.1 IDENTIFIABILITY BY RESAMPLING: The proof establishes the dichotomy by refining both candidates’ partitions and using that a nonzero polynomial has a Lebesgue-null zero set within a full-dimensional cell.Thus, any non-identical polynomial piece creates a positive-measure disagreement region.
A.1 IDENTIFIABILITY BY RESAMPLING: Detection power after m independent resamples is 1 − (1 − p)^m ≥ 1 − e^−pm, while the mandatory base instance is evaluated deterministically.A disagreement at the stated base instance is therefore detected with probability one, independently of the resampling bound.
A.1 IDENTIFIABILITY BY RESAMPLING: The certification semantics treats candidates differing only on a null set as behaviorally indistinguishable and therefore equivalent under the gate.Equivalence is explicitly defined as almost-everywhere equivalence.

A.2 POLICY-MATCHED CALIBRATION

Policy-matched calibration guarantees valid conformal testing and FDR control only when calibration and deployment use the same record-based policy. Replacing a single-pass calibration policy with an escalation ladder can invalidate the guarantee by an arbitrarily large factor, so deployment trajectories must also calibrate the ladder.

Policy definition: A policy maps each problem’s complete record—including text, panel randomness, and instance randomness—to a final admission score, with escalation ladders capped at K rounds.Borderline scores can trigger additional instances or candidates before the final score is emitted.
Guarantee: Same-policy calibration makes calibration and null-test scores i.i.d., yielding valid conformal p-values and FDR control at level α.The proof relies on a fixed measurable policy applied to independent records.
Failure under mismatch: Different policies can break the guarantee by an arbitrarily large factor when single-pass calibration is replaced by ladder-based deployment.For every K, score laws and ladders exist for which the realized null false rate exceeds the calibrated guarantee by an arbitrarily large factor.
Operational implication: Deployment ladders must also generate calibration trajectories, because retry-until-agreement escalation takes a maximum over dependent scores near the threshold.The same ladder used in deployment must produce the calibration data.

B PROTOCOL REGISTRATION AND SCORING

The study preregistered withdrawal, calibration, rerun, and decorrelation criteria before collecting data, and used a single released scorer with explicit label-matching tolerances. Its certification protocol extracts and freezes structure, perturbs parameters, evaluates diverse candidates, and retains informative instances for agreement analysis.

Preregistered decision criteria: K1 withdraws the mechanism if the best label-free arm achieves less than 70% of the ground-truth arm’s macro accuracy.
Preregistered decision criteria: K2 withdraws the consensus-mechanism claim if the gate fails to outperform majority vote downstream, while K3 flags calibration above 5% point-estimate or 10% 95% upper-bound false-discovery rate.One rerun with a stricter threshold is allowed and must be reported.
Certification protocol: The certification procedure freezes structural sizes, draws seeded perturbations around extracted base parameters, probes candidates from distinct families and strategies, records objective values and solver status, and keeps informative instances for at least three candidates.
Scoring rule: The released scorer counts predictions correct when they match labels within relative tolerance 10^-4 or reproduce their printed decimal rounding exactly.Unparseable predictions count as errors; labels are frequently rounded.

B.1 THE DATA-LAYER STAGE

The data-layer stage runs before Step 1 when at least two extractor families are configured, admitting panel entries only when their extracted specifications agree on keys, shapes, stated values, and compatible perturbation domains.

Data-layer stage: Panel entry requires identical key sets and shapes, elementwise agreement of stated values within Appendix B’s scoring tolerance, and domain compatibility after Step 2 guardrails.Each extractor family independently maps the text to named parameters, their shapes, stated values, and a perturbation domain.

C HOST REPRODUCTION DETAILS AND SUPPLEMENTARY TABLES

Host reproduction details show that published benchmark labels can reject correct models and materially affect evaluation, while behavioral certification can recover them. Audits also show that near-equal objective values do not establish model equivalence, and correcting one label changes ComplexOR accuracy to 72.22%.

Label-error case: 250 is the verified optimum for an instance labeled 200, causing the ground-truth judge to reject correct trajectories while AdmitOR certifies them from behavioral agreement.All four evaluation arms still return 250 and are penalized because the preregistered protocol retains published labels; AdmitOR never reads the label.
Disagreement attribution: 22 disagreements were human-audited, with no case attributed to a gate defect or legitimate alternative interpretation and the dominant category identified as a benchmark property.Table 4 attributes the disagreements to the benchmark rather than the solver.
Calibration audit: 0.023% separates certified and reference objective values even though the models differ by an entire constraint family, so similar values do not establish model equivalence.Exact solution of the fully printed instance reproduces the certified value, showing that the reference model is erroneous in this audited case.
Host reproduction: 72.22% is the corrected ComplexOR accuracy after replacing the published label 200 with the manually verified optimum 250 for the discussed instance.With n = 18, each item changes accuracy by 5.56 points, and the corrected score is 13/18 = 72.22%.