Tail-Influence Sampling for CVaR Policy Evaluation

• Anchored TIS combines influence-based allocation with visitation shares and has asymptotic MSE within a factor two of the oracle under the same conditions.Source: §1 · p. 2
• On exact grids where submaximal returns are rarer than the tail level, tail- and mean-optimal allocations coincide, and their divergence serves as a diagnostic.Source: §1 · p. 2
• TIS derives a CVaR-specific influence for shared kernels, whose variance supplies a fixed-design efficiency bound and oracle Neyman allocation.Source: §1 · p. 2
• Under fixed dimension, a positive quantile margin, and suitable pilot and exploration schedules, TIS attains oracle asymptotic variance and first-order MSE including pilot cost.Source: §1 · p. 2
• Experiments report matched-budget gains over learned occupancy, mean influence, and rollouts across CliffWalking, inventory, and frozen language-model workflows.Source: §1 · p. 2

• The setup is a finite-horizon MDP with finite state and action spaces, stationary dynamics, and a fixed policy.Source: §2 · pp. 2–3
• Figure 1 illustrates that equal visitation and reward moments can still yield different TIS query shares when only one kernel can fall below the CVaR cutoff.Source: Figure 1 · p. 3
• The target is CVaR at level α, the average return in the worst α-fraction of runs, with smaller α emphasizing rarer outcomes.Source: §2 · p. 3
• A query samples a conditional kernel for one state–action pair, and its fitted law is reused at every remaining horizon where that pair occurs.Source: §2 · p. 3
• Return probabilities are estimated on a fixed grid through successive Bellman updates, with grid approximation error bounded by H∆ relative to true-return CVaR.Source: §2 · p. 3

• Adjoint weights translate shortfall-coordinate errors to the root shortfall, and scaling by −1/α converts the effect to CVaR error.Source: §3 · p. 4
• A shared kernel’s one-draw effects are summed across all uses before their variance is measured, retaining cross-stage covariance.Source: §3 · p. 4
• For a fixed allocation, the asymptotic variance is an efficiency bound attained by the empirical categorical Bellman estimator under the stated positive-margin conditions.Source: §3 · pp. 4–5
• The oracle allocation follows Neyman proportions: a group with twice the influence scale receives twice the oracle share.Source: §3 · p. 5
• Equal visitation and conditional reward moments can conceal different tail influence when only one kernel has random shortfall below the CVaR threshold.Source: §3 · p. 5
• Even with identical visitation, conditional moments, and root return law, the occupancy-to-oracle ratio can range from 1 to G.Source: §3 · p. 5

4 TAIL-INFLUENCE SAMPLING

• Pilot outcomes are scored across stage effects, their variability sets query shares, and a floor reserves samples for groups the pilot may underestimate.The pilot fits a provisional model and estimates the root quantile and sensitivity weights before fresh main samples are drawn.Source: §4 · p. 5
• With a total pilot of order N^2/3 and floor N^-1/4, TIS attains oracle leading MSE including pilot cost for fixed G under the theorem’s conditions.The theorem permits individual zero-influence groups and controls learning the continuation model, quantile, influence scales, and allocation.Source: §4 · p. 6
• Anchored TIS averages influence-based and visitation-based shares, giving an asymptotic MSE constant between the oracle and twice the oracle.The guarantee does not ensure lower finite-budget MSE; the mixture protects against pilots that miss rare outcomes, though both components can underallocate the same group.Source: §4 · p. 6
• Learning an allocation repays its pilot when allocation advantages are large, the pilot is small, and the realized shares are accurate.A starved group is especially costly in the leading error term, which motivates the anchor.Source: §4 · p. 6
• On an exact categorical grid with submaximal-return probability below α, CVaR moves exactly with the mean and tail- and mean-optimal Neyman allocations coincide.Locally, each kernel’s categorical-CVaR influence is 1/α times its ordinary mean-return influence; the influence-share distance is zero in this regime.Source: §4 · p. 6

Q1. Controlled separation

• When tail influence is concentrated, plain TIS is .16–.22 of uniform MSE at t = 1; when uniform allocation is optimal at t = 0, TIS is 4.3–5.4× uniform MSE.The intermediate case also does not repay the pilot.Source: §5 · p. 7

Q2. Tabular benchmarks

• At 1,200 queries/block, inventory MSE/uniform is .657 for TIS, .918 for learned occupancy, .866 for learned mean influence, and 2.22 for complete rollouts at matched transition cost.Across 18 prespecified inventory cases, TIS and anchored TIS have resolved lower MSE than occupancy and rollouts in all 18, and than learned mean in 17.Source: §5 · p. 7
• At 400 queries/kernel, TIS lowers CliffWalking MSE by 40.9% vs. learned occupancy and 76.3% vs. complete rollouts at matched transition cost.At the largest budgets, Figure 2 also reports lower TIS MSE than learned occupancy, learned mean, and rollouts.Source: §5 · p. 7, Figure 2 · p. 8

Q3. Pilot reliability in language-model workflows

• At H = 6 and 400 queries/kernel, plain TIS exceeds uniform MSE for Qwen3-4B (1.38) and GLM-4-32B (1.77), while anchored TIS reaches .056–.093 of uniform MSE and the lowest MSE for 5/6 generators.Occupancy has lower observed MSE than plain TIS for all six generators; the 90th-percentile realized/oracle variance ratio falls from 136 to 5.8 with anchoring.Source: §5 · p. 8

Q4. Held-out financial numerical review

• At 100–200 queries/kernel, anchored TIS is .089–.100 of uniform MSE on Qwen3-4B and .216–.327 on Phi-4-mini, while at 400/800 queries it beats occupancy and rollouts in both Phi workflows.With six versus three calls, the Phi anchor’s MSE is .29–.64 of rollouts’ at every budget in both workflows.Source: §5 · p. 8

Q5. Is the tail score itself what helps?

• Against an equally regularized mean-influence blend, anchored TIS has resolved lower MSE in 23/24 confident-error MMLU-Pro cells, while mean blending is as good or better on FinQA.The pilot-estimated tail–mean distance is .000 on FinQA, while MMLU-Pro shows more frequent failures and distances of .08–.33.Source: §5 · p. 8, Table 2 · p. 9

From distributional inference to query design

• Distributional-RL work provides categorical representations and studies return-law and functional limits under specified, possibly nonuniform sampling laws (Bellemare et al., 2017; Rowland et al., 2018; Zhang et al., 2025).Quantile-based evaluation also has semiparametric efficiency guarantees (Cheng et al., 2026).Source: §6 · p. 9

Adaptive allocation with a learned Bellman model

• Adaptive stratified sampling learns variances for Neyman allocation, whereas this paper controls learning a Bellman-dependent score and allocation, including pilot cost (Neyman, 1934; ´Etor´e & Jourdain, 2010; Carpentier et al., 2015).Small-pilot failures and defensive mixtures motivate the anchor (Cai & Rafi, 2022; Hesterberg, 1995; Owen & Zhou, 2000).Source: §6 · p. 9
• Unlike risk-sensitive control that changes the policy or generative-access work on CVaR policy optimization, this paper fixes the policy and optimizes conditional-query allocation to estimate its tail functional (Tamar et al., 2015; B¨auerle & Ja´skiewicz, 2024; Deng et al., 2025).Trajectory designs such as SaVeR target mean evaluation.Source: §6 · p. 9

7 DISCUSSION

• The contribution is a learnable CVaR allocation signal for reused conditional laws, with oracle first-order MSE under stated assumptions and an anchored variant within a factor two of the oracle.On exact grids, the paper also gives a condition under which mean influence suffices.Source: §7 · p. 9
• TIS is most promising when conditional access is feasible, tail influence differs from visitation or mean influence, and pilots can learn that difference at a cost worth paying.The guarantees require fixed-dimensional categorical models, a positive quantile margin, independent queries, and the stated pilot and floor schedules; finite-budget MSE guarantees remain open.Source: §7 · p. 9

SUPPLEMENT: TIS FOR CVAR POLICY EVALUATION

• Appendices A–D develop the allocation theory from the stop-loss Bellman representation through fixed-design efficiency, learned oracle adaptation, and the anchored safeguard.Appendix E covers categorical representation error and experimental evidence; Appendix F discusses related foundations.Source: SUPPLEMENT: TIS FOR CVAR POLICY… · p. 13

A NOTATION AND ASSUMPTIONS

• The setup distinguishes a query group, which identifies a sampled law, from a Bellman row, which identifies one use of that law.The model assumes finitely many independently queryable laws, fixed bounded outcome spaces, and categorical Bellman conditions including a positive quantile margin.Source: A NOTATION AND ASSUMPTIONS · p. 14
• Shared-kernel observations contribute jointly across the required layer rows, so their layer effects must be summed before computing variance.The full-vector influence formula is given for shared state–action kernels.Source: A NOTATION AND ASSUMPTIONS · pp. 14–15

B.1 MINIMIZING THE ALLOCATION VARIANCE

• Once group influence scales are known, the variance-minimizing allocation assigns each group a share proportional to its scale.When some scales are zero, the infimum is approached by positive designs whose shares for zero-scale groups tend to zero.Source: B ALLOCATION AND BELLMAN IDENTITIES · p. 15, §B.1 · p. 15

B.2 PROJECTION AND CATEGORICAL CVAR IDENTITIES

• Categorical projection preserves shortfalls at every grid threshold, allowing the projected Bellman representation to retain the CVaR shortfall calculation.A positive quantile margin keeps the VaR index fixed locally, making the CVaR readout locally affine.Source: §B.2 · p. 16

B.3 EXACT EMPIRICAL FIXED-POINT EXPANSION

• The empirical fixed-point error separates into a leading sampling contribution and a smaller remainder controlled by bounded observations and the finite-horizon resolvent.When the minimum group sample size is of order N, the normalized squared remainder is O(N^-1).Source: §B.3 · p. 17

C FIXED-DESIGN LIMITS AND STRUCTURAL INTERPRETATION

• The fixed-design results turn the group influence calculation into an accuracy benchmark and show why visitation and reward moments alone cannot determine the best tail allocation.The untied case factors group influence into propagated adjoint mass and local variability.Source: C FIXED-DESIGN LIMITS AND STRUCTURAL… · p. 18, §C.3 · p. 20

C.1 FIXED-DESIGN CLT AND NORMALIZED MSE

• Under a stable allocation and positive quantile margin, the empirical CVaR estimator has a Gaussian limit and its normalized MSE converges to the design variance.The empirical and population VaR indices agree with exponentially high probability under the stated conditions.Source: §C.1 · pp. 18–19

C.2 SEMIPARAMETRIC EFFICIENCY

• For each fixed design, the empirical categorical Bellman estimator is regular and attains the semiparametric efficiency bound for CVaR.The bound is the canonical-gradient variance in the product model of unrestricted group laws on their declared outcome spaces.Source: §C.2 · p. 20

C.3 UNTIED FACTORIZATION FOR THE ALLOCATION ABLATIONS

• In the untied allocation ablations, oracle group scale is the product of adjoint mass and threshold-averaged local variability, not ordinary occupancy alone.Adjoint mass propagates visitation together with the remaining shortfall threshold.Source: §C.3 · p. 20

C.4 CONTROLLED SEPARATION WITH FIXED VISITATION, MOMENTS, AND ROOT LAW

• With equal visitation, reward moments, and root return law, tail influence can still concentrate on one group, making uniform mean-optimal allocation different from the tail oracle.In the constructed example, the tail oracle is a boundary design with variance 1/G^2, while the mean-optimal design is uniform.Source: §C.4 · p. 21

D LEARNED ALLOCATION

• TIS uses pilot-fitted conditional laws to estimate the root quantile and influence scales, then allocates fresh main samples using the learned shares.The pilot is charged to the total budget, and only fresh main samples form the final estimate.Source: D LEARNED ALLOCATION · p. 22

D.1 PILOT-SCALE CONSISTENCY AND LOWER-TAIL CONTROL

• Under fixed dimension and a positive quantile margin, pilot influence estimates are bounded, consistent, and protected against severe underestimation for positive-scale groups.For active groups, the probability of severe scale underestimation decays exponentially with pilot size under the stated conditions.Source: §D.1 · pp. 22–23

D.2 FINITE-PILOT DESIGN STABILITY

• Near an interior oracle, small influence-scale errors incur a quadratic variance cost, but this local stability result does not protect against severe underestimation.The bound uses the maximum pilot scale error and the allocation floor.Source: §D.2 · pp. 23–24

D.3 ORACLE ADAPTATION

• With the stated schedules and assumptions, learned TIS attains the oracle asymptotic variance and normalized MSE despite pilot estimation, rounding, and nonlinear fixed-point error.The proof controls both the leading influence term and the remaining fixed-point and wrong-quantile-cell contributions.Source: §D.3 · p. 25

D.4 ANCHORED DESIGN: VARIANCE SAFEGUARD AND ASYMPTOTIC MSE

• The anchor averages influence and occupancy shares, retaining at least half of either component; its pre-rounding worst-case factor is minimized at β=1/2.The safeguard compares component designs before rounding and does not bound finite-budget MSE; the full estimator’s limiting MSE includes pilot cost and rounding.Source: §D.4 · p. 26

D.5 WHEN LEARNING AN ALLOCATION REPAYS ITS PILOT

• Learning improves on a no-pilot allocation at the influence-term level only when allocation advantage exceeds share-estimation error and discarded pilot cost.The comparison is a diagnostic identity using population influences, not a finite-budget guarantee for the nonlinear CVaR estimator.Source: §D.5 · p. 27

E APPROXIMATION AND EXPERIMENTAL PROTOCOLS

• The experimental protocols separate categorical-grid approximation error from sampling error and specify the query and charged-budget rules.Rollouts use ⌊N/H⌋ full trajectories, while conditional allocations exhaust N; learned designs charge and discard their pilot.Source: E APPROXIMATION AND EXPERIMENTAL… · p. 28, §E.2 · p. 28

E.1 REPRESENTATION ERROR

• Randomized rounding to adjacent grid atoms couples projected and true returns, and a bounded return discrepancy transfers to a bounded CVaR discrepancy.The proof constructs the coupling recursively, adding at most one grid spacing per step.Source: §E.1 · p. 28

E.2 EXPERIMENTAL EVIDENCE AND PROTOCOLS

• MSE is averaged squared error across replications; its reported uncertainty bars are 1.96 Monte Carlo standard errors.Population references use exact laws, whereas learned-score designs pay for and discard a pilot.Source: §E.2 · p. 28

E.2.1 CONTROLLED STOCHASTIC MARKOV REWARD PROCESS

• At N=100,000, TIS reaches .615 of uniform MSE; across 50 instances, median Vunif/V* is 1.95344, 1.92311, and 1.82727 at α=.05, .1, and .2.The controlled experiment also reports that anchored TIS, learned mean, and learned occupancy are close, while rollouts are worse than uniform.Source: Figure 3 · p. 29, §E.2.1 · p. 29

E.2.2 LEARNED-BUDGET RUNS FOR THE CONTROLLED SEPARATION

• At t=1, the median TIS share of the informative kernel rises from .62 to .88 across budgets, while its 10th percentile is .10–.75 at 25–50 queries.The source identifies this lower-tail allocation behavior as underallocation that anchoring mitigates.Source: §E.2.2 · p. 30

E.2.3 SEASONAL BASE-STOCK INVENTORY EVALUATION

• Across every tested query budget, TIS has the lowest observed learned-method MSE; at 1,200 queries per block, its MSE is 15.4% above oracle+floor.Source: §E.2.3 · p. 30

E.2.4 SLIPPERY CLIFFWALKING WITH STATIONARY STATE–ACTION KERNELS

• At 400 charged queries per reachable kernel, TIS reduces CliffWalking MSE by 31.3% versus population occupancy, 12.9% versus population mean, and 18.3% versus learned mean.Source: Table 6 · p. 31, §E.2.4 · p. 32

E.2.5 ADDITIONAL PUBLIC STATIONARY GYMNASIUM ENVIRONMENTS

• At 400 queries, TIS MSE is .00791 (.00079) on FrozenLake and .00020 (.00002) on Taxi, while learned mean is better on FrozenLake and tied on Taxi.The checks support a structural limit: when tail and mean signals align, the smoother mean score can be as good as or better than tail targeting.Source: §E.2.5 · p. 32

E.2.6 STRUCTURED LANGUAGE-MODEL WORKFLOW EVALUATION

• Each workflow question is a finite-horizon Markov reward process with 91 query groups, and fixed state-action prompts can be queried directly rather than reached by rollout.Prompts omit history and stage index, so a fixed state-action pair has the same next-response law at each stage.Source: §E.2.6 · p. 33
• Closed grids make categorical recursion and stop-loss interpolation exact; full-horizon-only grids have median/90th-percentile absolute CVaR gaps of .02/.35, versus below 10^-14 on closed grids.The grids contain 121, 617, and 983 atoms at H = 2, 4, and 6.Source: §E.2.6 · p. 33
• At α = .1, occupancy is below plain TIS in all 18 settings; rollouts have the lowest sample-only point MSE for every generator at H = 2, while the anchor does so for five at H = 6.At H = 6, plain TIS exceeds uniform MSE for Qwen3-4B (1.38) and GLM-4-32B (1.77).Source: §E.2.6 · p. 34
• At H = 6, α = .1, 400 queries, raising the floor from .072 to .40 lowers Qwen and GLM TIS/uniform MSE from 1.38/1.77 to .47/.55, diagnosing pilot underallocation rather than establishing a tuning result.The sweep was retrospective; held-out FinQA tested the subsequently frozen anchor.Source: §E.2.6 · p. 34
• The primary outcome is estimation error rather than workflow choice because selection on the MMLU panel is nearly saturated.The study also reports a cautious policy and a disjoint high-stakes panel as post-hoc checks of the pilot-reliability diagnosis.Source: §E.2.6 · p. 35

E.2.7 COST-TO-ACCURACY ANALYSIS AND COMPUTATION ACCOUNTING

• On CliffWalking, TIS requires .32–.45 of rollout queries and .65–.84 of learned-occupancy queries across the studied accuracy targets; inventory ratios are .26–.30 versus rollouts and .55–.72 versus learned occupancy.FinQA results differ by generator: on Qwen both strong baselines require fewer queries at every target, while Phi comparisons are mixed or unresolved.Source: §E.2.7 · p. 35
• For FinQA Phi ordinary review, empirical envelopes favor the anchor at 14 targets versus occupancy and the 10 strictest targets versus rollouts, while baselines win at 4 and 8 targets, respectively.For unit-check review, the anchor is favored at 12 and 6 targets, while baselines win at 0 and 12.Source: §E.2.7 · p. 35
• A single target can mislead because uniform’s largest-budget RMSE cannot rank sample-only CliffWalking designs when all reach it at the smallest tested budget.The crossings use log–log first-crossing interpolation without extrapolation, and the reported envelopes are pointwise rather than simultaneous.Source: §E.2.7 · p. 35
• Full sampling-and-estimation pipeline timings are frozen-law replay times, not live-call costs or time-to-equal-accuracy.The tail-specific influence score adds no computation over the strongest mean-based baseline in the reported comparison.Source: §E.2.7 · p. 36, Table 13 · p. 36

E.3 FINQA TERMINAL-RISK PROTOCOL AND FULL RESULTS

• At 400/800 queries, anchor MSE is .81–.86 of occupancy and .68–.88 of rollouts across Phi workflows, while occupancy and rollouts remain better on Qwen.At 100–200 queries, all eight anchor/uniform contrasts are resolved, while plain TIS is unresolved in seven; Phi workflow-selection comparisons with occupancy or rollouts are unresolved.Source: §E.3 · p. 37, Table 14 · p. 38

E.4 INVENTORY DISRUPTION FAMILY

• Across 18 inventory cases, TIS median MSE/uniform is .74/.62/.61 at 150/600/1,200 queries per block, and it reaches more declared RMSE targets than uniform, occupancy, or rollouts.Plain TIS leads the anchor throughout, consistent with anchoring acting as insurance rather than a gain when pilots are reliable.Source: §E.4 · pp. 38–39, Table 16 · p. 38

E.5 BLENDING CONTROLS

• On held-out FinQA the anchor beats the equally regularized uniform blend in all 16 cells, but the mean blend has lower point MSE in all 16; on MMLU-Pro the anchor beats the mean blend in 23 of 24 settings.MMLU-Pro includes six generators at H = 6; the unresolved comparison is Phi-4-mini at 100 queries.Source: §E.5 · p. 39

E.6 RARE FAILURES AND THE TAIL–MEAN COINCIDENCE

• When the exact-grid condition places submaximal-return probability below α, tail and mean influences yield identical Neyman shares; held-out FinQA median tail–mean distance is .000.The paper also tests a prospective divergence rule in five new settings, with its predictions holding in all three settings where it makes predictions.Source: §E.6 · pp. 39–40

E.7 FINQA WITH CALCULATOR FAULTS

• With calculator faults, tail and mean influence remained close despite different occupancy shares; rollouts were usually strongest, and the mean blend beat the anchor throughout Qwen.The results support the diagnostic's negative prediction for settings where normalized tail and mean allocations nearly coincide.Source: §E.7 · pp. 40–41

E.8 LONGER REVIEW LOOPS

• At H = 6 and 400 queries, Phi workflows show anchored-TIS MSE ratios of .29/.41 versus rollouts, equivalent to 3.4×/2.4× lower MSE; longer loops favor Phi’s anchor, but not near-deterministic FinQA Qwen.Source: Table 17 · p. 41, §E.8 · p. 41

E.9 SELECTING THE ALLOCATION FROM THE PILOT; E.10 MECHANISM: WHAT THE PILOT SEES AND WHAT ANCHORING CHANGES

• Across 46 blending settings and 148 setting-budget cells, the pilot-distance rule is wrong in 7, with selected-design MSE relative to the better design having median 1.00; rare deviations missed by pilots remain a failure mode.In the examined nondegenerate cells, pilot-averaged leading-variance predictions closely track observed MSE, but near-deterministic questions can hide rare deviations.Source: §E.9 · p. 42, §E.10 · p. 42, Figure 5 · p. 42

F ADDITIONAL RELATED WORK

• Related work studies inference under specified data laws, whereas this work learns allocation scores tied to an unknown Bellman continuation model and CVaR cutoff (Zhang et al., 2025; Cheng et al., 2026).The allocation rule is Neyman allocation; the contribution is identifying and learning the Bellman influence scales it requires.Source: F ADDITIONAL RELATED WORK · p. 43, Table 18 · p. 43