PaperlayerView exact revision on arXiv

Source-linked AI summary

Pairwise AUC Optimization Needs Corrective Power: A Unified View

Jia Chen, Zhiyong Yang, Shilong Bao, Qianqian Xu, Qingming Huang

arXiv:2610.21743v1cs.LG

TL;DR

Direct pairwise AUC training aligns its objective with the ranking metric, yet may not reliably correct severe ranking errors. The paper analyzes this gap through Corrective Power and the U-statistic structure of pairwise gradients. It identifies conditions for weakened corrective motion and relates them to surrogate choice and initialization.

  • Problem

    Direct pairwise AUC-surrogate training aligns with the ranking metric, but the stochastic updates may still fail to focus corrective effort on severely misranked pairs.

    Source: §1 · p. 1

  • Method

    The paper analyzes projected drift and variance along a severe-error correction direction, using the U-statistic structure of pairwise mini-batch gradients.

    Source: §1 · p. 2, §2 · p. 4, §6 · p. 10

  • Results

    When the leading projected-variance term vanishes, useful noise drops from order B^-1 to order B^-2 and the projected escape-time scale changes from order B to order B^2.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The analysis connects corrective strength to surrogate weighting and starting geometry, with squared-hinge, CE warm-up, and pretrained parameter-efficient fine-tuning addressing these factors.

    Source: §1 · pp. 2–3, §6 · p. 10

    The warm-up argument applies when the endpoint has low severe-misranking mass and the early pairwise trajectory remains near that endpoint.

    Source: §4 · p. 7

Abstract

from arXiv · show

Direct pairwise AUC-surrogate training is attractive because it trains toward the ranking metric used at test time, especially under class imbalance. This creates a direct alignment between the training objective and the evaluation metric, but such metric alignment alone does not guarantee that stochastic training remains corrective. We study this gap through Corrective Power: whether pairwise mini-batch updates produce stronger and more reliable improvements for more severe positive-negative ranking errors. Our main observation is that direct pairwise training can enter regions where severe errors remain common but no longer produce useful corrective motion. In such regions, even the mini-batch noise that would normally help SGD move along the steepest severe-error correction direction can become much weaker. In this paper, we seek a theoretical explanation for this by exploring the degenerate property of the U-statistics formed from the pairwise gradients therein. Our results provide a mechanistic explanation for several empirical patterns: (a) squared-hinge loss is often an effective AUC surrogate; (b) cross-entropy warm-up can help models that enter the AUC phase from a poor starting region; and (c) parameter-efficient fine-tuning from a pretrained model can reduce that need by starting closer to a favorable ranking geometry. Experiments on 8 CV tasks and 8 NLP tasks, using ResNet50, DenseNet121, DistilBERT-base, and Qwen2.5-1.5B-Instruct, support these predictions.

1 Introduction

Direct pairwise AUC surrogates align training with the ranking metric, but that alignment does not ensure useful stochastic corrections. The paper frames this gap as Corrective Power and connects it to surrogate weighting and the starting region of pairwise training.

1 Introduction

  • In the controlled Figure 1 simulation, direct training moves weakly from a bad start, while squared-hinge succeeds after CE warm-up; hinge remains weak from a slightly better start, and both losses work from a good start.The comparison motivates requiring severe ranking errors to generate finite-batch updates that actively correct them.Source: §1 · p. 2, Figure 1 · p. 2
  • Pairwise AUC optimization can lose corrective strength when severe errors remain common but projected mean drift is weak and useful noise falls from order B^-1 to order B^-2.The projected escape-time scale correspondingly changes from order B to order B^2.Source: §1 · p. 2
  • The paper treats weak correction, surrogate choice, and starting-region interventions as connected aspects of why direct pairwise AUC training works or loses strength.It motivates an explanation beyond objective alignment alone.Source: §1 · p. 2
  • Squared-hinge increases correction with active-violation severity while turning off easy pairs, making its update weighting suited to difficult active pairs.The paper presents this as a weight-based explanation of surrogate behavior.Source: §1 · pp. 2–3
  • CE warm-up can help when the AUC phase starts in a poor region, while pretrained parameter-efficient fine-tuning can make separate warm-up less necessary by starting closer to favorable ranking geometry.The predictions are tested on computer-vision and NLP tasks using ResNet, DenseNet, DistilBERT, and Qwen models.Source: §1 · p. 3

2 Bad Regions in the Stochastic Dynamics of Pairwise AUC Optimization

Bad regions combine frequent severe misrankings with weak projected corrective motion, which can arise when first-order components of pairwise-gradient noise vanish. Under stated conditions, this variance collapse slows projected escape from order B to order B^2.

2 Bad Regions in the Stochastic Dynamics of Pairwise AUC Optimization

  • The correction direction is fixed before projected drift or variance is measured, targeting local reduction of loss from severely misranked pairs.Motion in other directions may affect easy pairs or other objective components without directly reducing hard-pair loss.Source: §2 · pp. 3–4
  • When the first-order Hoeffding components vanish, projected variance can fall from order B^-1 to B^-2, increasing the optimistic escape-time scale from order B to order B^2.A large mass of severe misrankings alone does not imply collapse; nearly constant severe-truncated row and column effects are a sufficient condition.Source: §2 · pp. 4–5
  • Figure 2(a)’s one-dimensional projected escape simulation shows 76% of runs cross the barrier by the plotted horizon with first-order B−1 noise, versus 3% with collapsed B−2 noise.Source: §2 · p. 5, Figure 2 · p. 5
  • A bad region has many severe misrankings but weak projected mean drift and a small or vanishing leading noise term along the severe-error correction direction.It need not be a local minimum, and the full trajectory may still progress in other directions.Source: §2 · pp. 3–5

3 Surrogate Losses and Their Weight Geometry

Pairwise surrogate losses differ in how their update weights allocate mass across margins. Among five canonical losses, squared-hinge both weights active violations by severity and silences already-separated pairs.

3 Surrogate Losses and Their Weight Geometry

  • Among logistic, exponential, square, hinge, and squared-hinge, only squared-hinge both ignores easy pairs and linearly upweights active violations by severity.On fully active mini-batches, squared-hinge adds a severity-dependent within-class correction to the class-mean score-gradient direction present in hinge.Source: §3 · pp. 5–6
  • Hinge treats all active violations equally, so when its class-mean direction is weak or misaligned it has no severity-dependent correction term.Squared-hinge responds to the distribution of active margins within the positive and negative classes.Source: §3 · p. 6
  • Logistic weights flatten in the severe tail, exponential can over-amplify it, and square continues assigning weight to already-separated pairs.In fully active settings, hinge can have zero projected mini-batch variance while squared-hinge retains a nonzero order-B^-1 term under class-score variation.Source: §3 · pp. 5–6

4 Changing Where Pairwise AUC Optimization Begins

The pairwise AUC phase depends partly on where optimization begins: CE warm-up can reduce severe misrankings, while pretrained parameter-efficient fine-tuning may begin in a more favorable region. The paper gives conditional bounds for these starting-state effects and their persistence during pairwise training.

CE warm-up

  • CE risk bounds the severe-misranking mass at the warm-up endpoint, linking pointwise warm-up to a more favorable start for pairwise AUC training.For every M > 0, π_M(θ) ≤ R_CE(θ)/log(1 + e^(M/2)); this does not require CE itself to optimize a pairwise surrogate.Source: §4 · p. 6
  • The warm-up argument is local: low misranking mass transfers to early pairwise training only if its trajectory remains near the warm-up endpoint.A poor endpoint or a trajectory that quickly leaves that neighborhood falls outside the argument.Source: §4 · p. 7

Pretrained parameter-efficient fine-tuning

  • Pretrained parameter-efficient fine-tuning may start in a favorable ranking region, with severe misrankings controlled and first-order stochastic motion retained along a contained trajectory.The finite-horizon transfer is conditional on a verified favorable trainable-subspace neighborhood and trajectory containment.Source: §4 · p. 7
  • A more favorable pretrained starting geometry can reduce the marginal value of CE warm-up, while weaker starting geometry leaves more room for warm-up to help.The pairwise gradient identity and Hoeffding split apply in the trainable coordinates of a head, adapter, or LoRA module.Source: §4 · p. 7

5 Experiments

Across the experiments, metric-aligned pairwise AUC training is not automatically reliable, and its performance depends on the surrogate and starting state. Trajectory and aggregate measures support the proposed corrective-power mechanism, while pretraining can reduce—but does not eliminate—the marginal value of warm-up.

RQ1: Is metric-aligned pairwise training enough by itself?

  • Direct pairwise AUC training can beat CE but also underperform it or other pairwise surrogates, and direct hinge may sustain high misranking ratios.The held-out AUC comparison is task by task; perturbations along the severe-error correction direction produce larger finite-horizon changes than equal-energy isotropic controls.Source: §5 · pp. 7–8, Table 1 · p. 8

RQ2: Which surrogate most consistently preserves useful correction?

  • Squared-hinge usually gives the best AUC among displayed DenseNet121 and Qwen settings and assigns greater relative weight to more severe wrong-margin intervals than hinge.The comparison is limited to the representative rows described; ResNet50 and DistilBERT-base results are reported in Table 2.Source: §5 · p. 8, Table 1 · p. 8

RQ3: When is CE warm-up helpful?

  • Warm-up gains are largest and most variable for hinge; direct squared-hinge can reduce misranking quickly and preserve stronger corrective motion, making warm-up less valuable.Figure 3 tracks projected first-order variance and misranking trajectories; Figure 4(a) summarizes warm-up gains.Source: §5 · p. 9, Figure 3 · p. 9, Figure 4 · p. 9

Stochastic mechanism check

  • Across APTOS, SNLI, and ANLI-R1, weakly corrective regions show more attenuated severe-component variance scaling than matched corrective regions.This matches the predicted shift toward the residual pair-interaction component as first-order projected motion weakens.Source: §5 · p. 9

RQ4: What changes under pretrained initialization and parameter-efficient tuning?

  • Pretrained initialization shifts and reduces the dispersion of warm-up gains, supporting lower average marginal benefit from warm-up while leaving it useful in some settings.The aggregate comparison matches same-backbone random and ImageNet-pretrained initialization; experiments cover eight CV and eight NLP tasks.Source: §5 · pp. 7, 10, Figure 4 · p. 9

Overall experimental takeaway

  • The experiments support a conditional prescription: preserve corrective weight on severe wrong pairs, and use warm-up or pretraining when it moves training into a state with corrective updates.The result is conditional on the surrogate and the trajectory's starting state.Source: §5 · p. 10

6 Conclusion

The paper explains why direct pairwise AUC training can lose corrective motion despite targeting the evaluation metric, and identifies loss weighting and initialization as ways to improve it. Its conclusion links this mechanism to U-statistic variance and reports supporting controlled simulations.

6 Conclusion

  • When first-order projected noise vanishes, useful mini-batch noise falls from order B−1 to B−2, changing the projected escape-time scale from order B to order B2.The conclusion identifies squared-hinge weighting, CE warm-up, and pretrained PEFT initialization as ways to improve corrective power; CV and NLP experiments are consistent with these predictions.Source: §6 · p. 10

A Related Work

  • Prior work covers AUC ranking, scalable optimization, surrogate validity, U-statistics, auxiliary-state methods, partial AUC, and warm-up or pretraining; this paper focuses on direct pairwise SGD dynamics.The related-work discussion frames the contribution as complementary to work making pairwise AUC computable at scale and analyzing statistical validity.Source: A Related Work · pp. 17–18

B.1 Details of the Controlled Two-Dimensional Simulation

  • The controlled two-dimensional simulation holds the landscape fixed while varying initialization and update rule to illustrate distinct direct-training and warm-up trajectories.Its configuration uses the same sample set and optimization settings across panels, which differ only in initialization.Source: §B.1 · pp. 18–19

B.2 Details of the Noise-Scale Escape Simulation

  • 76.275% of first-order-regime replicates escaped, versus 2.65% in the collapsed-variance regime.The corresponding figure reports these escape fractions as 76% and 3%.Source: §B.2 · p. 20

B.3 Main Experimental Protocol

The protocol compares held-out AUC for CE and five pairwise surrogate families, trained directly or after CE warm-up. It evaluates binary CV and NLP tasks constructed from public datasets, with task and split details specified separately.

B.3 Main Experimental Protocol

  • Each task is evaluated by held-out AUC, comparing CE with square, hinge, exponential, logistic, and squared-hinge pairwise losses.Pairwise training is run directly or after CE warm-up when both modes are included in the grid.Source: §B.3 · pp. 20–21
  • Results report mean held-out AUC and sample standard deviation across matching runs, using the validation-best checkpoint for each run.Hyperparameter choice and checkpoint selection are described in the protocol; the tables separate direct and warm-up rows.Source: §B.3 · pp. 20–21

B.4 Data Sources and Task Construction

  • The CV experiments use binary tasks from public image datasets, while NLP tasks use GoEmotions one-versus-rest detection and binary NLI contrasts from ANLI, SNLI, and WANLI.DistilBERT and Qwen use the same prepared NLP task definitions; official splits are used where available, with stratified splits otherwise.Source: §B.4 · pp. 21–22

B.5 Models, Optimization, and Reporting Protocol

The protocol specifies model configurations, AUC reporting, and training-mechanism measurements for direct and CE-warm-started pairwise training. Additional checks examine variance scaling, loss trajectories, and matched directional interventions.

Mechanism measurements

  • Mechanism logs measure training-batch misranking ratios and relative gradient weight by wrong-margin interval, with values above or below 1 indicating extra or reduced weight relative to pair frequency.These measurements use sigmoid-score differences and are computed on logged training mini-batches, not held-out test data.Source: §B.5 · p. 25

Practical diagnose–intervene–recheck protocol

  • The diagnostic protocol continues direct training when errors or corrective motion improve, and applies CE warm-up when both projected drift and first-order variance are weak.It estimates the severe-error direction on a selection batch and drift and variance on an independent logging batch, without a universal numerical threshold.Source: §B.5 · p. 26

Fixed-checkpoint batch-size variance scaling

  • Fixed-checkpoint variance scaling uses severe-error directions, batch sizes from 4 to 64, and three independent reference/logging-pool splits; Table 9 reports ranges of drift and fitted slope.Good and bad regions are classified by severe projected drift before slopes are fitted.Source: §B.5 · pp. 26–27, Table 9 · p. 27
  • The exact-degeneracy numerical control gives a log-variance–log-B slope of approximately −2.000, while measured severe projected variance remains 1.39 × 10^9 times its numerical floor.The exact-degeneracy control is not included in Table 9.Source: §B.5 · p. 27

Figure 5 results

  • On GoEmotions Annoyance and SNLI E/N, direct training remains separated from warm-started training for four losses, while squared-hinge shows much smaller separation.Figure 5 plots misranking trajectories for all five pairwise losses; black curves are direct training and colored curves are CE warm-up followed by AUC training.Source: Figure 5 · p. 27, §B.5 · pp. 27–28

Matched directional intervention

  • Matched interventions compare directional noise along the severe-error direction with equal-magnitude isotropic noise at fixed direct-hinge checkpoints, measuring test AUC and severe-misranking changes.The endpoints are measured after three intervention epochs relative to a shared no-noise baseline.Source: §B.5 · p. 28, Table 10 · p. 29

B.6 Multiclass, Multilabel, and Deep-AUC Extensions

The appendix extends AUC experiments to long-tailed multiclass, multilabel, and large-scale settings, and compares direct pairwise training with Deep-AUC. The validation-selected direct configuration is stronger in 13 of 17 task–model settings.

B.6 Multiclass, Multilabel, and Deep-AUC Extensions

  • Five CE, hinge, and squared-hinge training variants are evaluated on three long-tailed multiclass tasks using Macro-OVO-AUC and Macro-OVR-AUC.The tasks are Aircraft Airbus-6 LT and two CIFAR-100 subsets; training uses a randomly initialized ResNet18.Source: §B.6 · pp. 29–30, Table 11 · p. 29
  • Three CheXpert multilabel tasks are evaluated with a pretrained DenseNet121, patient-disjoint splits, and macro-average label-wise test ROC-AUC.Uncertain and missing labels are mapped to zero, and the official validation set is reserved for testing.Source: §B.6 · p. 29, Table 12 · p. 29
  • Large-scale multiclass experiments use natural long-tailed training distributions on iNaturalist 2018 and ImageNet-LT, with pretrained ResNet50 models.The reported large-scale results are in Table 13.Source: §B.6 · p. 30, Table 13 · p. 30
  • The squared AUC-margin objective admits an auxiliary-variable min–max reconstruction from class score means and second moments, unlike the listed margin-dependent surrogates.The comparison with Deep-AUC uses LibAUC AUCMLoss(v2) and the PESG optimizer.Source: §B.6 · p. 30
  • Deep-AUC is stronger in 4 of 17 task–model settings, while the validation-selected direct configuration is stronger in 13.Table 14 covers the 17 task–model settings and reports AUC%.Source: §B.6 · p. 31, Table 14 · p. 31

B.7 Assets, Licenses, and Terms

  • The experiments use existing public datasets, pretrained models, and standard software assets, whose sources and terms are documented without redistributing the assets.The paper says reproductions should obtain assets from their official sources and follow their associated terms.Source: §B.7 · pp. 31–32

B.8 Ethics, Broader Impacts, and Safeguards

  • The work introduces no deployed system or new data collection, and cautions that high-stakes deployment requires application-appropriate validation and safeguards.The reported experiments use existing public datasets and pretrained models.Source: §B.8 · p. 32

C Direct Pairwise SGD as a Stochastic Dynamics

The section models direct pairwise SGD as a stochastic process whose conditional drift and covariance can be analyzed from fresh mini-batches. It decomposes directional variance into first-order projections and a residual term, then extends the analysis to broader ranking objectives.

C Direct Pairwise SGD as a Stochastic Dynamics

  • Under local regularity and moment assumptions, the pairwise SGD process has a finite-time weak diffusion approximation with drift and covariance determined by its conditional moments.The section states the approximation for a Markov state with constant-step SGD dynamics.Source: C Direct Pairwise SGD as a Stochastic… · pp. 32–33
  • With fresh positive-negative mini-batches, direct pairwise gradients are unbiased and the conditional mean update is the negative population pairwise gradient scaled by the step size.The parameter process is Markov in the current parameter under the stated sampling setup.Source: C Direct Pairwise SGD as a Stochastic… · pp. 32–34
  • Directional variance splits into first-order Hoeffding projection terms scaling as B^-1 and a canonical residual scaling as B^-2 when the projections vanish.The latter scaling applies in the equal-size batch case.Source: C Direct Pairwise SGD as a Stochastic… · p. 36

C.1 Extensions Beyond Binary AUC

  • For multiclass, multilabel, partial-AUC, and top-k objectives, first-order projections must be aggregated across contributing comparisons, including any sampling fluctuation from estimated boundaries.Exact B^-2 scaling requires the complete aggregated first-order projection to vanish.Source: §C.1 · p. 36

D Bad Regions and Hoeffding Projection Collapse

This appendix characterizes high-error and severe-misranking regions where pairwise mini-batch SGD has weak corrective motion, including collapsed first-order stochastic projections and residual pair interactions. It provides conditions, constructions, and initialization-based bounds for identifying when this mechanism applies, and records how canonical pairwise-surrogate weights vary with positive-negative margin.

D Bad Regions and Hoeffding Projection Collapse

  • The two-sample Hoeffding decomposition separates first-order class projections from a residual pair-interaction term, with first-order variance scaling as B^-1 and residual variance as B^-2 when both class batch sizes grow proportionally.Unequal positive and negative batch sizes change only the constants on the class-projection terms; the same scaling holds when their sizes grow proportionally.Source: D Bad Regions and Hoeffding Projection… · pp. 39–40
  • A correction-direction bad region combines persistent high-error mass with weak projected drift and a small first-order Hoeffding contribution relative to residual pair-interaction variance.The criterion can apply to a fixed correction direction or uniformly across a cone of first-order corrective directions; it also covers the steepest severe-error direction when that direction lies in the cone.Source: D Bad Regions and Hoeffding Projection… · pp. 37, 40
  • Under severe-pair compression, increasing severe-misranking mass tightens an upper bound on first-order projections, but high severe-misranking mass alone does not guarantee projection collapse.Exact collapse requires fully severe pairs and constant conditional row and column means; the residual pair-interaction term can remain nontrivial.Source: D Bad Regions and Hoeffding Projection… · pp. 41–43
  • A construction shows that both first-order projections can vanish while the canonical pair-interaction variance remains positive, even when the severe-misranking mass reaches one.The appendix separately constructs a severe-misranking ray with mass one and a Rademacher example with zero first-order projections and nonzero residual variance.Source: D Bad Regions and Hoeffding Projection… · p. 43
  • The appendix gives high-error-start probability statements for Gaussian and label-blind initializations and a severe-misranking lower bound for random isotropic linear heads.The label-blind result assumes conditionally independent, identically distributed positive and negative scores; the isotropic-head passage establishes a lower bound of at least one half.Source: D Bad Regions and Hoeffding Projection… · pp. 43–44
  • High-dimensional scaled random heads retain nontrivial bad mass, and a lower bound on average severe-misranking mass implies a nontrivial probability of a high-error initialization.The initialization bound applies for thresholds τ below the lower bound µM on expected severe-misranking mass.Source: D Bad Regions and Hoeffding Projection… · p. 45
  • The empirical severe-misranking estimate is unbiased with variance at most 1/(4K), yielding deviation bounds that support finite-initialization lower bounds on high-error starts.The passages also give one-sided bounds for the true severe-misranking mass relative to its empirical estimate.Source: D Bad Regions and Hoeffding Projection… · pp. 45–46
  • Local score stability preserves severe-misranking neighborhoods: sufficiently nearby parameters retain the severe errors present at the reference parameter.The proposition assumes a score-change bound proportional to parameter distance and a margin buffer ρ.Source: D Bad Regions and Hoeffding Projection… · p. 47
  • In a high-error neighborhood, projected drift can be small while the first-order Hoeffding contribution to projected variance is at most ε1/B, leaving a nontrivial pair-interaction variance.If ε1 = 0, the B^-1 term vanishes and the projected variance has the stated B^-2 identity.Source: D Bad Regions and Hoeffding Projection… · p. 48
  • Near a stationary point, a Lipschitz risk gradient bounds correction in a fixed direction, so the neighborhood is bad for SGD when the stated high-error and variance conditions hold.The projected-drift bound is εdrift = LR · r.Source: D Bad Regions and Hoeffding Projection… · p. 48

E.1 Canonical Weights

  • The appendix compares logistic, exponential, square, hinge, and squared-hinge pairwise weights as margin changes; squared-hinge weight is 2 · (m − x)+ and hinge weight is 1{x < m}.The canonical-weight calculations specify the local weight profiles used in the main-text comparison.Source: E Surrogate Weight Calculations · p. 48, §E.1 · pp. 48–49

E.2 Active-Side Correction Versus Class-Mean Collapse

The results distinguish pairwise surrogates by how their weights preserve within-class correction, respond to severe errors, and shape mini-batch variability. They also formalize how warm-up and favorable pretrained starts can change the conditions under which direct pairwise training remains corrective.

E.2 Active-Side Correction Versus Class-Mean Collapse

  • On active batches, squared-hinge gradients retain a class-mean component and within-class score-gradient variation.The decomposition is 2 · (m − ¯s+ + ¯s−) · dB + 2 · cB.Source: §E.2 · pp. 49–50

E.3 Easy-Side Silence Versus Overshoot

  • Among logistic, exponential, square, hinge, and squared-hinge, only squared-hinge has nonzero affine weight slope on the active side and zero weight on the easy side.Its weight is 2 · (m − x) for x < m and zero for x ≥ m.Source: §E.3 · p. 51

E.4 Hard-Tail Alignment

  • For x < m − t, squared-hinge increases its pair weight by exactly 2 · t when the margin shifts from x to x − t.The comparison is restricted to pairs that remain active at both margins.Source: §E.4 · p. 52

E.5 Projected Stochastic Separation

  • Under the stated constant projected-gradient assumptions, hinge has zero projected variance, whereas squared-hinge retains a within-class variation term.The squared-hinge projected gradient contains the term 2 · cB.Source: §E.5 · p. 53

F Changing Where Pairwise AUC Optimization Begins

  • The formal results identify two routes to more favorable pairwise training conditions: CE warm-up can reduce severe misrankings, while parameter-efficient fine-tuning can begin and remain in a favorable neighborhood.For the latter, the stated conditions include controlled severe-misranking mass and persistent first-order projected stochastic motion over the finite horizon.Source: F Changing Where Pairwise AUC… · p. 53

F.1 CE Warm-Up and Severe-Misranking Mass

These sections formalize how CE warm-up can control severe-misranking mass and how favorable trainable-subspace conditions can support direct pairwise AUC training. They also state the local assumptions and model-class constructions underlying those conclusions.

F.1 CE Warm-Up and Severe-Misranking Mass

  • For every M > 0, severe-misranking mass is bounded by CE risk times log(1 + e^(M/2)).The bound is πM(θ) ≤ RCE(θ) log(1 + e^(M/2)).Source: §F.1 · p. 53

F.2 Parameter-Efficient Fine-Tuning in a Stable Trainable Subspace

  • A favorable trainable-subspace neighborhood with controlled severe-misranking mass and positive first-order projected variance supports finite-horizon direct training when the trajectory remains inside it.The paper lists favorable initialization, finite-horizon confinement, and pointwise transfer as checks for this conclusion.Source: §F.2 · p. 58

F.3 Local Pairwise AUC-Surrogate Descent in a Favorable Subspace

  • In a favorable trainable subspace, conditionally unbiased updates with the stated moment bound yield a finite-horizon descent bound; a local PL condition additionally gives contraction.The conclusion applies when the trajectory stays in the neighborhood over the finite horizon.Source: §F.3 · pp. 59–60

G Model-Class Realizations of Projected Bad Components

  • The constructed data-dependent Transformer neighborhoods realize weak drift, collapsed first-order stochastic variation, and nonzero residual pair-interaction variance along a specified trainable direction.These are existence statements for the constructed projected direction.Source: G Model-Class Realizations of… · p. 60

G.1 Data-Dependent One-Layer Construction

This section constructs a one-layer Transformer with persistent severe ranking errors but collapsed corrective drift and first-order gradient variation, then extends these conclusions to open parameter neighborhoods and deeper models. Its proof controls a full two-layer neighborhood while identifying the bad component in the projected direction vθ.

G.1 Data-Dependent One-Layer Construction

  • A one-head, one-layer residual Transformer can retain severe misrankings while its expected corrective drift is small and both first-order Hoeffding projections vanish.The construction uses a nonconstant, input-dependent attention weight and the pairwise square loss; its mini-batch variance is therefore due only to the residual pair-interaction component.Source: §G.1 · p. 60, §3 · pp. 60–62

3. The expected scalar gradient in the trainable direction satisfies

  • The scalar gradient kernel has zero first-order conditional projections, leaving only the canonical second-order term in mini-batch variance.The stated variance split follows because only centered pair terms with both indices matching have nonzero covariance.Source: §3 · pp. 61–62

G.2 Projected Bad Components in Trainable Transformer Neighborhoods

  • Full-parameter one- and two-layer Transformer neighborhoods contain severe misrankings with small projected drift and leading stochastic components, but a nonzero residual pair interaction.These are projected-component results for the constructed scalar-head direction, not claims about every trainable direction.Source: §G.2 · p. 63
  • The same one-layer pairwise square-loss dynamics, including their variance identities, can be embedded in two-layer and any finite-depth residual self-attention scorers.Additional identity residual blocks leave the scorer unchanged on the constructed input family.Source: §G.2 · pp. 64–65
  • The severe-misranking, drift, first-order projection, and residual-variance conclusions all hold on U(2).Source: §G.2 · p. 67
  • The proof controls a full two-layer parameter neighborhood, while the bad component is the projected direction vθ.Source: §G.2 · p. 67

H Localized Escape-Time Bounds for Stopped SGD

This section develops stopped-SGD escape-time bounds and applies them to projected corrective motion, translating directional variance scaling into lower-bound scales for escape time.

H Localized Escape-Time Bounds for Stopped SGD

  • The escape-time argument applies to stopped SGD trajectories independently of a specific loss, separating the probability bound from the AUC-specific variance calculation.The main-text application uses a projected interval along the optimistic steepest correction direction.Source: H Localized Escape-Time Bounds for… · p. 67
  • The section gives one-dimensional bounds for projected movement and a vector-valued local escape-time bound for stopped processes.The projected corollary applies these bounds to motion along a fixed direction.Source: H Localized Escape-Time Bounds for… · pp. 67, 71
  • Under the stated small-drift and small-step conditions, directional second moment O(B^-1) yields an order-B lower-bound scale for expected escape time, while O(B^-2) yields order B^2.The second case applies when the first-order stochastic term vanishes in the projected direction.Source: H Localized Escape-Time Bounds for… · p. 71
  • The two-sample Hoeffding variance split explains the scaling: nonzero first-order projections give a B^-1 term, while vanishing projections leave a B^-2 canonical term.For fixed step size, the centered variance contribution is multiplied by η^2; the corollary controls projected drift separately.Source: H Localized Escape-Time Bounds for… · p. 71

I Limitations

The analysis bounds projected escape only along the steepest severe-error correction direction, an optimistic direction for escape.

I Limitations

  • The analysis tests escape along the steepest severe-error correction direction; slow escape there leaves open whether other directions are even harder.The paper focuses on this optimistic direction because it cannot enumerate infinitely many possible directions.Source: I Limitations · p. 71

Related papers

Loading 2610.21743v1…