PaperlayerView exact revision on arXiv

Source-linked AI summary

Not All Objectives Are Born Equal: Priority-Constrained Descent for Hierarchical Multi-Objective Optimization

Dara Varam, Mohamed I. Alhajri

arXiv:2606.29521v2cs.LGstat.ML

TL;DR

Deep-learning optimization often pairs a primary task with secondary constraints, while standard multi-objective methods treat objectives symmetrically. PCD anchors updates to the primary gradient and minimally distorts them to guarantee secondary progress. The paper reports improved compression results over symmetric baselines and a controllable trade-off through τ.

  • Problem

    Standard multi-objective methods treat objectives symmetrically despite deep-learning problems often having a primary objective and secondary constraints.

    Source: §1 · pp. 1–2

  • Method

    PCD projects the primary gradient with minimal distortion needed to guarantee first-order progress on secondary objectives, with τ controlling the distortion.

    Source: §1 · p. 2, §4.2 · p. 7

  • Results

    PCD consistently outperforms symmetric baselines at matched compression; on CIFAR-100/ResNet-34 it achieves 70.9% accuracy at 90% sparsity versus 73.3% unpruned accuracy.

    Source: §1 · p. 3, §6 · p. 25

  • Takeaways & Limitations

    The reported compression results support a controllable trade-off for retaining the primary deliverable while meeting secondary constraints.

    Source: §6 · pp. 24–25

    The fixed-point characterization is pointwise, not a stability result: PCD directions can become arbitrarily small near non-CMS points.

    Source: §4.7 · p. 12

Abstract

from arXiv · show

Deep learning problems rarely involve objectives that are equal in importance. A primary objective defines the goal, whilst secondary objectives, such as sparsity, compression, or robustness constrain the solution. While existing multi-objective methods have proven effective in practice, they have a clear symmetry problem and neglect the inherent objective hierarchy built into these objective spaces. We introduce Priority-Constrained Descent (PCD), a gradient-based optimization framework designed to explicitly exploit hierarchical objective structures. PCD preserves the direction of primary descent whilst allowing for the minimal distortion necessary to guarantee progress on secondary objectives, controlled by a single $\tau \in [0, 1]$ that dictates the strength of the distortion. The resulting formulation is invariant to objective scaling and admits exact closed-form solutions for problems with two and three objectives. We evaluate PCD within structured network compression settings, unstructured sparsity and low-rankness, and across a variety of synthetic experiments, showing Pareto dominance and better per-objective performance with secondary progress guarantees over existing methods, further exhibiting the interpretable trade-off that $\tau$ provides.

1 Introduction

Deep-learning objectives often have an asymmetric hierarchy, but standard multi-objective methods treat them symmetrically. PCD anchors updates to the primary objective while minimally distorting them to secure secondary progress, and the introduction reports compression and synthetic results in its favor.

Motivation

  • Standard MOO frameworks overlook the designed hierarchy between a primary capability and secondary constraints, and symmetric methods can halt primary progress at a suboptimal conflict equilibrium.The paper frames that equilibrium as a failure mode rather than a feasible practitioner solution.Source: §1 · pp. 1–2

Figure 1

  • In the 2D toy problem, PCD reaches a joint minimum from all three starts, while weighted sum, MGDA, PCGrad, CAGrad, and FAMO stop with L2 non-zero.PCD crosses a saddle barrier to reach the point where both objective gradients vanish.Source: Figure 1 · p. 2, §1 · pp. 2–3

Priority-Constrained Descent

  • PCD solves a small convex QP that minimally distorts the primary gradient while guaranteeing first-order progress on secondary objectives; τ controls the distortion.The formulation uses per-objective scale-normalized margins and has exact closed-form solutions for two and three objectives.Source: §1 · pp. 2–3

Structured Network Compression

  • 70.9% accuracy at 90% sparsity on CIFAR-100/ResNet-34, versus 73.3% unpruned accuracy; PCD also exceeds tuned pruning-specific and constrained baselines at matched sparsity.The reported comparison holds throughout the compression regime; other comparisons fall within seed noise.Source: §1 · p. 3

Unstructured Pruning and Low-Rankness

  • 71% accuracy with 92.6% unstructured sparsity and effective rank 24.5 on CIFAR-100/ResNet-34; PCD is decisive on two of three axes and competitive on the third.The task uses ℓ1 and nuclear-norm objectives.Source: §1 · p. 3

Synthetic Verification

  • Synthetic experiments verify asymptotic scale invariance and feasibility as objective count increases, and report that PCD reaches CMS points without infeasibility.Source: §1 · p. 3

2 Related Works

Prior MOO work includes scalarization and direct gradient manipulation, but both retain limitations for hierarchical objectives. Related approaches introduce preferences or asymmetric mechanisms, while PCD places the hierarchy directly in update geometry.

Scalarization and Direct Directions

  • Weighted-sum scalarization collapses objectives before optimization but requires brittle tuning and cannot reach non-convex Pareto-front regions; direct-direction methods avoid scale sensitivity but retain objective symmetry.The section traces scalarization to Marler & Arora (2010) and direct Pareto descent to Fliege & Svaiter (2000).Source: §2 · p. 3

Preference Steering

Asymmetric Methods

  • AuxiNash assigns learned bargaining power to auxiliaries, while Cooper establishes hierarchy through Lagrangian multiplier dynamics.AuxiNash builds on Nash MTL’s proportionally fair update directions.Source: §2 · p. 4

Anchored Projections and Dynamic Barriers

  • Anchored methods impose progress constraints around a primary gradient; PCD’s distinction is per-objective scale-normalized margins that make one dimensionless τ usable across applications.The section relates PCD to DBGD, Pareto Navigation Gradient Descent (Ye & Liu, 2022), and GEM/A-GEM, equivalent to PCD at τ = 0.Source: §2 · p. 4

PCD’s Distinction

  • PCD projects the primary gradient onto a secondary-progress polyhedron, guaranteeing per-step first-order secondary progress and targeting composite multi-stationary rather than merely Pareto-stationary fixed points.This places the hierarchy in update geometry rather than learned coefficients or application-specific mechanisms.Source: §2 · p. 4

3 Preliminaries and Definitions

The setup formalizes optimization with one primary objective and convex secondary objectives. The stationarity definitions distinguish joint individual stationarity from Pareto stationarity, which can also arise through cancellation between objectives.

3.1 Problem Setup

  • The problem has K ≥ 2 objectives over θ ∈ R^n: a differentiable primary loss and secondary losses that are proper, convex, and lower semi-continuous.The stated class includes ℓ1, Group Lasso, nuclear norm, total variation, and hinge-type objectives.Source: §3.1 · p. 5

3.2 Stationarity Concepts

  • Composite multi-stationarity requires every objective to be individually stationary; Pareto stationarity also allows gradients to cancel, so a Pareto-stationary point need not be CMS.The section identifies such cancellation points as conflict equilibria where symmetric methods such as MGDA can halt without any objective being stationary.Source: §3.2 · p. 5

4 Priority-Constrained Descent

The section describes how PCD anchors updates to the primary gradient while imposing normalized secondary-progress constraints, with τ controlling the required progress. It characterizes the guarantees and conditions governing these updates, including when their normalized direction can vanish, and clarifies their dependence on the oracle, optimization path, and stationarity or trade-off regime.

4.1 The PCD Quadratic Program

  • For τ > 0, the feasible set is guaranteed nonempty for two objectives with a nonzero secondary gradient, but for K > 2 incompatible secondary constraints can make it empty.For K > 2, infeasibility occurs exactly when the origin lies in the convex hull of the nonzero secondary gradients.Source: §4.1 · p. 6

4.2 Geometry and KKT Structure

  • PCD computes the feasible direction closest in Euclidean distance to the primary gradient, with the primary coefficient fixed at one and nonnegative secondary corrections determined by active constraints.If the primary gradient already meets every secondary constraint, the update is unchanged; increasing τ shrinks the feasible set and can increase its distortion.Source: §4.2 · p. 7

4.3 Secondary-Progress Guarantee

  • When the feasible set is nonempty and τ > 0, PCD guarantees tolerance-controlled first-order progress on every secondary; smooth objectives strictly decrease for sufficiently small steps, while convex nonsmooth objectives incur a kink tax.The guarantee concerns the constructed direction; the kink tax vanishes unless the update moves mass into currently zeroed coordinates, groups, or singular directions.Source: §4.3 · pp. 8–9

4.4 Gradient Normalization

  • Per-objective EMA normalization makes PCD’s normalized gradients and QP solution invariant to positive objective rescaling in the limit ε → 0, unlike weighted-sum updates whose weight effects depend on raw magnitudes.This gives τ a scale-consistent interpretation across objectives with widely differing magnitudes.Source: §4.4 · p. 9

4.5 Closed Form Solutions and the Role of τ

  • For two objectives PCD has a closed-form projection, and with unit normalized gradients it preserves primary descent in nonconflicting cases but can lose it under sufficiently strong conflict as τ increases.For three objectives, active-regime multipliers also have a closed form via Cramer’s rule.Source: §4.5 · pp. 9–10

4.6 Algorithm

  • Algorithm 1 solves the two-objective QP in closed form and uses KKT working-set enumeration for K ≥ 3, with the QP adding negligible overhead relative to K backpropagations.The algorithm specifies branches for a zero primary gradient, a zero solution, or an empty feasible set.Source: §4.6 · p. 10

4.7 Scope of Guarantees

  • The guarantees characterize the geometric direction-construction step, not the optimizer’s realized update; Algorithm 1 has no proved convergence theorem, including convergence to CMS points or rates.Under deployed Adam, the secondary-constraint sign survives on 92.3–99.8% of steps; the SGD-with-momentum control flips it on 47–73%.Source: §4.7 · pp. 10–11
  • With adaptive optimizers, the trajectory-level bias of gradient-manipulation methods is not a per-step invariant and can lead to the same accuracy–sparsity frontier.The passage identifies this gap across PCGrad, CAGrad, MGDA, and FAMO.Source: §4.7 · p. 12
  • Theorem 4.8 is not a stability result: arbitrarily near non-CMS points, the normalized PCD direction can become vanishingly small.With a secondary gradient of norm δ anti-parallel to the primary, the QP gives ∥˜d∗∥ = τδ → 0 while ∥˜g1∥ stays fixed; this caveat applies to constrained-direction methods generally.Source: §4.7 · p. 12

4.8 Relation to Pareto Stationarity

  • For τ > 0, PCD’s normalized direction vanishes exactly when all oracle gradients vanish, ruling out gradient-cancellation endpoints of symmetric methods.The characterization implies CMS, but its converse requires returning 0 whenever 0 lies in each subdifferential; it is not a stochastic convergence claim, and τ = 0 recovers Pareto stationarity instead.Source: §4.8 · p. 12
  • For regularizer-style hierarchies, varying τ yields a continuous curve from Pareto-stationary to CMS endpoints; genuine trade-off problems have no CMS fixed point.With τ > 0 in genuine trade-off regimes, iterates need not converge; the framework targets the regularizer-style hierarchy of Eq. (1).Source: §4.8 · p. 12

4.9 Comparison with Existing Methods

  • Weighted sums, MGDA, PCGrad, and CAGrad encode no objective hierarchy, whereas PCD pins its update to the primary gradient and constrains secondary progress.PCD has three regimes—an inactive constraint, an active constraint without conflict, and an active constraint with conflict—rather than the two regimes described for existing methods.Source: §4.9 · pp. 12–13
  • When g1 = 0, PCD with τ > 0 still drives non-stationary secondaries toward stationarity, unlike MGDA, but deployed rescaling returns d∗ = 0.This advantage applies to the normalized direction ˜d∗, not Algorithm 1 as deployed.Source: §4.9 · p. 13

5 Experimental Results

The experiments evaluate PCD on structured pruning, unstructured sparsity and low-rankness, and synthetic problems, examining task performance, resource trade-offs, scale invariance, and multi-objective feasibility. In a separable conflict-equilibrium experiment, they also assess whether PCD and combination methods reach the composite multi-stationary point across tested values of K and τ.

5.1.1 Comparison with General Multi-Objective Methods

  • Across all eight structured-compression configurations, PCD exceeds every baseline at compression targets up to 95%; on ResNet-34/CIFAR-100 it retains 73.0% accuracy at 80–85% parameter reduction.At those targets, MGDA reaches 1.2%, FAMO 2.7%, PCGrad 7.4%, and CAGrad 7.3%. The exception above 95% is MobileNetV2/CIFAR-100 at 99%, where weighted sum reaches 1.9% against PCD’s 1.2%, and CAGrad ties, with accuracy collapsed for every method.Source: §5.1.1 · p. 14, Figure 4 · p. 15
  • 5.3× FLOPs reduction, 1.8× latency reduction, and 9.4× file-size reduction accompany PCD at τ = 0.01 on DenseNet-121/CIFAR-10 while matching unpruned accuracy (94.0% vs. 94.5%).At τ = 0.05, PCD reaches 45.4× FLOPs reduction and 83× file-size reduction at 89.2% accuracy. ResNet-34/CIFAR-100 shows the same pattern of small accuracy costs traded for order-of-magnitude resource savings.Source: §5.1.1 · pp. 14–15, Table 1 · p. 15
  • Across every shown compression level and resource axis, PCD’s accuracy–efficiency curve lies above baseline operating points, and its τ sweep traces the trade-off frontier.Baselines use their best swept operating points where applicable; PCD’s τ is an interpretable compression dial, with small values yielding lighter compression and larger values more aggressive compression.Source: §5.1.1 · pp. 15–17, Figure 5 · p. 16

5.1.2 Comparison with Tuned Pruning and Constrained Baselines

  • At ≥85% native sparsity, PCD wins 28 of 47 matched comparisons, loses 1, is within seed noise in 18, and Pareto-dominates 25 baseline points.The single loss is tuned scalarization’s 53.8% accuracy at 98.1% sparsity on ResNet-34/CIFAR-100, versus PCD’s 51.9% at 97.9% sparsity (τ = 0.09). At the same nominal τ = β = 0.01, PCD’s accuracy-retention range across four configurations is 1.09%, compared with DBGD’s 13.23%.Source: §5.1.2 · pp. 18–19, Table 3 · p. 19, Table 4 · p. 19

5.2 Unstructured Sparsity and Low-Rankness

  • In the three-objective setting, PCD traces a continuous accuracy–sparsity–effective-rank curve; its accuracy exceeds every baseline across low-to-moderate τ, while its effective rank is far lower at small τ.Baselines initially have greater sparsity, but PCD approaches their sparsity cluster near τ ≈ 0.2. Across configurations PCD retains the highest accuracy relative to dense networks; AuxiNash achieves lower effective rank on both Inception configurations but trails PCD by 30–40% in accuracy. PCD’s operating set dominates a substantially larger normalized hypervolume than any baseline, and leads under equal-cardinality and alternative-normalization checks.Source: §5.2 · pp. 20–21

5.3.1 Scale invariance of τ

  • With τ = 0.3, PCD’s operating point varies by only 0.0075 in L2 as the secondary is rescaled across ten orders of magnitude, while weighted sum (w = 0.5) traverses the entire front.Weighted sum’s achieved secondary ranges from 18.5 at c = 10^-4 to effectively 0 at c = 10^6; PCD’s EMA normalization makes τ transferable across objective scales.Source: §5.3.1 · p. 22, Figure 10 · p. 22

5.3.2 Feasibility and active set with more objectives

  • Across K = 2–8, none of 1,785 sampled secondary-gradient configurations made the QP infeasible; at K = 6, the active set grows from 2.54 to 4.96 of five constraints as τ approaches 1.Each secondary binds with high frequency for τ ≳ 0.05, with binding distributed roughly uniformly rather than concentrated on a subset.Source: §5.3.2 · pp. 22–23, Figure 11 · p. 23

5.3.3 Conflict equilibria and CMS escape as the number of objectives grows

  • At the conflict equilibrium, MGDA and CAGrad halt exactly, while the primary loss they leave unconverged grows linearly in K.At K = 5, their learning curves remain pinned at a primary gradient of 3.07; WS is omitted because its behavior depends on weights.Source: §5.3.3 · pp. 23–24
  • ∼10^-15 primary gradient and ∼10^-29 primary loss at convergence show PCD reaches the composite multi-stationary point for every tested K and every τ > 0.PCD’s primary coefficient is pinned at one, and its update is structurally not a convex combination; Theorem 4.8 says its normalized direction vanishes only when every oracle gradient does.Source: Figure 12 · p. 24, §5.3.3 · p. 24
  • PCGrad traps at K = 2 but reaches CMS for K ≥ 3 through pairwise gradient projection, whereas AuxiNash drifts into the barrier and leaves the largest primary loss.The authors note that PCGrad’s escape is specific to this construction’s separable geometry; it is dominated at every compression target in the non-separable compression landscapes of Sec. 5.1.Source: §5.3.3 · p. 24
  • The K −1 scaling law applies to the combination methods, while PCD’s fixed points are exactly CMS points under the stated oracle hypothesis.The construction is separable across the secondary directions, and the conflict equilibrium requires all K gradients to balance with equal weight.Source: §5.3.3 · p. 24

6 Conclusion

PCD encodes objective hierarchy by anchoring updates to the primary gradient and minimally distorting it to maintain first-order secondary progress. Across compression and synthetic experiments, it offers a controllable trade-off, subject to stated theoretical and deployment limitations.

6 Conclusion

  • PCD minimally distorts primary-gradient updates to maintain first-order secondary progress, with an interpretable scalar τ controlling the trade-off.Its theoretical guarantees rely on proper, convex, lower semi-continuous secondary objectives.Source: §6 · pp. 24–25
  • PCD provides a stable, controllable trade-off curve and consistently outperforms symmetric baselines at matched compression across the reported settings.The conclusion describes results across neural-network compression and synthetic experiments.Source: §6 · p. 25
  • The theoretical guarantees exclude inherently non-convex constraints, and extending convergence guarantees to the deployed normalized stochastic update remains open.For K > 2, mutually incompatible secondary constraints could also make the quadratic program infeasible.Source: §6 · p. 25
  • Future directions include adaptive τ schedules, multi-level priority hierarchies, and applications with hardware-aware constraints or structured generative modeling.The paper identifies adaptive selection of per-objective tolerances without exponential grid search as an open challenge.Source: §6 · p. 25

A Roadmap of results

The roadmap links PCD's theoretical results and their consequences to the paper's algorithm, while the appendices expand experimental and theoretical details. The framework also permits per-objective tolerances, although the paper uses a shared scalar.

A Roadmap of results

  • Appendices E, F, and G expand structured-pruning results, three-objective experiments, and measurements of the per-step guarantee, respectively.Appendix B.8 contains the nonsmooth-case analysis and closed forms for the kink tax.Source: A Roadmap of results · p. 29
  • Figure 13 maps PCD's foundations, core theorems, corollaries, guarantees, endpoint, and corner case through to Algorithm 1.Its nodes link to the corresponding statements, making the figure a navigable table of contents for the technical material.Source: Figure 13 · p. 29, A Roadmap of results · p. 29

B Full Details and Proofs for Section 4

  • PCD generalizes to per-objective tolerances τj without changing its proofs or algorithm, but experiments use one shared τ.The per-objective tolerances add flexibility to the framework.Source: B Full Details and Proofs for Section 4 · p. 29

B.1 Feasibility Analysis

The appendix characterizes when PCD's secondary-progress constraints are feasible and how to compute the closest feasible direction to the primary gradient. It also supplies low-objective-count closed forms, a projection interpretation, and boundary-case behavior.

B.1 Feasibility Analysis

  • For τ > 0, infeasibility occurs exactly when zero lies in the convex hull of the nonzero normalized secondary gradients; with K = 3, this requires anti-parallel gradients.For K = 2, a single nonzero secondary gradient always yields a feasible constraint set.Source: §B.1 · pp. 29–30

B.2 KKT Proof and Active-Set Solver

  • When feasible, the unique optimizer is the projection of the primary gradient onto the constraint polyhedron, and a KKT-valid active set yields that optimizer.The enumeration checks at most 2K−1 working sets after computing the Gram matrix once.Source: §B.2 · pp. 30–31

B.3 K=2 and K=3 Closed Forms

  • For K = 2, the solution is either the primary gradient or its constraint-adjusted direction; for K = 3, the both-active closed form requires non-collinear secondary gradients.The K = 3 active-set formula comes from a 2 × 2 Gram system.Source: §B.3 · p. 32

B.4 Geometric Interpretation: Half-Space Projection

  • Geometrically, the K = 2 solution projects the primary gradient onto a half-space, while general K projects it onto the feasible polyhedron.The active set identifies the faces containing the projection.Source: §B.4 · p. 32

B.5 Boundary Cases

  • When the primary gradient is zero, the computed PCD direction can still drive non-stationary secondaries, but the deployed rescaling halts at exactly that point.If all secondary gradients are zero, the solution instead reduces to the primary gradient.Source: §B.5 · pp. 32–33

B.6 Scale Invariance Proof

The appendix establishes scale invariance for PCD's normalized direction and contrasts it with weighted sums, then clarifies per-step progress behavior and the treatment of nonsmooth secondary objectives. It also states where the smooth-case guarantee weakens near sparsity.

B.6 Scale Invariance Proof

  • PCD's direction is asymptotically invariant to positive objective rescaling, and the alternative normalization makes this invariance exact; weighted-sum directions generally change.The weighted-sum direction depends on relative objective magnitudes.Source: §B.6 · p. 33
  • With deployed ε = 10^-8, scale invariance degrades as gradient norms approach √ε.The paper reports this finite-ε residual as a practical qualification to the invariance result.Source: §B.6 · p. 33

B.7 Per-Step Progress and MGDA Failure

  • When g1 = 0, MGDA returns a zero direction, whereas the computed PCD direction continues secondary progress; deployed rescaling also halts at exactly g1 = 0.The PCD per-step advantage therefore applies to the computed direction, not the deployed update at that boundary.Source: §B.7 · p. 33

B.8 The Nonsmooth Case: the Kink Tax and its Closed Forms

  • The kink tax measures the gap between an oracle-subgradient linearization and the one-sided directional derivative, and vanishes unless an update moves mass into zeroed coordinates, groups, or singular directions.The appendix gives closed forms for the ℓ1, group-lasso, and nuclear-norm penalties used in the paper.Source: §B.8 · p. 34
  • As groups shrink, the smooth-case step window collapses and its guarantee degrades to the kink-tax form in the sparse limit.The paper identifies this as a limitation of reading the guarantee as a smooth-case result.Source: §B.8 · p. 35

C.1 Proof of Theorem 4.8

For τ > 0, the normalized QP’s zero direction characterizes composite multi-stationarity, subject to an oracle qualification for the converse. At τ = 0 the fixed-point condition instead recovers an MGDA-style Pareto-stationarity guarantee, while the convergence sketch is conditional and certifies primary stationarity only.

C.1 Proof of Theorem 4.8

  • For τ > 0 and a nonempty feasible set, the normalized QP returns zero exactly when all normalized objective gradients are zero.The converse identification with composite multi-stationarity requires the oracle to return zero whenever zero belongs to a secondary subdifferential.Source: §C.1 · p. 35
  • The zero-direction equivalence holds uniformly across problem sizes because its proof uses primal feasibility and KKT stationarity, not the number of objectives or variables.The oracle qualification matters for penalties such as the nuclear norm; the stated hypothesis holds for ℓ1 and group-lasso penalties under the described automatic-differentiation oracle.Source: §C.1 · p. 35

C.2 Fixed Points with τ = 0

  • At τ = 0, a zero direction requires the negative primary gradient to lie in the cone of secondary gradients, recovering an MGDA-style Pareto-stationarity guarantee.For positive tolerances, fixed points reduce to composite multi-stationarity only under the oracle hypothesis.Source: §C.2 · pp. 35–36

C.3 Sufficient-Decrease Sketch

  • The sufficient-decrease sketch conditionally gives min_{t<T} ||g1,t||^2 = O(1/T), but certifies primary stationarity rather than convergence to a composite multi-stationary point.It assumes smoothness, a lower-bounded primary loss, feasible QPs, bounded iterates, and a positive alignment bound; that alignment can fail under severe conflict.Source: §C.3 · p. 36

D Feasibility at K > 2

The appendix gives a sufficient angle-based feasibility condition for multiple secondary objectives, then reports expanded structured-pruning results and two analyses of gradient normalization. The baseline comparison finds that normalization helps MGDA but does not close the reported operating-range gap with PCD.

D Feasibility at K > 2

  • Under the stated sufficient angle condition, the secondary-gradient cone excludes zero and the feasible set is nonempty; for K = 3, feasibility holds unless the two secondary gradients are exactly anti-parallel.The bound is sufficient but not necessary; for K ≥ 4 it is tight at the equilateral configuration.Source: D Feasibility at K > 2 · pp. 36–37

E Expanded Results: Structured Pruning

E.1 Ablation: Gradient Normalization in PCD

  • Disabling gradient normalization costs 11.2–23.9 accuracy points, while EMA at τ = 0.01 delivers +0.7% to +20.7% group sparsity at an accuracy cost of at most 0.20.Across β ∈ {0.9, 0.99, 0.999}, accuracy varies by 0.16% to 1.93%, against seed standard deviations of 0.03 to 3.20.Source: §E.1 · p. 37

E.2 Baselines under PCD’s Normalization

  • Across the normalized-baseline grid, methods occupy a narrow band above 98.67% sparsity, while PCD’s τ traverses 76–99% sparsity.EMA normalization lifts MGDA from 1.23% accuracy at effectively 100% sparsity to 33.29%, but does not explain the remaining performance gap.Source: §E.2 · p. 42

E.3 Mechanism Measurements During Training

Training measurements show that PCD’s secondary constraint is active and met at every step, while its update remains close to the primary direction. Additional studies document tuned-baseline failures and qualify matched-sparsity comparisons for unstructured pruning and low-rankness.

E.3 Mechanism Measurements During Training

  • The secondary constraint is active at every measured step, and post-solve slack is zero to machine precision in 14 of 16 configurations.The deployed direction stays within 13° of the pure cross-entropy direction at τ = 0.2; measured conflict is negligible in magnitude.Source: §E.3 · p. 42

E.4 Tuned-Baseline Study: Full Range and Per-Configuration Tables

  • Cooper does not enter the compression regime, while tightening ε destabilizes it before compression and a 100× dual-learning-rate sweep moves sparsity only from 43.5 to 52.8%.The full-range study also reports collapse for proximal methods when pushed toward compression.Source: §E.4 · p. 44, Figure 16 · p. 44

F Expanded Results: Unstructured Sparsity and Low-Rankness

F.1 Matched-Sparsity Comparison

  • At matched unstructured sparsity, PCD has an accuracy advantage where comparisons are defined, but MGDA, PCGrad, and FAMO never reach its sparsity range.The effective-rank axis does not support dominance: the matched-range minimum is negative against AuxiNash on three of four configurations.Source: §F.1 · p. 49

F.2 Hypervolume Robustness

The robustness analyses test whether PCD’s hypervolume advantage depends on choosing many τ values, selecting one operating point, or the normalization frame. The reported ordering remains favorable to PCD across these variants.

F.2 Hypervolume Robustness

  • Even the worst subset of 10 τ values exceeds the strongest baseline on every configuration, with ratios of 1.40/1.27/1.37/1.36.The test evaluates all 3,003 subsets; τ and baseline weight w are not interchangeable, and using the baseline grid would remove useful τ < 0.1 operating points.Source: §F.2 · p. 50
  • Selecting one hypervolume-maximizing point per method yields PCD-to-strongest-baseline ratios of 2.60/2.14/1.32/1.48.Selection is by each method’s own hypervolume rather than accuracy alone.Source: §F.2 · p. 50
  • Under alternative normalization frames, PCD’s ratios over the strongest baseline change, but the method ordering never changes.The reported ratios are 3.41/3.38/2.54/2.75 with failed-run bounds and 2.43/2.78/2.80/4.03 in a global frame.Source: §F.2 · p. 50

G Adam and the Sign of the Guarantee

Under deployed Adam, the secondary-progress sign survives on nearly all steps, while SGD with momentum often flips it yet reaches the same accuracy–sparsity frontier. The SGD control reaches that frontier later, and the guarantees apply to the constructed direction rather than necessarily to the optimizer’s realized step.

G Adam and the Sign of the Guarantee

  • The study logs sign flips and promised versus realized secondary alignment at every step across three-seed runs, two configurations, two tolerances, and both optimizers.The comparison includes deployed Adam and an SGD-with-momentum control with momentum 0.9.Source: G Adam and the Sign of the Guarantee · p. 50
  • Across four configuration–tolerance cells, SGD with momentum plateaus 22–66 epochs later than Adam under the same 300-epoch cosine schedule.Plateau is measured as the first epoch within one point of the run’s own final accuracy.Source: G Adam and the Sign of the Guarantee · p. 50
  • Under deployed Adam, the secondary sign survives on 92.3–99.8% of steps, with realized alignment at least 5× the QP’s guaranteed margin.Table 18 measures sign preservation across all steps of 300-epoch runs using three seeds.Source: G Adam and the Sign of the Guarantee · p. 50, Table 18 · p. 51
  • Despite sign flips on 47–73% of steps, SGD with momentum reaches the same accuracy–sparsity frontier as deployed PCD at both tolerances.The caption identifies the SGD control as open stars; the prose reports matched-sparsity comparisons on ResNet-34/CIFAR-100 and DenseNet-121/CIFAR-10.Source: G Adam and the Sign of the Guarantee · p. 50, Figure 17 · p. 51
  • The guarantee concerns the constructed direction d*, not necessarily the optimizer-realized step, whose final accuracy differs by at most 2.6% in either direction.The passage notes that this gap is shared by gradient-manipulation methods handing a constructed direction to an adaptive optimizer.Source: G Adam and the Sign of the Guarantee · p. 51

H Reproducibility Details

The reproducibility details specify the default training setup, operating-point selection, seed reporting, and compute requirements. They also report a forthcoming reproducibility checklist with the code implementation.

H Reproducibility Details

  • Default runs used Adam with β1=0.9 and β2=0.999, learning rate 10^-3, batch size 128, cosine annealing, and 300 epochs.No warm-up or weight decay was used; five seeds were standard, with three for ablations.Source: H Reproducibility Details · p. 51
  • Swept baselines use their highest mean test-accuracy operating point, while τ=0.02 is the representative fixed setting unless otherwise stated or swept.No held-out validation split was used.Source: H Reproducibility Details · p. 51
  • PCD’s Table 5 accuracy standard deviation was 0.15–0.40 across four configurations, versus a 1.26 baseline average and 6.14 maximum; authors report no conclusions changed with seed noise.Means alone appear in the main table for readability; seed standard deviations accompany the specified tables.Source: H Reproducibility Details · p. 51
  • K=2 PCD deployment took 3.64 ± 0.03 hours versus 3.58 ± 0.03 hours for weighted sum; K=3 PCD’s wall-clock ratio to weighted sum was 1.63 to 1.95.The K=2 measurement includes EMA normalization, QP solving, magnitude rescaling, and per-step instrumentation; K=3 repeated SVDs are used by all seven methods.Source: H Reproducibility Details · p. 52
  • A more comprehensive reproducibility checklist is planned to accompany the paper’s code implementation.Source: H Reproducibility Details · p. 52

Related papers

Loading 2606.29521v2…