Source-linked AI summary
A Complete Resolution of Forsythe's Conjecture for Restarted Conjugate Gradients
Matthew J. Colbrook, George Stepaniants, Alex Townsend
TL;DR
The paper resolves whether restarted conjugate gradients universally terminate or approach a two-cycle in their normalised residual directions. It combines a degree-independent spectral argument with low-degree fixed-set analysis and a Hopf-based construction. The conjecture is true exactly for s∈{1,2,3} and false for every s≥4, with diagonal SPD counterexamples of dimension s+4.
Problem
Forsythe’s conjecture asks whether every nonterminating exact-arithmetic restarted-CG iteration on an SPD problem has separately convergent even and odd normalised residual directions.
Method
The proof uses double orthogonality and fixed-set analysis for the positive cases, then certifies a transverse Hopf point and applies periodic-orbit, shadowing, and degree-elevation arguments for counterexamples.
Results
The termination-or-two-cycle alternative holds universally for s∈{1,2,3}, while for every s≥4 it fails for a diagonal positive definite matrix of dimension s+4.
Takeaways & Limitations
The conjectured universal conclusion is completely classified in finite dimensions, with restart length four as the sharp threshold between convergence and nonconvergence.
Abstract
from arXiv · showhide
Forsythe's conjecture, published in 1968, asserts that for each restart length $s$, every exact-arithmetic restarted conjugate-gradient iteration on a real symmetric positive definite problem either terminates or has normalised residuals that converge separately along the even and odd restart subsequences. Apart from the classical steepest-descent case, this asymptotic question remained unresolved in full generality for nearly six decades. We give a complete classification by restart length in the original finite-dimensional setting and identify a sharp threshold. For $s=2$ and $s=3$, every problem either terminates or has convergent even and odd residual directions. For every $s\ge4$, there is a diagonal positive definite counterexample of dimension $s+4$ which never terminates and whose even residual directions do not converge. Together with Akaike's theorem for $s=1$, this shows that the conjectured universal conclusion is true precisely for $s\in\{1,2,3\}$ and false for every $s\ge4$. The positive results follow from a degree-independent double-orthogonality identity and an analysis of the low-degree fixed-point sets. At restart length four, rational interval arithmetic and Sturm sequences certify a transverse Hopf point of the leading vector field of the rescaled squared-weight map. Analytic periodic-orbit and shadowing arguments yield the counterexample at restart length four, and degree elevation extends the construction to every larger restart length.
1. Introduction
The paper resolves Forsythe’s conjecture by proving a sharp threshold at restart length four: universal two-cycle convergence holds for s=1,2,3 but fails for every s≥4. Its proof combines degree-independent spectral identities with low-degree fixed-set analysis and a Hopf-based counterexample construction.
- Problem: Forsythe’s conjecture asks whether every nonterminating restarted-CG sequence has separately convergent even and odd normalised residual directions.The question concerns directional dynamics, which classical residual-norm convergence estimates do not determine.
- Classification: For s∈{2,3}, every exact-arithmetic SPD instance either terminates or has convergent even and odd residual directions.Together with Akaike’s theorem for s=1, this establishes the conjectured alternative for restart lengths 1, 2, and 3.
- Classification: For every s≥4, a diagonal SPD instance of dimension s+4 never terminates and has nonconvergent even normalised residuals.Thus the universal termination-or-two-cycle alternative fails at every restart length at least four.
- Counterexamples: At restart length four, a certified transverse Hopf point generates a periodic slow profile that an exact restarted-CG orbit shadows; degree elevation extends this mechanism to all larger restart lengths.Finite inequalities are computer-certified, while the periodic-orbit, shadowing, and degree-elevation arguments are analytic.
- Proof framework: A degree-independent double-orthogonality identity yields convergence of even and odd orthogonal factors and localises nonterminating limits to supports containing between s+1 and 2s eigenvalues.The associated monotone two-block energy drives the spectral analysis.
Part A. Convergence at restart length two
For restart length two, the support bound leaves only three- and four-node accumulation points, and convergence reduces to controlling motion on a compact one-parameter family of exact four-node two-cycles.
- Part A: At restart length two, accumulation points have either three or four active spectral nodes.The three-node cases are rigid, while four-node limits form a compact one-parameter family of exact two-cycles.
- Part A: Convergence at restart length two is reduced to controlling motion along the compact family of four-node two-cycles.The family structure isolates the remaining asymptotic degree of freedom.
A.1. The theorem for restart length two and the spectral map
For restart length two, the iteration either terminates or its normalized residual directions converge separately along even and odd subsequences. Spectral reduction and a two-block polynomial map reduce this result to convergence within three- and four-node limiting geometries.
- Theorem A.1.1 states that every restart-length-two iteration either terminates or has convergent even and odd normalized residual directions.
- Spectral reduction: Spectral reduction represents each normalized residual using fixed orthonormal spectral vectors and associated active-node weights.Repeated eigenspaces contribute only along the line generated by the initial projection.
- The spectral map: A restart-length-two block is governed by a monic quadratic orthogonal polynomial, with residual update π = P_w/P_w(0) and invariant zero coordinates.Termination occurs exactly when the active support has at most two nodes.
- The two-block map: The two-block dynamics has a monotone energy whose increments are summable, forcing the even and odd polynomial factors and their quartic product to converge.The same energy also yields convergence of same-parity displacement.
- Spectral localization: Every ω-limit support lies in a resonant set of at most four nodes, while nonterminating limits retain at least three active nodes.The support bound follows because the limiting monic quartic minus its energy level vanishes on every active node.
- Limiting geometry: Three-node limits are rigid, whereas four-node limits form a compact one-parameter family of exact two-cycles whose tangential motion is nevertheless convergent.External neutral modes are eliminated, and both parity weight sequences converge in the four-node case.
B.1. The theorem for restart length three
For restart length three, restarted CG either terminates or its normalized residual directions converge separately along even and odd restart subsequences. The proof reduces the iteration to spectral weights and signed residual directions, with permanent support loss and termination characterized algebraically.
- Theorem: At restart length three, every SPD problem either terminates or has even and odd residual directions converging to unit vectors y_e and y_o.The result is stated for every dimension, SPD matrix, and initial pair.
- Numerical illustration: The direct computation uses A=diag(1,...,10), equal initial coordinates, and 120-digit Galerkin solves; both parity distances tend to zero.The plotted limits are fixed points of F^2, equivalently a two-cycle of the one-block map F.
- Spectral reduction: The squared spectral coordinates determine the monic orthogonal cubic, while coordinate signs recover the residual direction.The parity return is the two-block map, so squared weights are analyzed before signed directions.
- Spectral reduction: A zero coordinate remains zero, making support loss permanent; when at most three active nodes remain, the next block terminates.For more than three active nodes, the relevant moment and Gram systems remain nondegenerate for the block computation.
B.3. Parity energy and ω-limit sets
For restart length three, a finite parity energy forces accumulation states onto regular signed two-cycles with shared support. Uniform separation from small supports and a phase-rigidity argument then constrain the omega-limit sets and restore signed convergence.
- Parity energy: The parity energy is nondecreasing and bounded, so its increments are summable and the associated parity motion becomes asymptotically rigid.The interlaced monic-norm quantities share a positive finite limit.
- Support separation: Every omega-limit state has at least four positive coordinates, and its degree-three moment Gram matrix is uniformly nonsingular.This excludes limiting supports of size at most three and provides uniform control of the moment equations.
- Omega-limit localisation: Each parity omega-limit state has between four and six positive coordinates, and paired accumulation points have the same support and form an exact signed two-cycle.The support upper bound follows from a degree-six polynomial vanishing on every support node.
- Omega-limit localisation: The parity omega-limit sets and paired omega-limit set are compact and connected because successive parity steps tend to zero.Connectedness prevents the limiting sequence from switching indefinitely between separated compact pieces.
- Phase rigidity: Phase rigidity shows that every signed lift of a regular weight two-cycle is a signed two-cycle rather than merely a projective cycle.The sign analysis rules out a nonempty negative-sign subset on the common support.
- Phase rigidity: Consequently, accumulation pairs lie in the signed fixed set of the two-block map, providing the fixed-point coordinates needed to prove convergence.The fixed-set analysis must remain regular through common-root collisions and support loss.
B.4. Algebra of fixed two-cycles and collision coordinates
The fixed points of the two-block return are parameterized by completion polynomials and regular collision coordinates across four-, five-, and six-node supports. This algebraic description identifies the fixed-cycle equation and remains valid through common-root collisions and support losses.
- Collision coordinates: The completion coordinates remain regular when nodes are lost or the phase cubics acquire a common root, enabling parity-energy decay to be converted into convergence.The section’s lemmas combine fixed-component structure, collision extensions, and drift comparison for this purpose.
- Fixed-point coordinates: A regular fixed state with support size four through six admits unique monic completion polynomials of degrees |J|-4 and 6-|J|.These polynomials complete the spectral nodes, with multiplicity, to a monic sextic associated with the two phase cubics.
- Fixed-point coordinates: For six fixed nodes, the completion quadratic has roots (ρ,σ) in [λ_2,λ_3]×[λ_4,λ_5], with the rectangle interior, edges, and corners representing six-, five-, and four-node supports.The parameter rectangle captures support loss without changing the fixed-cycle description.
- Fixed-cycle equation: The affine normal equation U=0 is exactly the fixed-cycle equation near a coprime fixed component.The local return has quadratic leading drift in the completion coordinates.
- Fixed-cycle equation: The fixed relation PQ=D+C holds on the six-node stratum, and equality of the two squared monic norms yields exact two-block return.Conversely, a signed two-block return implies this polynomial identity and therefore U=0.
- Collision coordinates: The collision formulas have finite, strictly positive limits at off-node common-root collisions, so the coordinates extend across those degeneracies.The collision ideals are reduced and radical in the regular local analytic state ring.
- Collision coordinates: On four nodes, the two-block map is exactly the identity, while the fixed-point set is a reduced principal complete intersection.The identity gives zero drift on the four-node stratum.
B.5. Convergence of the completion polynomial and external modes
The completion polynomial and its cubic factors converge, while external spectral modes split into stable, unstable, and persistent neutral regimes. This classification reduces the remaining dynamics to a convergent limiting core plus controlled external corrections.
- Completion convergence: The completion data converge to a monic sextic D*, monic cubics P* and Q*, and C*=τ^2>0, with P*Q*=D*+C*.The coefficient errors satisfy a quantitative bound controlled by τ−e_a,n.
- Completion convergence: Every active node of an ω-limit state is a root of D*, so the limiting support contains at least four and at most six nodes.The upper bound follows because D* is a nonzero monic sextic.
- External modes: External modes are classified by their limiting multiplier: stable modes decay exponentially, unstable modes are eventually deleted, and persistent neutral modes have multiplier −1.This trichotomy determines which external coordinates can continue influencing the dynamics.
- External modes: When a neutral mode persists, stable weight is asymptotically smaller than neutral weight, and the core deviation is controlled by the parity-energy tail.The estimates include cX_n≤−x_n≤CX_n and |x_n|+∥U_n∥≤CX_n.
- Core reduction: The normalized limiting core is constructed from the exact full state, while restricted core quantities provide analytic and uniformly controlled approximations.Moment-Gram invertibility and polynomial identities support the uniform estimates, including support-loss and collision strata.
B.6. Intrinsic convergence and removal of stable modes
The intrinsic limiting face converges, and stable external coordinates can be removed without changing the limiting behavior. The resulting shadowing estimates also control stable forcing when neutral modes persist.
- Stable-mode removal: The intrinsic dynamics are separated from external forcing by combining convergence on the six-node face with shadowing after removing summable stable coordinates.Stable forcing is negligible whenever neutral modes persist.
- Intrinsic convergence: An orbit confined to the six-node limiting face converges to one fixed point, uniformly through collisions and asymptotic support loss.The conclusion also covers simple or simultaneous off-node common roots and core weights tending to zero without exact deletion.
- Intrinsic convergence: The limiting core’s completion coordinates have a coercive drift: A_ρ≤−c_ρ<0, A_σ≥c_σ>0, and the quadratic terms remain uniformly bounded.These signs yield summable directional variation of the root coordinates.
- Intrinsic convergence: The completion factors and core coordinates converge coordinatewise, including tails in which one or two core weights tend to zero without becoming exactly zero.Exact deletion places the remaining tail on an invariant five- or four-node face.
- Stable-mode removal: After stable modes are removed, the full tail is exponentially close to an exact orbit on the invariant limiting face; with persistent neutral modes, stable forcing is o(E_neu,n).For quantities even in deleted signed amplitudes, the comparison error improves to O(E_st,n).
B.7. Four-node limits and five-node limits without persistent neutral modes
Four-node two-block dynamics are the identity, while five-node fixed states form a one-dimensional pivot interval whose exact tails still converge. This resolves all low-support cases without persistent neutral external modes.
- Four-node limits: The exact signed and squared two-block return on four nodes is the identity.The factor relation gives P(λ_i)Q(λ_i)/(ab)=1 at every retained node.
- Five-node limits: For five nodes, the strict positivity conditions define one bounded open pivot interval whose endpoints each correspond to one vanishing weight.The other four endpoint weights remain positive.
- Five-node limits: Every exact five-node tail without external forcing has a convergent squared parity state, including tails that eventually delete a coordinate.Deletion is permanent and reduces the problem to the four-node case.
- Five-node limits: The pivot is eventually monotone up to a single sign choice and therefore converges within its bounded interval.The remaining cubic factors, norm, and squared weights then converge as well.
- External modes: With no persistent neutral external mode and limiting support of at most five nodes, the original tail inherits convergence from an exponentially close restricted-core orbit.Unstable modes are deleted before the shadowing reduction is applied.
B.8. Neutral external modes at a five-node limit
Persistent neutral external modes do not prevent convergence at a five-node limit. Geometry of the second-kind roots and one-sided forcing reduces the remaining pivot dynamics to finite variation.
- Root geometry: The remaining completion root lies outside the pivot interval, and neutral nodes cannot lie inside that interval because their multiplier sign conflicts with positivity there.The second-kind roots are real, simple, and interlace the orthogonal cubic.
- One-sided forcing: If persistent neutral modes lie on one side of the pivot interval, the forcing has one common sign and the pivot crosses the neutral boundary at most once.Stable terms are lower-order relative to persistent neutral forcing.
- Convergence with neutral modes: After the possible single crossing, the pivot has finite variation and therefore converges within the bounded interval.The stable contribution is summable, while boundedness supplies the variation in the opposite direction.
- Convergence with neutral modes: Every five-node tail with a persistent neutral mode has a singleton squared-weight ω-limit set.This includes nonspectral or confluent completion roots, factor collisions, pivot endpoints, and exact deletion into a four-node face.
- Convergence with neutral modes: The resulting convergence transfers to the core and all retained squared weights because the limiting denominators remain nonzero.This also covers asymptotic approach to a pivot endpoint with one zero limiting weight.
B.9. Neutral external modes at a six-node limit
At a six-node limit, neutral external modes constrain root motion through endpoint signs and a logarithmic cocycle. The analysis leaves only two boundary faces for separate treatment, which are then shown to converge.
- Root forcing: At six-node limits, persistent neutral modes force the completion roots to have definite endpoint signs.The endpoint signs are combined with a logarithmic cocycle to restrict root accumulation.
- Boundary reduction: Every six-node tail with a persistent neutral mode has convergent roots except possibly an IL-mode approaching λ5 or an IR-mode approaching λ2.These are the only remaining boundary faces after the sign analysis.
- Root convergence: If no persistent neutral node lies in IL or IR, the corresponding root has finite total variation and therefore converges.The proof uses forward-invariant half-spaces, summable stable forcing, and bounded root coordinates.
- Boundary reduction: If an IL-mode persists without an IR-mode, the roots converge unless σ approaches λ5; reversing node order gives the symmetric exception.The remaining inner-corner case is also forced to converge.
- Boundary completion: The two residual boundary cases have convergent roots and squared core weights, uniformly through endpoints, factor collisions, and exact deletion to a five-node face.This completes the boundary analysis required for six-node completion tails.
Appendix B.A1.5.5 proves the energy estimate
The energy estimate controls the six-node tail by comparing radial decay with smaller neutral and stable perturbations. This yields convergence of core coordinates and covers exact support losses through lower-support reductions.
- Conclusion: The interior and boundary analyses together settle every possible six-node limiting support, including exact support loss.This assembles the six-node completion result from the preceding cases.
- Six-node convergence: If no core coordinate is exactly deleted, roots, factor coordinates, and all six squared core weights converge, while every external weight tends to zero.The result remains valid through factor collisions and includes weights that vanish asymptotically.
- Support loss: Exact deletions are permanent and reduce the analysis to five-node or four-node faces; three or more simultaneous deletions contradict the lower bound on nonterminating support.The four-node parity map is the identity, and external squared masses are summable.
B.10. Signed recovery, the opposite parity, and the original problem
The signed-recovery argument converts convergence of squared spectral weights into convergence of both parity directions. A fixed spectral isometry then transfers these limits to the original restarted-CG residuals, including repeated-eigenvalue and support-loss cases.
- Signed recovery: Convergent squared weights determine a signed even-parity limit, and the opposite monic parity also converges.Positive limiting coordinates acquire fixed signs, while zero coordinates vanish.
- Scope of recovery: Repeated eigenspaces contribute a single fixed line, initially zero spectral components remain inactive, and later zero coordinates stay zero.These properties ensure the recovery remains valid under repeated eigenvalues and permanent support loss.
- Original problem: A fixed isometry makes convergence in active spectral coordinates equivalent to Euclidean convergence of the original signed residual vectors.This isometry preserves the relevant residual representation.
- Restart length three: For restart length three, the finite-support analysis yields convergence of even squared weights in every four-, five-, or six-node case.Signed recovery then gives both parity limits and completes the theorem for the original SPD problem.
B.A1.5. Analysis of the remaining boundary faces.
The remaining boundary analysis uses sign information and analytic coordinate changes to reduce exceptional root behavior to controlled faces. These reductions support the broader counterexample classification beginning at restart length four.
- Boundary coordinates: The boundary coordinates satisfy y^2 = wα and α − σ = u_y y^2 with u_y > 0 analytic.This signed-amplitude chart makes support-boundary behavior analytically tractable.
- Analytic extension: Analytic coordinate changes extend the five-node description through the support boundary and factor collisions.The relevant determinant remains a nonzero unit, preserving the local coordinate structure.
- First-order expansion: The first-jet analysis shows that the unforced return is tangent to the fixed components, with external effects entering through controlled higher-order terms.Squared weights are even in the signed omitted amplitude, and the displayed first derivatives vanish on the fixed components.
- Counterexample extension: For every s ≥ 4, degree elevation produces a nonterminating SPD restarted-CG problem in dimension s + 4 whose even normalized residuals do not converge.The restart-length-four construction supplies the base counterexample.
C.3. The certified Hopf point at restart length four
At restart length four, a certified six-node core with two external neutral modes yields a transverse Hopf point in the leading rescaled squared-weight dynamics. Rational interval and Sturm certificates establish the critical branch and spectral conditions needed for the later periodic-orbit construction.
- Certified configuration: The seed combines a six-node two-cycle with two external nodes whose squared two-block multipliers equal one.The parameter q moves the leading field through a transverse Hopf crossing.
- Certification: The finite certification uses rational isolating intervals, root-separation bounds, and Sturm counts for the selected polynomial roots.These certificates ensure the relevant roots remain real, simple, separated, and continuously labelled across the parameter interval.
- Hopf point: A unique critical parameter q* has a simple pair of eigenvalues ±iω* with 3.72 < ω* < 3.724, while all other eigenvalues avoid the imaginary axis.The Hopf eigenvector has a nonzero recovery-root component, and the crossing is transverse.
- Coordinate transfer: The scalar leading field and the weight-coordinate leading field are analytically conjugate near the critical branch.Thus the certified Hopf point transfers to the squared-weight coordinates used to construct the exact restarted-CG orbit.
- Consequence: An orbit shadowing the resulting periodic curve will provide a restarted-CG counterexample.The subsequent construction uses this periodic profile rather than convergence to a fixed two-cycle.
C.8. Accumulation along the periodic orbit
The periodic slow profile is sampled on a harmonic time mesh whose phase accumulation visits every point of the period. Consequently, the reconstructed squared weights have multiple accumulation points, and the resulting fixed SPD instance never terminates while its even residual directions fail to converge.
- Phase accumulation: The harmonic time mesh has increments tending to zero and accumulates at every phase modulo the periodic orbit's period.This converts shadowing of the periodic curve into repeated sampling of all phases.
- Nonconstant profile: The recovery-root coordinate is nonconstant, so two phases yield distinct accumulation values in the weight-coordinate sequence.The first Fourier coefficient is nonzero, providing the required phase separation.
- Weight accumulation: The reconstructed squared-weight sequence therefore has at least two distinct accumulation points.Figure 3 depicts the leading periodic modulation behind this nonconvergence.
- SPD reconstruction: The construction fixes an eight-dimensional diagonal SPD matrix from the selected spectral roots and associated positive weights.The eigenvalues are shifted roots satisfying 1 < λ_i < 10, so the matrix is positive definite.
- Exact counterexample: All eight weights remain positive, every restart block has grade eight, and no restart-length-four block terminates.Nonconvergence of the squared weights contradicts convergence of the even normalised residuals because coordinate squares would otherwise converge.
- Extension: The degree-elevation construction adjoins distant nodes and repeats the mechanism for larger restart lengths.At each step, the critical branch and transverse Hopf crossing are preserved analytically.
C.13. Persistence of the critical point and Hopf crossing
Degree elevation preserves the critical branch and transverse Hopf mechanism when a distant core node is added. Uniform convergence of the elevated coefficients and an implicit-function argument produce a nearby positive Hopf point at every sufficiently large elevated degree.
- Elevated data: Adding a distant core node yields coefficient and kernel data converging in C2 to the unelevated configuration.The normalized polynomial evaluations and external coupling matrices retain the required limiting structure.
- Persistence theorem: For every sufficiently large finite R, the elevated restart length r + 1 has a positive transverse Hopf point near the unelevated one.Invertibility at the original critical point enables the parameter-dependent implicit-function argument.
- Branch continuation: The elevated critical branch converges in C1 to the original branch, and the reduced Hopf matrices converge in C1 as well.These convergences include the parameter derivatives needed to continue the critical point and crossing.
- Hopf persistence: The Hopf crossing remains transverse, and all strict positivity margins persist under elevation.The elevated system therefore retains a nonzero recovery-root component in its Hopf eigenvector.
- Spectrum: Elevation adds one real eigenvalue +1 while preserving the nonreal pair ±iω_R and excluding other imaginary-axis eigenvalues.The universal θ = 2 mode produces λ = −1 and 2 in the elevated scalar field.
C.14. The two-block return map at arbitrary restart length
The two-block return map admits analytic local coordinates at arbitrary restart length, with a fixed-point family and one intrinsic normal direction. At restart length four and above, this structure supports periodic profiles whose exact restarted-CG shadows yield nontermination and nonconvergent even residuals.
- Local coordinates: At restart length r ≥4, positive coprime configurations have analytic local coordinates (a, C, H, U), with U the single intrinsic normal direction.The coordinates form an analytic chart on the core simplex through an invertible differential and the analytic inverse-function theorem.
- Return map: The two-block map fixes the family (a, 0, 0), has identity normal derivative there, and has explicitly controlled higher-order remainders.The weighted-order estimates give Ra = O_w(2), RU = O_w(3), Rb = O_w(3), and RL,h = O_w(2).
- Local coordinates: The coefficient-to-weight differential is injective and therefore an analytic local diffeomorphism between spectral coordinates and the mass-one core simplex.Injectivity follows from the orthogonality relations and dimension equality.
- Return map: The scalar Hopf point is also a Hopf point of the leading field of the exact rescaled map.The coefficients in the exact map coincide with those used in the scalar continuation.
- Periodic profile: For every sufficiently small nonzero amplitude, a nearby parameter, frequency, and nonconstant periodic solution exist independently of the first Lyapunov coefficient.The phase condition and implicit-function argument produce the periodic profile.
- Exact construction: Shadowing this periodic profile preserves positive weights and nonzero block factors, yielding r + 4 active nodes and preventing termination at restart length r.The resulting grade is r + 4 > r, so no restart block terminates.
- Exact construction: The reconstructed instance has nonconvergent even squared weights, which implies nonconvergence of the even normalised residuals.The exact SPD instance has positive initial spectral weights and never terminates.
- Classification: For every s ≥4, the construction produces a dimension-s + 4 SPD instance that is nonterminating and violates even-subsequence convergence.The proof combines the restart-four construction with induction and degree elevation.