Source-linked AI summary
The Complex Gradient Operator and the CR-Calculus
Ken Kreutz-Delgado
TL;DR
The paper addresses how to differentiate and optimize real-valued functions of complex-valued vectors when standard complex derivatives do not apply. It develops a CR-calculus framework with complex gradient and Hessian operators, including second-order analysis and applications to gradient-based optimization. The framework supports treatments of complex gradients, second-order methods, and adaptive filtering within an engineering-oriented calculus.
Problem
Real-valued objectives over complex parameters are not generally holomorphic, while complex-gradient definitions vary across authors and can be difficult to compare.
Method
The paper develops a CR-calculus framework that represents real gradients with complex operators and extends the treatment to multivariate second-order derivatives.
Results
The paper provides a calculus for complex gradients and Hessian matrices and applies gradient-based optimization to complex parameters, including LMS adaptive filtering.
Takeaways & Limitations
The framework supports engineering analysis of real-valued optimization problems involving complex-valued vectors without relying on nonexistent standard complex derivatives.
Takeaways & Limitations
Some derivations depend on the chosen coordinate representation, and results from special cases such as Ω_z=I must be checked before applying them generally.
Abstract
from arXiv · showhide
A thorough discussion and development of the calculus of real-valued functions of complex-valued vectors is given using the framework of the Wirtinger Calculus. The presented material is suitable for exposition in an introductory Electrical Engineering graduate level course on the use of complex gradients and complex Hessian matrices, and has been successfully used in teaching at UC San Diego. Going beyond the commonly encountered treatments of the first-order complex vector calculus, second-order considerations are examined in some detail filling a gap in the pedagogic literature.
1 Introduction
The paper motivates a complex-gradient framework for optimizing real-valued objectives over complex parameters, where standard complex differentiation is unavailable and terminology is inconsistent.
- Complex-valued vectors conveniently represent signals and system parameters in applications such as array processing, communications, and adaptive filtering.
- Standard complex derivatives apply only to analytic functions, whereas nonconstant real-valued functions of complex variables are generally nonanalytic.
- Viewing the objective over its real and imaginary components yields an equivalent real gradient for optimization.
- The paper introduces a redefined complex gradient operator to simplify derivations while extending standard differentiation to non-complex-analytic functions.
- There is no universally consistent complex-gradient definition, and row-versus-column conventions further complicate comparisons across authors.
- The CR-calculus framework unifies real- and complex-function perspectives and develops multivariate second-order calculus for real-valued cost optimization.
2 The Derivative of a Holomorphic Function
The section contrasts the restrictive standard complex derivative with real differentiability of complex-variable functions, establishing why real-valued objectives require an alternative calculus.
- The complex-variable notation z=x+jy can also describe a real differentiable mapping from R^2 to R^2 when standard complex differentiability is too restrictive.
- The standard complex derivative requires a limit independent of the direction through which the complex increment approaches zero.
- Holomorphicity, continuous complex differentiability, Cauchy-Riemann conditions, and convergent power-series expansions are presented as equivalent conditions.
- For f(z)=u(x,y)+jv(x,y), holomorphicity imposes harmonicity on u and v and a strong relationship through the Cauchy-Riemann equations.
- Examples such as z^n, e^z, ln(z), sin(z), and cos(z) are holomorphic, while conjugation, |z|^2, and nonconstant real-valued functions are not generally complex differentiable.
- For real-valued squared-error loss, conventional complex differentiation fails because the loss is nonholomorphic, motivating the alternative gradient treatment developed later.
3 Extensions of the Complex Derivative – The CR-Calculus
The CR-calculus extends differentiation to possibly nonholomorphic functions by combining real-variable gradients with complex conjugate coordinates. It provides R-derivatives and complex gradient tools for stationary-point analysis, while distinguishing competing derivative conventions and preserving agreement with the standard C-derivative in the holomorphic case.
- Motivation: Real-valued nonconstant functions generally lack a conventional complex derivative because they are not holomorphic, but they can have gradients as mappings from R2 to R.This motivates using the underlying real and imaginary components during optimization.
- R-Derivative: The R-derivative extends the standard partial derivative to nonholomorphic functions while incorporating real-gradient information within the complex-variable framework.The conjugate-coordinate representation writes functions in terms of z and its complex conjugate.
- Derivative conventions: A commonly used generalized complex partial derivative is rejected because for f(a)=a it yields 0 instead of the standard derivative 1, although it can locate stationary points.The paper notes that this definition is a scaled version of the conjugate R-derivative.
- Holomorphic case: An R-differentiable function is holomorphic in z if and only if it is independent of z̄, equivalently its conjugate partial derivative is zero.In this case, the R-derivative coincides with the standard C-derivative.
- First-Order Optimality Conditions: For real-valued functions of complex variables, vanishing of the R-derivative is necessary and sufficient for a stationary point with respect to the real parameters.In adaptive estimation, setting the relevant derivatives to zero yields the Wiener-Hopf equations for the optimal MMSE estimate.
- Multivariate CR-Calculus: The framework distinguishes the row-vector complex cogradient from the vector complex gradient, which gives the direction of steepest ascent.The paper develops this multivariate framework to clarify existing approaches and extend them toward second-order CR-calculus.
4 Multivariate CR-Calculus
The multivariate CR-calculus develops a framework for functions on C^n using conjugate coordinates, real-vector relationships, and Jacobian-based differentiation. It supports stationary-point analysis and provides machinery for gradient- and Hessian-based optimization.
- Framework: The paper expands the multivariate CR-calculus presentation of Brandwood and van den Bos for functions on C^n.The discussion incorporates insights from multivariate complex analysis, differential geometry, and real-vector calculus.
- Framework: C^n has a natural isomorphism with R^2n, allowing complex vectors to be related to real and imaginary coordinate representations.Each component is written as z_i = x_i + j y_i, with x and y collecting the real and imaginary parts.
- Framework: The framework assumes C^n is a Riemannian manifold with a Hermitian positive-definite metric tensor, making each tangent space a Hilbert space.This metric determines the inner-product structure used for multivariate gradients.
- Differentiation: The cogradient and conjugate cogradient operators provide formal partial derivatives with respect to z and z̄, enabling Jacobian and conjugate-Jacobian constructions.Their components are treated as R-derivatives, while z and z̄ are formally handled as conjugate coordinates.
- Caveats: The paper warns that z and z̄ cannot truly be varied independently, and complex differentiation with respect to the combined vector c is not well-defined.It instead identifies c with an n-dimensional real vector space for taking a real cogradient.
- Optimization: For real-valued functions, equivalent first-order stationarity conditions can be expressed using derivatives with respect to z or z̄.The equivalence follows from the differential identities for real-valued functions.
- Optimization: Constructing gradients and Hessians is necessary for general nonlinear iterative optimization and for checking second-order optimality conditions.The remainder of the note develops the machinery for these scalar-valued-function derivatives.
5 The Gradient Operator ∇z
The gradient operator ∇_z represents the real gradient of a real-valued function of complex variables without reformulating the problem entirely in real coordinates. Its metric dependence determines steepest-ascent and steepest-descent directions.
- Motivation: A real-valued function of a complex variable can have a real gradient even when its standard complex derivative does not exist.Viewing the function as a mapping from R^2 to R supplies the relevant gradient.
- Definition: The complex gradient is introduced as a redefined operator representing the equivalent real gradient within the complex-variable framework.This avoids the awkwardness of directly reformulating optimization in the real domain.
- Steepest Directions: For unit directions v, the directional differential is bounded by 2∥∇_z f∥ through the Cauchy-Schwarz inequality.The bound follows from |df_c(v)| = 2|Re{⟨∇_z f,v⟩}| ≤ 2∥∇_z f∥∥v∥.
- Steepest Directions: The gradient gives the directions of steepest increase, with +∇_z f for ascent and −∇_z f for descent.This result is stated generally for the metric framework and specializes to the Euclidean case when Ω_z = I.
- Stationarity: The stationary-point condition ∇_z f = 0 can be replaced by the simpler condition ∂f/∂z = 0, which does not require the metric tensor.The distinction disappears when Ω_z = I.
- Practical Use: Real-vector differentiation identities carry over to CR-calculus when the variables treated as constant are tracked explicitly.This approach avoids memorizing additional complex derivative identities and is used to derive the complex LMS algorithm.
- Metric Dependence: True gradient descent requires the metric tensor, whose application-specific determination can be highly nontrivial and can make algorithms more complex.Using Ω_z = I instead corresponds to the naive gradient rather than the metric-dependent natural gradient.
6 2nd-Order Expansions of a Real-Valued Function on Cn
The paper develops equivalent real and complex second-order representations for real-valued functions on C^n, enabling Hessian-based optimization while clarifying admissibility and algorithmic trade-offs.
- Admissibility: Admissibility conditions characterize which vectors and operators represent valid mappings between the constrained complex coordinate spaces.For M ∈ C^2n×2n, admissibility is equivalent to M = S M̄ S, or equivalently M̄ = S M S.
- Hessian properties: The c-complex Hessian is computationally convenient because it is equivalent to the c-real Hessian for analyzing numerical behavior and local or global minima.The two Hessian representations have corresponding definiteness properties, while the real Hessian eigenvalues are twice those of the complex Hessian and share its condition number.
- Equivalent second-order representations: The c-real and c-complex perspectives yield equivalent second-order terms in the power-series expansion of f.This equivalence follows by exploiting the relationships between the coordinate representations.
- Quasi-Newton simplification: Ignoring the Newton Hessian’s block off-diagonal elements defines an alternative quasi-Newton algorithm rather than necessarily approximating Newton’s method.The simplification can improve numerical robustness and reduce computational burden, but its justification depends on the problem setting.
- Newton and Gauss-Newton methods: The Newton Hessian can be indefinite or negative definite, unlike the Gauss-Newton Hessian, which is positive semidefinite and often positive definite under one-to-one mappings.The additional second-order term in the Newton Hessian is responsible for this difference in definiteness.
- Optimization trade-offs: Gradient descent provides stable updates for sufficiently small step sizes, whereas Newton’s faster convergence requires constructing and inverting its Hessian.The paper also states that Gauss-Newton is simpler and, under suitable assumptions, asymptotically equivalent to Newton.
7 Applications
The applications show how CR-calculus handles nonholomorphic complex least-squares problems, deriving exact solutions, Newton and Gauss-Newton updates, and the complex LMS algorithm.
- Nonlinear least squares: Nonlinear least-squares in z can be linear in c = (z, ¯z)^T, allowing an exact second-order expansion despite nonholomorphicity.The corresponding c-complex Hessian can retain nonzero off-diagonal entries.
- Nonlinear least squares: If |α|^2 ≠ |β|^2, the stationary solution is a global minimum; equality corresponds to loss of model identifiability.The global-minimum condition follows from positive definiteness of the Hessian.
- Newton and Gauss-Newton methods: For the simple problem linear in c, Newton and Gauss-Newton Hessians coincide because the second partial derivatives of g(c) vanish.This makes the Newton and Gauss-Newton algorithms identical in that example.
- Newton and Gauss-Newton methods: The full Newton algorithm reaches the optimum in one update, whereas the less complex pseudo-Newton algorithm converges more slowly.The paper presents this as a complexity-versus-convergence-speed trade-off.
- The Complex LMS Algorithm: The complex LMS application assumes a Euclidean parameter space with Ω_a = I and derives updates for estimating a complex vector from stochastic observations.The loss is generalized to a complex vector parameter a ∈ C^n.
- The Complex LMS Algorithm: The complex LMS derivation uses the gradient direction of steepest descent and yields the standard Wiener-Hopf equations at stationarity.An instantaneous stochastic-gradient approximation produces the online LMS update.