PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational and optimal control representations of conditioned and driven processes

Raphael Chetrite, Hugo Touchette

arXiv:1506.05291v3cond-mat.stat-mech

TL;DR

Rare deviations of time-integrated observables raise the question of how to describe the corresponding conditioned Markov-process trajectories. The paper characterizes the driven process through large-deviation variational principles and optimal stochastic control, and identifies it as their solution, as well as the conditioned process when equivalence holds.

  • Problem

    The paper addresses how to describe Markov-process trajectories conditioned on rare deviations of time-integrated observables.

    Source: §I · p. 3

  • Method

    It derives variational principles for large-deviation functions and relative entropies and connects them to stochastic control.

    Source: §V · pp. 20–21

  • Results

    The variational and control problems are solved by the driven process and, when equivalence holds, by the conditioned process.

    Source: §V · p. 21

  • Takeaways & Limitations

    In the long-time limit, rare-event trajectories can be described by a conditioning-free Markov process.

    Source: §I · p. 3

Abstract

from arXiv · show

We have shown recently that a Markov process conditioned on rare events involving time-integrated random variables can be described in the long-time limit by an effective Markov process, called the driven process, which is given mathematically by a generalization of Doob's $h$-transform. We show here that this driven process can be represented in two other ways: first, as a process satisfying various variational principles involving large deviation functions and relative entropies and, second, as an optimal stochastic control process minimizing a cost function also related to large deviation functions. These interpretations of the driven process generalize and unify many previous results on maximum entropy approaches to nonequilibrium systems, spectral characterizations of positive operators, and control approaches to large deviation theory. They also lead, as briefly discussed, to new methods for analytically or numerically approximating large deviation functions.

I. INTRODUCTION

The paper connects rare-event conditioning of Markov processes to variational principles and optimal stochastic control, with the driven process as the shared object. It extends these connections across noise regimes and to current-type costs, spectral characterizations, and maximum-entropy methods.

I. INTRODUCTION

  • The driven process admits variational and optimal-control characterizations tied to large-deviation functions and relative entropies.Source: §I · p. 3

A. Control approaches to large deviations

  • In the long-time Donsker–Varadhan setting, the optimal control process is the driven process, linking control's change of measure to generalized Doob transforms and conditioning.The connection also applies to low-noise Freidlin-Wentzell-Graham problems and systems not necessarily perturbed by weak noise.Source: A. Control approaches to large… · p. 4

B. Stochastic control with current-type costs

  • For current-type costs involving jumps or increments, the paper gives an alternative derivation of the HJB equation and relates quadratic costs to large deviations.Its simple proof in Appendix A.6 relies only on Itō calculus.Source: B. Stochastic control with… · p. 4

C. Spectral characterizations of positive operators

  • The Donsker–Varadhan characterization of dominant eigenvalues follows as a special case of level-2.5 variational principles whose optimizer is the driven process.Source: C. Spectral characterizations of… · p. 5

D. Variational methods for computing large deviations

  • The derived principles include earlier variational approaches to large-deviation functions, and their driven-process optimizer supports approximations and bounds for the scaled cumulant generating function.Source: D. Variational methods for computing… · p. 5

E. Path maximum entropy and maximum caliber

II. THEORY OF CONDITIONED AND DRIVEN PROCESSES

  • The theory section presents pure diffusions for simplicity, while jump processes, Markov chains, and hybrid processes are also covered or treatable.Source: §II · p. 6

A. Conditioned process

  • Conditioning on rare time-integrated observable values is generally non-Markovian at finite time, but converges to a homogeneous Markov driven process as T→∞.The observables can represent quantities such as work or heat exchanged with the environment.Source: A. Conditioned process · pp. 7–8

B. Driven process

The driven process is defined through a transform of the tilted generator and provides a long-time Markov description of a process conditioned on a rare time-integrated observable. Under stated spectral and convexity conditions, it is asymptotically equivalent to the conditioned process in typical values, though not necessarily in fluctuations.

B. Driven process

  • The tilted generator’s dominant eigenvalue is the SCGF when its spectrum has a gap, and its right and left eigenfunctions enter the driven-process construction.The operator combines the observable functions f and g with the diffusion matrix D.Source: B. Driven process · pp. 8–9
  • The driven process has generator L_Fk, obtained by a generalized Doob transform of the tilted generator using its dominant eigenfunctions.For diffusions, the transform changes the original drift to the driven drift.Source: B. Driven process · p. 9
  • A spectral gap, an LDP for A_T, and convexity of I(a) support equivalence when k = I′(a); nonconvex I(a) admits no equivalent driven process.The paper also assumes the large deviations do not arise as a time-boundary effect and assumes differentiability of I(a) for the stated relation.Source: B. Driven process · p. 9
  • The equivalence is established through the canonical path measure: its time-dependent Markov process converges to the homogeneous driven process, then canonical and conditioned measures become equivalent.The latter equivalence holds when I(a) is convex and k = I′(a).Source: B. Driven process · pp. 9–10
  • When I(a) is convex, conditioned and driven processes have the same long-time typical states and ergodic values, although their fluctuations can differ.In particular, their ergodic density and probability current agree; the driven process achieves the conditioned hard constraint in a soft way.Source: B. Driven process · p. 10

III. VARIATIONAL REPRESENTATIONS

The section characterizes the driven process through variational principles involving large-deviation functions and relative entropy. Its central construction links solutions of these principles to the driven process by representing observables through path statistics of Markov processes.

III. VARIATIONAL REPRESENTATIONS

  • The variational principles unify and generalize earlier results, and their solutions identify the driven process that realizes the conditioned process in the long-time limit.The principles use the SCGF, rate function, and relative entropy.Source: §III · p. 11

A. Main ideas

  • For suitable path observables B_T, the conditioning observable A_T is a function of B_T, whose rate function supports contraction and Laplace variational principles.B_T must satisfy an LDP, and A_T must admit a representation in terms of B_T.Source: A. Main ideas · p. 11
  • For convex rate functions, the contraction and Laplace principles have the same solutions, linking conditioned and driven processes.The stated equivalence uses k = I′(a).Source: A. Main ideas · p. 12
  • When these principles are applied to path statistics corresponding one-to-one with Markov path measures, their optimizers correspond to the driven process.The constrained contraction solution is the value of B_T on which its conditional distribution concentrates exponentially; the dual solution is typical under the canonical path measure.Source: A. Main ideas · pp. 11–12
  • For the Markov observables considered, the joint empirical density and current provide a level-2.5 LDP contraction and determine a Markov process uniquely.For Markov diffusions, the Radon–Nikodym derivative between homogeneous diffusions can be expressed as a function of density and current.Source: A. Main ideas · p. 12

B. SCGF

The SCGF is represented variationally using empirical density and current, with an optimizer tied to the driven process. The formulation also shows that a general process-and-observable problem can be expressed as an optimization over control drifts.

B. SCGF

  • The SCGF variational representation is maximized over normalized densities and sourceless currents, with the optimizer corresponding to the driven process.The optimizing drift is u* = Fk, the modified drift of the driven process.Source: B. SCGF · pp. 12–13
  • Rewriting the current as the stationary current of a free drift u turns the variational representation into a contraction over density and current.The sourceless-current condition makes the density invariant for that drift.Source: B. SCGF · pp. 12–13
  • The level-2.5 representation generalizes Fleming’s control result and prior one-dimensional diffusion results for particular observables.It also recovers the cited ring-model result for an overdamped Langevin equation with constant drive and periodic potential.Source: B. SCGF · p. 13
  • The general variational problem requires only maximization over the control drift u; the additional constraint used for the ring-model derivation is not fundamental.The drift u* makes the solution pair (ρ*, j*) typical, but is not itself interpretable as a most probable drift.Source: B. SCGF · pp. 13–14

C. Rate function

For convex rate functions, the drift minimizing the constraint-based representation is the driven drift, and the conditioned and driven processes share typical empirical values. Path-relative-entropy principles likewise characterize the driven process as the closest homogeneous Markov process satisfying the constraint.

C. Rate function

  • When I(a) is convex, the constrained minimizer is the driven drift Fk with k = I′(a), and the conditioned and driven processes have the same typical values of ρT and JT.The driven process maximizes the probability of (ρT, JT) subject to AT = a.Source: C. Rate function · p. 14

D. Relative entropy

  • The path-relative-entropy variational principles select the driven process as the optimizer for the SCGF and, under AT = a, for the rate function.For convex I(a), the constrained optimal drift is Fk with k = I′(a).Source: D. Relative entropy · p. 15
  • A simple relative-entropy limit yields the path-level large-deviation function and provides an alternative route to proving the paper’s variational and control results.The paper also gives proofs based on relative-entropy positivity and on the canonical path measure.Source: D. Relative entropy · p. 16
  • In the long-time limit, the driven process is the homogeneous Markov process closest in relative entropy to the conditioned process Xt|AT = a.The canonical path measure and driven process have the same ergodic states and asymptotic relative entropy relative to the original process.Source: D. Relative entropy · pp. 15–16

E. Spectral eigenfunctions

The paper gives variational representations for spectral eigenfunctions and frames the driven process as an optimal stochastic control. The control cost combines the cost of meeting the observable constraint with the relative-entropy cost of changing the dynamics.

E. Spectral eigenfunctions

  • One variational representation for the spectral elements has maximizer h∗ = rk, the right eigenfunction of the tilted operator.The authors identify this as a generalization of a result for jump processes.Source: E. Spectral eigenfunctions · p. 17
  • For g = 0, the two-function representations have stationary points l∗ = lk and r∗ = rk, rather than an established global extremum.The authors do not know whether these stationary-point representations extend to current-type observables with g ≠ 0.Source: E. Spectral eigenfunctions · p. 17

IV. OPTIMAL CONTROL REPRESENTATIONS

  • The control formulation makes the driven process an optimizer of functionals related to large deviations of AT, with a value function satisfying an HJB equation.The resulting optimal control drift converges to the driven drift in the ergodic limit.Source: §IV · p. 17

A. Controlled process

  • The controlled-process cost combines a Lagrange-multiplier cost for reaching AT = a with a relative-entropy cost for changing the drift, and is additive in time and quadratic in the control drift.The first cost is accumulated over [t, T], while the second compares the controlled process with the original process.Source: A. Controlled process · p. 18

B. Optimal controller

The optimal controller follows from an HJB equation, and its optimal drift converges in the long-time limit to the driven drift. The paper connects this control process to the conditioned process and contrasts its formulation with earlier control approaches.

B. Optimal controller

  • The optimal control law maximizes the HJB equation’s objective and depends explicitly on the initial and final times of the control horizon.For g = 0, it reduces to the standard stochastic HJB solution.Source: B. Optimal controller · p. 18
  • In the ergodic limit, the driven process is the optimal controller, with optimal drift u∗ = Fk and mean expected cost converging to Λk.The time-rescaled cost also converges in probability to the SCGF and dominant eigenvalue Λk.Source: B. Optimal controller · p. 19

C. Comparison with previous works

  • Unlike Fleming’s approach, the paper treats a g term for observables depending on displacements or increments as well as the state.For g = 0, the expected cost is equivalent to Fleming’s cost (3.10) in.Source: C. Comparison with previous works · p. 20
  • The optimal control process has a probabilistic interpretation as the driven process in the ergodic limit and, by equivalence, as the process conditioned on AT = a.The paper transforms the nonlinear HJB PDE into a linear Feynman–Kac PDE.Source: C. Comparison with previous works · p. 20
  • Bierkens, Chernyak, Chertkov, and Kappen derive an equation for the optimizer and a linear equation for the spectral elements, but do not identify the optimal control as the driven or conditioned process.Their approach generalizes the DV characterization of dominant eigenvalues of positive operators.Source: C. Comparison with previous works · p. 20

V. CONCLUSIONS

The paper characterizes the driven process through variational principles and optimal control, then discusses implications for nonequilibrium fluctuations and ways to approximate or compute large deviation functions.

V. CONCLUSIONS

  • The paper derives level-2.5-based variational representations for Λk and I(a), links them to control, and identifies their solution with the driven process and, when equivalence holds, the conditioned process.These representations generalize earlier work and connect variational and control characterizations.Source: §V · pp. 20–21
  • The proposed applications and numerical methods are brief, with full implementation and comparisons against methods such as cloning left as a substantial challenge for future work.The authors plan to develop the remarks, especially those on numerical estimation, in future publications.Source: §V · p. 21

A. Physics of nonequilibrium fluctuations

  • The driven process is the unique homogeneous Markov reweighting closest to the original in relative entropy that makes the fluctuation AT = a typical, giving a first-principles justification for maximum-entropy approaches.Unlike low-noise FWG conditioning, which selects a single fluctuation path, this conditioning generally identifies a whole process or set of paths.Source: A. Physics of nonequilibrium… · pp. 21–22
  • Open questions include whether boundary-driven nonequilibrium systems can be represented as conditioned equilibrium systems and when the driven process has the FWG adjoint-dynamics limit.The paper frames both questions in terms of its optimal-control problem and low-noise stochastic control.Source: A. Physics of nonequilibrium… · p. 22

B. Variational approximations

  • Restricting the optimization to a chosen control design can make approximations tractable, while sup principles give lower bounds and inf principles give upper bounds that improve as the control class expands.Approximation quality depends on the system and observable and is related to the relative entropy between true and approximate control solutions.Source: B. Variational approximations · pp. 22–23

C. Numerical algorithms for large deviation functions

  • The paper outlines three numerical routes: variational approximations with control designs, dynamic programming for the HJB equation, and importance sampling with the driven process.The driven process makes the conditioned event typical, avoiding the exponentially large-in-T sample requirement of simulating the original process for the rare event.Source: C. Numerical algorithms for large… · p. 23

Appendix A: Proofs

The appendix verifies the variational optimizer, recovers the Donsker–Varadhan case when g = 0, and relates the variational and relative-entropy representations to the driven process.

1. Optimal ρ∗and j∗

  • The optimizer of the variational principle yields the Perron–Frobenius eigenvector rk and dominant eigenvalue Λk, with the resulting density relation ρinv Fk = rklk.The derivation introduces multipliers for the constraints on the empirical density and current.Source: §1 · p. 24

2. Donsker-Varadhan result for g = 0

  • For g = 0, direct minimization over j reduces representation (40) to the Donsker–Varadhan principle for the empirical-density level-2 rate function; in the stated reversible case it reduces to Rayleigh–Ritz.The reversible case is specified with respect to the invariant density.Source: §2 · p. 25

3. Relative entropy representations via level-2.5 large deviations

  • The variational and relative-entropy representations of Λk and I(a) are equivalent, and their optimizer gives the typical empirical density and current in the driven process.In the drift representation, the optimizing drift is u∗ = Fk.Source: §3 · p. 26

4. Eigenfunction representation (59)

The appendix derives an eigenfunction representation, discusses the limits of related stationary-point formulations, and connects current costs with an HJB equation.

4. Eigenfunction representation (59)

  • Representation (59) follows by contracting the drift-based variational principle, and its transform defines a diffusion with a drift and stationary density.The transform is identified as the generalized Doob h-transform of Lk.Source: §4 · p. 26

5. Eigenfunction representations (60) and (61)

  • For representations (60) and (61), it is unclear whether the change of variables preserves the minimizing property, so the resulting optimization is stated in terms of stationary points.The solution is obtained using the known optimizer ρ∗ = rklk.Source: §5 · p. 27

6. Modified HJB equation for current costs

  • The current-cost control formulation yields a modified HJB equation; the derivation recovers the earlier quadratic-cost results of.The added current or Stratonovich cost had previously received a partial solution for quadratic costs in the control drift.Source: §6 · pp. 27–28

7. Control representation

The appendix relates the PDE for the exponential cost to the tilted generator in the control representation.

7. Control representation

  • The PDE for the exponential cost GT is rewritten using the tilted generator Lk, as claimed in (69).The derivation uses the solution for u∗ in the HJB equation.Source: §7 · p. 28

Appendix B: Jump processes

For pure jump processes, the paper translates its driven-process construction into jump rates and represents the relevant large deviations through empirical densities and flows. The driven process is characterized by variational and control formulations, with corresponding equivalence results for conditioned processes.

Appendix B: Jump processes

  • A generalized Doob transform of the tilted jump generator defines the driven process and its transition rates.The tilted generator is characterized through the scaled cumulant generating function and its eigenfunctions.Source: Appendix B · p. 29
  • The driven and conditioned jump processes have the same limiting empirical density and asymptotic empirical current.The density converges to the invariant density of the driven process.Source: Appendix B · p. 29
  • The jump-process variational representation uses empirical density and flow, with minimizers given by the invariant density and driven rates.Reparameterizing the flow by transition rates yields the driven transition-rate matrix as the minimizer.Source: Appendix B · pp. 30–31
  • Choosing transition rates as controls gives a jump-process control representation analogous to the paper’s diffusion control result.The empirical version of the rate function appears in this representation; a mean-form result was previously obtained by Jack and Sollich.Source: Appendix B · p. 31
  • The paper does not restate the jump-process relative-entropy variational representations, which follow similarly to the diffusion case.These representations place results derived by Monthus in a large-deviation framework.Source: Appendix B · p. 31

Appendix C: Markov chains

The appendix translates the driven-process results to discrete-time Markov chains using pair empirical densities. It gives chain analogues of the variational and control representations and states asymptotic equivalence with conditioned chains.

Appendix C: Markov chains

  • The discrete-time driven chain is defined through modified transition probabilities associated with the tilted matrix and its eigenvectors.The scaled cumulant generating function is the logarithm of the tilted matrix’s dominant eigenvalue.Source: Appendix C · p. 32
  • The pair empirical density suffices to represent the accumulated observable, and its driven-chain limit is the stationary joint distribution.The empirical pair density is the chain analogue of the jump-process empirical flow.Source: Appendix C · pp. 32–33
  • When the observable’s rate function is convex, the conditioned chain and driven chain have the same stated asymptotic limits.The conditioning parameter is k = I′(a).Source: Appendix C · p. 33
  • The text presents a transition-matrix variational formula derived from the Laplace principle, a control representation, and an empirical relative-entropy representation associated with a controlled Markov chain.Source: Appendix C · p. 33

Related papers

Loading 1506.05291v3…