PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual Effect and Certainty Equivalence Revisited for a Linear-Gaussian Dual Control Problem

Ying Wang, Yue Ju, Håkan Hjalmarsson

arXiv:2610.30034v1math.OCcs.SYeess.SY

TL;DR

Dual control inputs affect both immediate cost and the information available for future decisions, complicating the question of when certainty equivalence is optimal. The paper analyzes a scalar linear-Gaussian model and characterizes Bellman-side no-dual-effect. It shows this condition can hold even when classical posterior-side NDE fails, and characterizes the resulting functions and cost-to-go decomposition.

  • Problem

    Beyond LQG, CE can be suboptimal because it ignores the value of information, while classical posterior-side NDE does not hold in the setting considered.

    Source: A. Posterior-side No-Dual-Effect · pp. 4–5

  • Method

    The paper studies a generic cost-to-go at a single Bellman recursion in a scalar linear-Gaussian model and characterizes Bellman-side NDE analytically.

    Source: §I · p. 2

  • Results

    Bellman-side NDE is characterized by a heat equation and Gaussian-convolution representation, and cost-to-go decomposes uniquely into NDE and dual-effect components.

    Source: §I · p. 2, §VIII · p. 11

  • Takeaways & Limitations

    CE can remain optimal when the expected cost-to-go is insensitive to the current input, even if posterior-side NDE fails.

    Source: A. Posterior-side No-Dual-Effect · pp. 4–5, §VIII · p. 11

    The analysis uses a static scalar model that strips away system dynamics and multidimensional exploration, and does not solve the full Bellman recursion.

    Source: §I · p. 2

Abstract

from arXiv · show

This paper revisits the relation between dual effect and certainty equivalence (CE) through a scalar linear-Gaussian model. In dual control, the current input affects not only the immediate cost but also the information available for future decisions, and hence the future cost. Dynamic programming (DP) provides a natural framework for formulating dual control problems. At each step of the Bellman recursion, an optimal input minimizes the sum of the stage cost and the expected cost-to-go. We restrict attention to stage costs minimized by the CE input. However, CE may not minimize the Bellman objective because of the input-dependent expected cost-to-go. Classical no-dualeffect (NDE) conditions guarantee CE optimality in certain problem classes by requiring the conditional distribution of the estimation error to be independent of past controls. In the present model, we observe that CE can remain optimal at the level of a single Bellman recursion even when the classical NDE conditions fail, provided that the expected cost-to-go is independent of the current input. Motivated by this observation, we introduce a notion of NDE formulated directly at the level of the Bellman objective, which we call Bellman-side NDE. Under some mild regularity conditions, we show that Bellman-side NDE can be characterized by a heat equation. This characterization identifies the class of Bellman-side NDE functions, and further yields a decomposition of a generic cost-to-go function into a Bellman-side NDE component and a dual-effect component.

I. INTRODUCTION

The paper studies when certainty equivalence (CE) remains optimal in dual control, where an input can affect both immediate cost and future information. It analyzes a scalar linear-Gaussian model and develops Bellman-side conditions for CE optimality.

I. INTRODUCTION

  • Beyond LQG, CE is generally suboptimal because it ignores the value of information, and a transparent characterization of when it is optimal is lacking.Classical posterior-side no-dual-effect conditions connect CE optimality to estimation-error distributions, while the paper examines input dependence directly in the Bellman objective.Source: §I · p. 1

II. A SCALAR LINEAR-GAUSSIAN DUAL CONTROL PROBLEM

  • The analysis uses a static scalar linear-Gaussian model with Gaussian prior, and studies CE optimality for a generic cost-to-go at one Bellman recursion rather than solving the full recursion.The model removes system dynamics and multidimensional exploration while retaining the dual effect.Source: §I · p. 2

I. INTRODUCTION

  • Bellman-side NDE requires the expected cost-to-go to be insensitive to the current input; the paper characterizes it through a heat equation, Gaussian convolutions, and a cost-to-go decomposition.The decomposition separates a Bellman-side NDE component from a dual-effect component, which carries the exploration incentive.Source: §I · p. 2

A. Information-Based Dual Control Formulation

  • Because the information history grows over time, optimizing directly over its decision rules is intractable; the formulation instead recasts the problem as dynamic programming with a two-dimensional state.The known-parameter problem begins with a loss-minimizing oracle input, while the unknown parameter is learned from input-output measurements.Source: A. Information-Based Dual Control… · pp. 2–3

B. Belief-State Dynamics

  • The Gaussian posterior is summarized by the belief state (m_t, P_t), whose update separates input-induced variance reduction from randomness in a standardized innovation.The posterior remains Gaussian, and the belief state captures its mean and variance.Source: B. Belief-State Dynamics · p. 3

C. Belief-State DP Formulation

The section formulates the problem as a belief-state dynamic program and restricts attention to stage costs already minimized by CE. It then specifies how the Bellman objective combines that stage cost with expected future cost.

C. Belief-State DP Formulation

  • The posterior distribution of the unknown parameter is summarized by the belief state, allowing the partially observed problem to be represented as a fully observed belief-state dynamic program.The posterior mean and variance parameterize the Gaussian belief, and the DP uses the belief-state transition.Source: C. Belief-State DP Formulation · p. 3
  • Each Bellman step minimizes a stage cost plus expected cost-to-go, with a zero terminal cost making the belief-state problem coincide with the information-based formulation.The recursion uses the standardized innovation in expectation.Source: C. Belief-State DP Formulation · p. 3

D. CE-Optimal Stage Cost

  • The analysis restricts the stage costs to those for which CE already minimizes posterior risk, so CE is the one-step greedy policy.Under the stated assumption, the stage cost has a unique minimum at the CE input.Source: D. CE-Optimal Stage Cost · p. 4
  • For the quadratic benchmark with an affine oracle input, the posterior-risk stage cost is the squared deviation from the CE input plus k^2P.The CE input is km + b.Source: D. CE-Optimal Stage Cost · p. 4
  • For the quartic benchmark, the posterior-risk stage cost includes quartic and variance-weighted quadratic deviation terms, plus a variance-squared term.The benchmark uses the same affine oracle input as the quadratic example.Source: D. CE-Optimal Stage Cost · p. 4
  • Even with the two-dimensional belief state, the backward Bellman recursion generally lacks closed-form cost-to-go functions, motivating analysis of a single recursion.The paper therefore examines CE optimality for a generic cost-to-go rather than verifying it at every stage.Source: D. CE-Optimal Stage Cost · p. 4

E. Research Problem

The paper asks when certainty equivalence remains optimal for a single Bellman recursion and develops a Bellman-side no-dual-effect condition to address that question. It contrasts this condition with posterior-side NDE and relates it to the input’s effect on expected cost-to-go.

E. Research Problem

  • The analysis tests whether a generic optimal or approximate cost-to-go changes the CE minimizer of a single Bellman objective.The stage cost is assumed minimized by the CE input; the expected cost-to-go can also depend on the input through the belief-state transition.Source: E. Research Problem · p. 4

III. BELLMAN-SIDE NO-DUAL-EFFECT

  • Posterior-side NDE is a standard sufficient condition for CE optimality,, but it fails in this model, motivating a Bellman-objective perspective.The paper asks instead whether the expected cost-to-go shifts the Bellman minimizer away from CE.Source: A. Posterior-side No-Dual-Effect · pp. 4–5, §III · p. 4

A. Posterior-side No-Dual-Effect

  • Here, posterior-side NDE requires the posterior variance to evolve independently of the input, but the variance update depends on the input.The parameter’s estimation error has conditional distribution N(0, P), so input dependence of P violates the condition.Source: A. Posterior-side No-Dual-Effect · p. 5

B. Dual-Effect Function

  • The dual-effect function measures how a variance reduction changes expected cost-to-go: negative, zero, and positive changes indicate beneficial, neutral, and unfavorable information, respectively.The variance reduction is treated as information gain.Source: B. Dual-Effect Function · p. 5

C. Bellman-side DE and NDE

  • Bellman-side NDE holds at a belief state when the dual-effect function is zero for every variance reduction; it may hold only on part of the state space.In the threshold example, NDE holds when P ≤ γ, whereas dual effect occurs when P > γ.Source: C. Bellman-side DE and NDE · p. 5

D. Sufficient Condition for CE Optimality

Bellman-side NDE is a sufficient condition for the CE input to uniquely minimize the Bellman objective. The paper then describes a source-based characterization under regularity assumptions, with a stated boundary on its applicability.

D. Sufficient Condition for CE Optimality

  • When Bellman-side NDE holds, the CE input uniquely minimizes the Bellman objective at that state, and throughout the belief-state space if NDE holds everywhere.The condition is sufficient, not necessary: CE may still minimize the objective when the expected cost-to-go varies with input but is minimized at CE.Source: D. Sufficient Condition for CE… · pp. 5–6

IV. CHARACTERIZATION OF BELLMAN-SIDE NDE

  • The characterization section uses regularity conditions to represent the dual-effect function through a source and characterize value of information and CE optimality.The section also relates Bellman-side NDE to the source vanishing identically.Source: §IV · p. 6

A. Regularity Conditions

  • The characterization assumes differentiability and locally uniform polynomial growth; it does not apply to the threshold cost-to-go that fails the regularity condition at P = γ.That example still satisfies Bellman-side NDE on its subregion, but falls outside this section’s results.Source: A. Regularity Conditions · p. 6

B. Source Representation of Dual-Effect Function

The source function turns variance reduction into a local differential quantity for analyzing the expected cost-to-go. Its sign characterizes whether information has positive, neutral, or negative value locally and, when constant over the state space, globally.

B. Source Representation of Dual-Effect Function

  • The source S_J represents the sensitivity of expected cost-to-go to variance reduction, replacing direct analysis of the change for every reduction size.This representation shifts the analysis to a local differential quantity.Source: B. Source Representation of… · pp. 6–7
  • The derivative of expected cost-to-go with respect to variance reduction η equals the negative expected source along the corresponding Gaussian belief update.The identity holds under Assumption 3 for η between zero and the current variance P.Source: B. Source Representation of… · p. 7

C. Source Characterization of Value of Information

  • The sign of S_J(m, P) determines the first-order effect of a small variance reduction, while its magnitude gives the change per unit reduction.A zero source is neutral to first order; positive and negative sources imply locally negative and positive changes in expected cost-to-go, respectively.Source: C. Source Characterization of Value of… · p. 7
  • If S_J has a constant sign over X, the corresponding positive, neutral, or negative value-of-information conclusion holds for every variance reduction.For the quadratic cost-to-go J(m, P) = m^2 + λP, λ > 1 gives positive VOI, λ = 1 information neutrality, and λ < 1 negative VOI.Source: C. Source Characterization of Value of… · p. 7

D. Source Characterization of Bellman-side NDE and CE Optimality

Under regularity conditions, Bellman-side NDE is characterized by a zero source, which is sufficient for CE optimality. The paper represents these functions through Gaussian convolution of a complete-information boundary function and describes the resulting function space.

D. Source Characterization of Bellman-side NDE and CE Optimality

  • J is Bellman-side NDE on X if and only if its source vanishes there; under Assumption 1, this condition makes the CE input uniquely minimize the Bellman objective at every belief state.The characterization and optimality result assume the stated regularity conditions.Source: D. Source Characterization of… · p. 8

A. Gaussian convolution representation of Bellman-side NDE functions

  • The boundary function is the limit of J as P decreases to zero and represents cost-to-go under complete information about the parameter.The posterior uncertainty vanishes in this limit.Source: A. Gaussian convolution representation… · p. 8
  • The complete-information boundary functions of J in H are continuous and of polynomial growth, forming the class B used in the convolution representation.The boundary trace maps H into B.Source: A. Gaussian convolution representation… · p. 8
  • Every Bellman-side NDE function in H is the Gaussian convolution of its complete-information boundary function, and each such boundary function in B generates a Bellman-side NDE function.The representation is unique for each function in the Bellman-side NDE space.Source: A. Gaussian convolution representation… · pp. 8–9

B. Polynomial Bellman-side NDE Function Space

The polynomial Bellman-side NDE functions form a finite-dimensional hierarchy generated by heat polynomials. A boundary polynomial of degree at most n corresponds exactly to membership in the space generated through degree n, while the full NDE class also contains non-polynomial functions.

B. Polynomial Bellman-side NDE Function Space

  • The heat-polynomial basis begins h_0 = 1, h_1 = m, h_2 = m^2 + P, h_3 = m^3 + 3mP, and h_4 = m^4 + 6m^2P + 3P^2; each has zero source.These functions are Gaussian convolutions of monomial boundary functions.Source: B. Polynomial Bellman-side NDE… · p. 9
  • The polynomial NDE spaces form a nested hierarchy, but Bellman-side NDE functions need not be polynomials of finite degree.The paper gives a non-polynomial Bellman-side NDE example.Source: B. Polynomial Bellman-side NDE… · p. 9
  • A Gaussian-convolved cosine provides a Bellman-side NDE example that is not a finite-degree polynomial in (m, P).Its source vanishes even though the function is outside the polynomial subspace.Source: B. Polynomial Bellman-side NDE… · p. 9
  • A Bellman-side NDE function has a polynomial complete-information boundary of degree at most n if and only if it lies in H_n, the span of the heat-polynomial basis through degree n.The space has dimension n + 1.Source: B. Polynomial Bellman-side NDE… · p. 9

VI. BELLMAN-SIDE FUNCTION DECOMPOSITION

A generic cost-to-go function is uniquely decomposed into a Bellman-side NDE component and a residual dual-effect component. The residual has zero boundary trace and carries the full source and variance-reduction effect, so it alone determines whether the expected cost-to-go shifts CE minimization.

VI. BELLMAN-SIDE FUNCTION DECOMPOSITION

  • The unique decomposition assigns the Gaussian convolution of the complete-information boundary function to J_NDE and the residual to J_DE.The NDE component has the same boundary trace as J.Source: §VI · pp. 9–10
  • J_DE has zero boundary trace, carries the full source of J, and accounts for the entire variance-reduction effect.The expected-cost change satisfies ΔJ(x, η) = ΔJ_DE(x, η).Source: §VI · p. 10
  • The NDE component contributes an input-independent term to the Bellman objective, so CE optimality for a fixed stage cost is decided by J_DE alone.When J_DE vanishes, Bellman-side NDE holds and, under Assumption 1, CE uniquely minimizes the Bellman objective.Source: §VI · p. 10
  • Bellman-side NDE is sufficient but not necessary for CE optimality; CE may remain optimal with nonzero J_DE if its contribution does not shift the Bellman-objective minimizer.Identifying broader sufficient conditions is outside the paper’s scope.Source: §VI · p. 10

VII. NUMERICAL EXAMPLES

The numerical examples show how the sign of a cost-to-go component changes whether the Bellman minimizer follows certainty equivalence or favors or discourages exploration. The conclusion characterizes Bellman-side NDE and identifies a decomposition of cost-to-go functions, while leaving extensions and policy-level conditions for future work.

A. Quadratic Cost Example

  • In the quadratic benchmark, λ = 1 gives CE, λ > 1 shifts the minimizer toward larger |u| for m ≠ 0, and λ < 1 shifts it toward smaller |u|.The figure compares minimizers for λ = 0.5, 1, and 2.Source: Figure 1 · p. 11, A. Quadratic Cost Example · p. 11

B. Quartic Cost Example

  • For the quartic benchmark, β = 0 recovers CE, β > 0 encourages exploration, and β < 0 discourages it.The examples associate these cases with positive, zero, and negative value of information, respectively.Source: Figure 2 · p. 11, B. Quartic Cost Example · p. 11

VIII. CONCLUSION

  • Bellman-side NDE is a sufficient condition for CE optimality and can hold when classical posterior-side NDE fails.The relevant Bellman condition is that expected cost-to-go is insensitive to the current input.Source: §VIII · p. 11
  • Bellman-side NDE holds if and only if the cost-to-go solves a heat equation; its functions are Gaussian convolutions of complete-information boundary functions.The paper uses this representation to characterize the class of Bellman-side NDE functions.Source: §VIII · p. 11
  • The representation yields a unique decomposition into no-dual-effect and dual-effect terms; studying the latter for less restrictive CE conditions is left for future work.The paper also leaves extending the characterization to multi-input settings and identifying policy-level CE conditions for concrete DP problems as future work.Source: §VIII · p. 12

APPENDIX I PROOF OF LEMMA 1

The first proof expresses the observation innovation as a conditionally standard normal variable and updates posterior variance through the input-dependent variance reduction. The next proof establishes regularity properties needed for its argument.

APPENDIX I PROOF OF LEMMA 1

  • The normalized innovation is conditionally standard normal given the belief state and input, and the next posterior variance is P − η(P, u).The proof handles u = 0 separately using the convention sgn(0) = 1.Source: Appendix I · p. 12

APPENDIX II PROOF OF LEMMA 4

  • The proof justifies differentiating the expectation defining F_x by interchanging differentiation and expectation on interior intervals.It uses the domination bound and an integrable bound independent of η.Source: APPENDIX II PROOF OF LEMMA 4 · p. 12
  • Under the stated assumptions, the function F_x is continuous and Lipschitz on [0, η̄], hence absolutely continuous there.The proof establishes continuity at the interval endpoints and extends the Lipschitz bound to the closed interval.Source: APPENDIX II PROOF OF LEMMA 4 · pp. 12–13

APPENDIX III PROOF OF LEMMA 5

The proof links the small-variance behavior of the expected cost change to the source function, then establishes regularity and growth of the complete-information boundary function.

APPENDIX III PROOF OF LEMMA 5

  • As η decreases to zero, the expected source approaches S_J(m, P), so the sign of S_J determines the sign of the cost change for sufficiently small positive η.If S_J is zero, the cost change is o(η); if positive, the change is negative, and if negative, it is positive.Source: APPENDIX III PROOF OF LEMMA 5 · p. 13

APPENDIX IV PROOF OF LEMMA 6

  • The boundary function ϕ_J is continuous and has polynomial growth under the stated assumptions on J.Continuity follows from locally uniform convergence of continuous functions as P decreases to zero.Source: APPENDIX IV PROOF OF LEMMA 6 · p. 13

APPENDIX V PROOF OF LEMMA 7

The proof shows that Gaussian convolution of a suitable boundary function yields a regular Bellman-side NDE function. It then establishes the converse characterization used for the theorem.

APPENDIX V PROOF OF LEMMA 7

  • For a continuous, polynomially growing boundary function, its Gaussian convolution is smooth and satisfies the regularity bounds required for membership in the Bellman-side NDE class.The proof also establishes that its source is identically zero.Source: Appendix V · pp. 13–14
  • The boundary function of a Gaussian convolution recovers the original function, completing the construction of Bellman-side NDE functions from boundary functions.The proof verifies the relevant boundary and regularity assumptions.Source: Appendix V · pp. 13–14

APPENDIX VI PROOF OF THEOREM 4

  • A cost-to-go is Bellman-side NDE if and only if it equals the Gaussian convolution of its complete-information boundary function.The proof uses the boundary limit for one direction and Lemma 7 for the converse.Source: APPENDIX VI PROOF OF THEOREM 4 · p. 14
Loading 2610.30034v1…