PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous Quantum Feedback Control via Kraus-Parameterized Belief Reinforcement Learning

Priyanshi Singh, Krishna Bhatia

arXiv:2608.15715v1quant-phcs.LG

TL;DR

Quantum feedback control must infer the underlying quantum state from noisy continuous measurement records, but generic recurrent memories do not guarantee physically valid beliefs. This paper combines a Kraus-constrained recurrent belief encoder with PPO control, achieving modestly higher fidelity and substantially lower return variance than an LSTM-history baseline across nominal and out-of-distribution conditions.

  • Problem

    Quantum feedback controllers must infer posterior quantum-state beliefs from noisy, partial measurement histories, while generic recurrent representations lack physical-consistency guarantees.

  • Method

    A Kraus-constrained recurrent encoder produces positive-semidefinite, trace-normalized density-matrix beliefs that supply physically consistent features to a PPO controller.

  • Results

    Across three evaluation conditions, Kraus-belief PPO achieves modestly higher mean return and fidelity with substantially lower return variance than the LSTM-history baseline.

  • Takeaways & Limitations

    The pipeline provides physically valid, interpretable belief features and more stable feedback under the evaluated measurement inefficiency and dynamics shifts.

Abstract

from arXiv · show

Quantum feedback control requires acting on noisy continuous measurement records without direct access to the underlying quantum state. We propose Kraus-Parameterized Belief Reinforcement Learning, a pipeline in which a recurrent encoder, constrained to the Stiefel manifold, produces density-matrix estimates that are guaranteed positive-semidefinite and trace-normalized by construction, embedding quantum state geometry directly into the learning loop. A Proximal Policy Optimization (PPO) actor then maps these physically valid belief states to continuous control actions. On a simulated continuously monitored qubit, the resulting policy achieves stable feedback control, maintaining a measurement-conditioned belief fidelity of approximately 0.77-0.80 and exhibiting substantially lower return variance than a parameter-matched LSTM-history baseline across both nominal and out-of-distribution conditions. Although gains in raw target fidelity are modest, the geometric constraint guarantees a physically valid, interpretable belief representation and yields markedly more stable control under measurement inefficiency and abrupt dynamics switches. These results indicate that physics-informed neural memory is a practical inductive bias for reliable quantum feedback control.

I. INTRODUCTION

Quantum feedback control must infer a physically meaningful quantum-state belief from noisy, partial measurement records. This paper proposes a Kraus-constrained recurrent encoder paired with PPO and compares it against an LSTM-history PPO baseline across nominal and shifted conditions.

  • Motivation: Continuous qubit feedback acts on inefficient, noisy measurement records rather than the true density matrix, making posterior-belief approximation central to control.The optimal feedback law depends on the full posterior belief conditioned on observed history.
  • Motivation: Generic recurrent hidden states lack physical-consistency guarantees, so their latent coordinates may not represent valid density matrices or physically meaningful Bloch vectors and purities.This motivates explicitly embedding quantum-state structure into the learned belief representation.
  • Method: The proposed two-stage pipeline pretrains a Kraus-constrained recurrent encoder, then trains PPO with its frozen output and explicit quantum features.Kraus parameterization guarantees positive-semidefinite, trace-normalized belief outputs at every step.
  • Evaluation: The Kraus-belief PPO pipeline is compared with standard LSTM-history PPO across three independently trained seeds and nominal, measurement-efficiency-shift, and hard-switch conditions.The baseline replaces the belief encoder with a plain recurrent network.

II. PROBLEM FORMULATION · A. System Dynamics and Measurement

The problem is to stabilize a continuously monitored open quantum system, specifically a single qubit, toward a target state using closed-loop feedback. The controller acts from discrete measurement increments rather than direct access to the underlying density matrix.

  • II. PROBLEM FORMULATION: The task stabilizes a single continuously monitored qubit to a target state using closed-loop feedback.The system evolves under a control-dependent Hamiltonian and continuous homodyne measurement.
  • A. System Dynamics and Measurement: The true system state is represented by the density matrix ρ_t.Its dynamics are modeled with a simulated stochastic master equation.
  • A. System Dynamics and Measurement: The state follows a diffusive stochastic master equation combining Hamiltonian evolution, dissipation, and measurement-driven noise.The equation is dρ_t = −i[H(u_t), ρ_t] dt + D[L]ρ_t dt + √η H[L]ρ_t dW_t.
  • A. System Dynamics and Measurement: The control-dependent Hamiltonian H(u_t), scalar field u_t, operator L, and efficiency η∈(0, 1] determine the dynamics and measurement response.D and H denote dissipative Lindblad and measurement innovation superoperators, while dW_t is a Wiener increment.
  • A. System Dynamics and Measurement: At each discrete time step, the controller receives a measurement outcome increment dy_t derived from the continuous record, not the true state ρ_t.This establishes partial observability as the central feedback-control setting.

B. Task Definition · III. METHOD · A. Data Generation and Pretraining

The method formulates qubit steering as a bounded-control POMDP and learns physically valid belief states from simulated measurement-action histories. A Kraus-constrained recurrent encoder produces differentiable density-matrix estimates that remain positive semidefinite and trace normalized by construction before reinforcement learning.

  • B. Task Definition: The task is to drive a qubit to the pure target state ρ⋆ aligned with the positive y-axis using y-axis control ut ∈[−4, 4] against fixed z-axis drift.The drift has angular frequency ωz.
  • B. Task Definition: The control problem is a Partially Observable Markov Decision Process in which the agent maximizes expected cumulative reward.The reward balances target fidelity against control effort and action smoothness.
  • B. Task Definition: The reward penalizes instantaneous control energy with α = 0.001 and action roughness with β = 0.0001 while incorporating fidelity Ft = Tr(ρtρ⋆).The roughness penalty encourages smooth control pulses.
  • A. Data Generation and Pretraining: The offline dataset contains 2000 simulated quantum trajectories of length T = 256 generated under a random exploratory control policy.Each step records the true density matrix ρt, the prior action ut−1, and measurement outcome dyt.
  • A. Data Generation and Pretraining: The encoder processes a recent history ht of length H = 32 to generate time-dependent Kraus operators that recursively update the density-matrix estimate.At each policy step, ˆρ0 = |0⟩⟨0| is reinitialized and the Kraus recursion spans all H = 32 observation-window steps.
  • A. Data Generation and Pretraining: An LSTM backbone maps the measurement-action history into an unconstrained latent vector, followed by a linear projection into a generic complex block matrix.This standard neural representation supplies the input to the physically constrained output head.
  • A. Data Generation and Pretraining: The output head orthogonalizes the projected matrix onto the Stiefel manifold, yielding a CPTP isometry split into Kraus rank two with two 2 × 2 operators satisfying completeness.The resulting recurrent CPTP map is differentiable and keeps the belief inside the physical state space.
  • A. Data Generation and Pretraining: The estimated state ˆρt is guaranteed to remain positive semidefinite and trace normalized, and the pretrained Kraus encoder weights are frozen during reinforcement learning.The encoder is pretrained on the offline trajectory dataset by minimizing the Frobenius loss.

B. Kraus-Belief PPO

The PPO actor uses a frozen Kraus encoder to convert measurement histories into quantum belief estimates, then maps physically interpretable geometric features to continuous control actions. Positive-semidefinite and trace-normalized construction removes redundant constraint-error inputs while preserving a geometrically constrained policy observation space.

  • At step t, the actor passes history h_t through the frozen Kraus encoder to obtain the belief estimate ˆρ_t.
  • The actor receives matrix and geometric belief features, including vec(ˆρ_t), the estimated Bloch vector, purity, minimum eigenvalue, recurrent hidden state, target Bloch vector, and Bloch error.
  • PSD violation and trace error are exactly zero by construction, so both are excluded from the network input as redundant constants.
  • A two-stage 128-dimensional MLP processes the composite features into a policy embedding, which the PPO actor maps to the final continuous control action.
  • The physics-informed feature vector changes neither the environment dynamics nor the control process; it defines a geometrically constrained observation space for the policy.

C. LSTM-History Baseline

The LSTM-history PPO baseline feeds the observation history directly through a standard LSTM actor–critic without imposing density-matrix structure, while matching the Kraus-Belief model’s 128-dimensional hidden and feature sizes.

  • Baseline architecture: The LSTM-history PPO baseline passes the same observation history h_t through a standard LSTM directly to an actor–critic head.Its actor receives an unconstrained feature vector, with no imposed density-matrix structure.
  • Input conditioning: 1/√dt-scaled homodyne increments provide the LSTM baseline with order-one measurement fluctuations without changing raw observations or Kraus encoder inputs.The previous-action channel remains unchanged.
  • Capacity matching: 128-dimensional hidden and feature dimensions are used by both the Kraus-Belief model and the LSTM baseline.This configuration provides numerical fairness between the recurrent baseline and the proposed model.

IV. EXPERIMENTS · A. Setup

The experiments compare Kraus-Belief PPO with an LSTM-history baseline under randomized training and nominal or out-of-distribution evaluation conditions. Both methods use identical training settings, deterministic 256-step evaluations, and matched PPO hyperparameters.

  • A. Setup: Training randomizes measurement efficiency η over [0.6, 1.0], drift frequency ωz over [0.5, 2.0], and damping parameter γ over [0.2, 1.0].
  • A. Setup: The Kraus-belief PPO pipeline converts observation history into a physically valid density-matrix belief before passing explicit physical features to the PPO actor.The LSTM baseline replaces this belief stage with a plain recurrent encoder.
  • A. Setup: Hard-switch dynamics are enabled during training, with switch times drawn from [64, 192] and post-switch ωz drawn from [−2.0, 2.0].Both methods use these settings; each episode lasts 256 steps with dt = 0.01.
  • A. Setup: All three evaluation configurations use identical nominal parameters ωz = 1.0 and γ = 0.5, deterministic policies, and 256-step episodes.The configurations are Nominal, Eta04 OOD, and Hard-switch OOD.
  • A. Setup: Nominal evaluation uses η = 1.0 with no dynamic switch, whereas Eta04 OOD fixes η = 0.4 in an unseen, highly noisy measurement regime.
  • A. Setup: Hard-switch OOD applies a severe mid-episode shift to an unseen drift range of [−4.0, 4.0].
  • A. Setup: Both methods share PPO settings including 300 000 interaction steps, rollout length 2048, learning rate 3×10−4, γdisc = 0.99, λ = 0.95, and clipping coefficient 0.2.Each method uses three independent seeds and 50 deterministic evaluation episodes per seed.
  • A. Setup: Reported quantitative results are means ± standard deviation across three seed-level averages, with 50 evaluation episodes per seed and 256-step episodes.

B. Metrics

The study evaluates control using episodic return, target fidelity, terminal target fidelity, and belief fidelity against the true simulated state.

  • Metrics: The reported metrics are mean episodic return, mean target fidelity over each episode, final terminal target fidelity, and belief fidelity.Belief fidelity compares the encoder’s estimate with the true simulated state.

V. RESULTS AND DISCUSSION

Kraus-belief PPO slightly outperforms LSTM-history PPO in return and fidelity across conditions, while its geometry-constrained beliefs remain physically valid and substantially reduce return variability under distribution shift.

  • Return and fidelity: 0.500±0.000 mean fidelity is achieved by the zero-control baseline against the +y target across 50 episodes.The uncontrolled qubit decoheres to a maximally mixed state.
  • Return and fidelity: Kraus-belief PPO achieves slightly higher mean return and mean fidelity than LSTM-history PPO in every evaluated condition.Mean-return improvement is consistent, while return-variance reduction is more pronounced under harder out-of-distribution conditions.
  • Return and fidelity: η = 0.4 under Eta04 OOD remains moderately below the training-time floor of η = 0.6, helping explain its mild improvement relative to Nominal evaluation.The passage characterizes this as plausible rather than a severe distributional shift.
  • Belief validity and state tracking: 0.77±0.17 nominal belief fidelity and 0.80 ± 0.16 under Eta04 OOD quantify tracking relative to the true hidden state.The Kraus encoder guarantees positive semidefinite, trace-normalized density-matrix outputs by construction.
  • Belief validity and state tracking: Zero PSD violation and trace error below 10−7 are confirmed in all evaluated episodes.The actor therefore consumes physically valid explicit geometric features.
  • Variance under distribution shift: 4.4 vs 25.8 standard deviation of return for Kraus-belief PPO versus LSTM under Eta04 OOD, and 3.6 vs 14.0 under hard-switch OOD, show lower variability.The reported reductions associate geometrically structured beliefs with more stable feedback under the evaluated distribution shifts.

VI. RELATED WORK · VII. CONCLUSION

The paper positions Kraus-parameterized belief reinforcement learning as a physics-preserving alternative to raw-record policies, using physically valid density-matrix beliefs for qubit feedback control. Across three evaluation conditions and three training seeds, Kraus-belief PPO achieves modestly higher mean return and fidelity than an LSTM-history baseline.

  • VI. RELATED WORK: Continuous-measurement quantum feedback is closely connected to quantum filtering theory.
  • VI. RELATED WORK: Data-driven feedback methods are increasingly necessary when noise models are intractable.
  • VI. RELATED WORK: Recent work infers individual quantum trajectories directly from raw observations, with high-capacity models maximizing raw fidelity.
  • VI. RELATED WORK: The proposed approach is orthogonal to raw-record policies because it constrains the agent’s internal memory rather than relying on unconstrained or attention-based representations.
  • VI. RELATED WORK: CPTP constraints are enforced strictly through matrix manifold optimization, providing a lightweight physics-preserving inductive bias.
  • VII. CONCLUSION: The pipeline maps measurement-action history to a physically valid density-matrix belief, extracts explicit physical features, and supplies them to a PPO controller.
  • VII. CONCLUSION: Across three evaluation conditions and three training seeds, Kraus-belief PPO achieves modestly higher mean return and fidelity than the LSTM-history baseline.
Loading 2608.15715v1…