Full-window branch discovery and loss-selected EnKF continuation for data assimilation

1.1. Main results.

• APK differentiation, an optimized Gaussian initial law, and loss-weighted population mixing combine to search across initial-state basins and recombine successful path portions.Mixing assigns credit across physical time and state coordinates, allowing information from different runs to contribute when no single run follows a good branch throughout the window.Source: §1.1 · p. 1
• Mean offline path RMSE is 0.698 for APK versus 3.02 for population weak-4D-Varx; mean online RMSE is 0.0959 versus 6.12 for ordinary EnKF when both remain finite.The experiments use twenty independently seeded realizations; ordinary EnKF remained finite in thirteen.Source: §1.1 · p. 2

1.2. Literature review.

• The introduction distinguishes causal filtering from retrospective full-window branch discovery, then outlines the framework, Lorenz–96 evaluations, and future adaptive exploration.It also contrasts local feedback correction and coordinate-wise population mixing with earlier nudging and genetic-search roles.Source: §1.2 · pp. 2–3, §1.3 · p. 3
• Unlike prescribed-cycle or lag choices, restart time is selected per path by minimizing a truth-free local loss over admissible interior times.The EnKF is initialized from the corresponding state, replayed through the remaining offline observations, and then continued online.Source: §1.2 · p. 3

• The model state is x_n in R^M, observations are y_n in R^m with m ≤ M, and x_0:N and y_0:N denote their full paths.The offline assimilation window is [0, T], where T = NΔt.Source: §2 · p. 3
• A causal filter uses observations only through time n, whereas APK uses the full observation window to estimate the state path, allowing later observations to correct earlier estimates.This is the paper’s smoothing or path-space viewpoint.Source: §2 · p. 3
• The online filter is applied only after offline initialization; path-space optimization is the expensive stage, followed by fast local continuation.Displayed derivatives and expectations are assumed well-defined.Source: §2 · p. 4

• APK gradients are developed for an optimized random initial condition, with a prescribed transition from kernel-based exploration to pathwise exploitation.Source: §3 · p. 4
• The framework includes offline path reconstruction and population mixing across independently optimized paths.Source: §3 · p. 4
• It also presents optional EnKF continuation whose restart state is selected using a local loss.Source: §3 · p. 4

3.1.1. Finite-time path-kernel identity.

• The finite-time path-kernel identity splits sensitivity between a pathwise derivative and a kernel term that recovers the contribution removed by damping.When α_n > 0, damping stabilizes unstable tangent directions without changing the exact parameter gradient.Source: §3.1.1 · p. 4

3.1.2. Gaussian initial-law extension.

• The Gaussian initial-law extension treats the initial draw as an auxiliary transition and optimizes its mean and scale alongside the path.Setting α_ic = 0 gives the usual reparameterization derivative; α_ic = 1 transfers the mean derivative completely to the Gaussian kernel.Source: §3.1.2 · p. 5

3.1.3. Full-window APK adjoint.

• The full-window objective depends on the sampled path, and tangent-adjoint duality yields APK gradients through a backward adjoint recursion.The stated gradient directions include the initial-law parameters μ_ic and σ_ic.Source: §3.1.3 · pp. 5–6
• Each update estimates expectations using L stochastic paths that share the current parameters, then averages their resulting gradients.The same-batch centered loss estimator is biased because it includes each path’s loss, but has the lowest variance among standard same-batch centering choices and is more stable for small L.Source: §3.1.3 · p. 6

3.2. Exploration and exploitation under prescribed schedules.

• The path-kernel damping and both stochastic scales decay together, while the linear correction remains fixed.The schedule uses α0 for the initial scales and K for the number of exploration updates.Source: §3.2 · p. 6
• Early large α stabilizes long-window tangent and adjoint sensitivities by transferring them to centered probability-kernel terms.Broad stochastic paths and the centered kernel contribution make low-loss excursions more likely to inform the optimization [18].Source: §3.2 · p. 7
• As α, σ, and σic decrease, the fixed deterministic attraction toward x∗ dominates, and updates mainly improve the path center.In this late regime, γ′ stays fixed while the kernel contribution vanishes as sampled paths and centered loss fluctuations collapse.Source: §3.2 · p. 7
• Choosing γ′ comfortably above λmax makes corrected error dynamics contractive despite chaotic underlying dynamics.The early kernel-differentiation regime helps move x∗ and µic toward a useful branch, while correction-stabilized perturbations concentrate exploration into an accurate path.Source: §3.2 · p. 7

3.3. Population mixing: construct mixed path proposals.

• Unlike random crossover between globally ranked trajectories, mixing uses temporally averaged coordinatewise losses to recombine locally successful path portions.Subsequent APK updates restore dynamical coherence; mixing occurs across population runs, not stochastic paths within a run.Source: §3.3 · p. 7
• Each population member starts from an independent model rollout and is optimized independently between prescribed mixing updates.The reported experiments mix runs every 1000 updates.Source: §3.3 · pp. 7–8
• Local spatiotemporal losses combine observation discrepancy shared across coordinates with coordinatewise correction discrepancy.The local loss is evaluated on each run’s deterministic representative path, retaining the correction while setting stochastic increments to zero.Source: §3.3 · p. 8
• A Gaussian temporal neighborhood smooths coordinate losses before soft-min weights favor lower-loss runs at each time and coordinate.The bell radius S is set to half the decorrelation time, and β controls the strength of preference for lower local losses.Source: §3.3 · pp. 8–9
• Time- and coordinate-varying weights mix the auxiliary path and initial mean, but do not mix scalar parameters.Each update replaces the run with the largest deterministic full-window objective; the best existing objective cannot increase, though later APK updates may be needed to repair mismatch.Source: §3.3 · p. 9

3.4. Loss-selected EnKF restart and continuation.

• A locally dominant, concentrated selected branch can make a single-mode uncertainty approximation such as a Gaussian reasonable for EnKF continuation.The EnKF is initialized with a small Gaussian spread around the selected APK state, replayed through the remaining offline observations, then continued causally beyond T.Source: §3.4 · pp. 9–10
• The selected path is the run with the smallest best-attained deterministic full-window objective.The restart criterion uses instantaneous loss along that selected path and a bell-shaped temporal average to avoid an isolated fluctuation determining the restart time.Source: §3.4 · p. 9
• Restart candidates are restricted to times at least one model-time unit from both window endpoints.The restriction reflects that loss is less constrained near the reconstruction-window endpoints.Source: §3.4 · pp. 9–10
• The restart score is computable without the true path, but under partial observations it does not automatically determine full-state error.Its interpretation depends on branch identifiability from observations and dynamical consistency over the temporal neighborhood.Source: §3.4 · p. 10
• EnKF continuation is expected to remain reliable only while the ensemble stays concentrated within the selected branch and the dynamics and observation map remain locally well approximated.A large innovation relative to predicted observation spread can indicate departure from this local regime; the reconstructed APK path remains available if continuation is not viable.Source: §3.4 · p. 10

3.5. Complete procedure.

• Population APK optimization with occasional mixing forms the first stage of the procedure.Algorithm 1 summarizes the stages; numerical settings are given in Section 4.1.Source: §3.5 · p. 10
• The second stage selects a path and restart time using loss.Algorithm 1 summarizes the stages; numerical settings are given in Section 4.1.Source: §3.5 · p. 10
• Optional EnKF continuation is the final stage.Algorithm 1 summarizes the stages; numerical settings are given in Section 4.1.Source: §3.5 · p. 10

4.1. Shared Lorenz–96 setting and evaluation.

• Across two 40-dimensional Lorenz–96 settings, statistics use twenty independently seeded experiments per setting; observations repeat one coordinate observed then four missed, yielding 8 observed and 32 hidden.Offline path RMSE is computed over 1 ≤ t ≤ T − 1, excluding endpoints; online RMSE covers T ≤ t ≤ 2T.Source: §4 · p. 10, §4.1 · pp. 11, 13

4.2. APK versus weak-4D-VarX.

• Direct long-window strong-constraint 4D-Var and weak-4D-Varq yield nonfinite gradients at the first optimization step, so weak-4D-Varx is the variational comparator.Both APK and weak-4D-Varx remain finite across all twenty experiments.Source: §4.2 · p. 13
• APK’s mean interior path RMSE is 4.3 times smaller than population weak-4D-Varx’s, with a mean paired RMSE reduction of 2.33 across twenty experiments.Both methods remain finite in all twenty experiments, and APK is more accurate in every paired experiment.Source: §4.2 · p. 13, Table 1 · p. 13

4.3. APK–EnKF versus ordinary EnKF.

• The APK-initialized EnKF remains finite in all twenty experiments; among thirteen experiments where both filters remain finite, its mean online RMSE is 63.8 times smaller.Ordinary EnKF becomes nonfinite in seven experiments.Source: §4.3 · p. 15, Table 2 · p. 15

4.4. Separate nonlinear-observation setting.

• With squared observations, APK’s mean interior path RMSE is 1.91 ± 0.692 versus 3.73 ± 0.543 for weak-4D-Varx, but five APK continuations select poor branches.APK–EnKF has median online RMSE 0.0134 but mean 1.68 ± 2.96; the passage attributes this to five continuations selecting poor branches.Source: §4.4 · p. 18

5. Conclusion and future work

• Against weak-4D-Varx, APK finds more accurate paths and restart states; APK–EnKF remains accurate when ordinary EnKF is inaccurate or nonfinite.Source: §5 · p. 18
• The framework combines APK gradients, an optimized random initial law, scheduled path-kernel and diffusion scales, and mixing across independent runs.Source: §5 · p. 18
• In the direct long-window implementations tested, strong-constraint 4D-Var and weak-4D-Varq have nonfinite gradients at the first optimization step.Source: §5 · p. 18
• Local spatiotemporal loss may guide adaptive exploration toward times and coordinates with larger loss, while smaller-loss locations use smaller exploration scales.The passage notes that local loss and diagnostic RMSE tend to agree, but not exactly.Source: §5 · p. 18

Data availability statement

• The study's code and generated numerical data are available from the author upon reasonable request.Source: Data availability statement · p. 18