Lagrangian--Hamiltonian Flows for Video Prediction and Image Generation: A Symplectic Perspective

1 INTRODUCTION

• Standard flow matching acts on channel values at fixed pixel locations, leaving spatial-structure changes implicit and motivating a common geometric representation.The paper situates this question alongside flow-based image generation and video prediction, though their initial conditions and objectives differ.Source: §1 · p. 1
• LHFM represents each image as an exact Lagrangian graph and evolves that graph with Hamiltonian flows, a representation described as a first of its kind to the authors’ knowledge.The graph jointly encodes pixel locations and channel values.Source: §1 · pp. 1–2
• Hamiltonian flows preserve the Lagrangian property but may not preserve graph form; a Hamilton–Jacobi condition characterizes flows transporting prescribed exact Lagrangian graphs.This characterization yields an image-velocity parameterization using transport and source fields.Source: §1 · pp. 1–2
• The paper’s main empirical focus is deterministic video prediction; LHFM-I is evaluated only against a matched I-CFM baseline, without a state-of-the-art image-generation claim.Source: §1 · p. 2
• The transport–source parameterization includes standard flow-matching velocity as the special case Ut ≡ 0.Source: §1 · p. 2
• LHFM-V has the lowest reported FLOP count among compared recurrent predictors at similar accuracy, while LHFM-I attains lower FID than a matched I-CFM baseline.The primary focus is deterministic video prediction; the image-generation experiment is a controlled comparison, not a state-of-the-art claim.Source: §1 · p. 2

2 RELATED WORK

• Flow matching trains continuous normalizing flows by simulation-free regression of conditional vector fields; LHFM-I retains the independent objective with transport–source image velocities.Source: §2 · p. 2
• Riemannian Flow Matching and Metric Flow Matching (Kapu´sniak et al., 2024) extend flow matching geometrically, whereas LHFM uses exact Lagrangian image graphs.The paper applies the classical connection between Lagrangian submanifolds and Hamilton–Jacobi dynamics to image representations.Source: §2 · pp. 2–3
• Recurrent video predictors advance hidden states frame by frame, and LHFM-V joins this category by predicting transport and source fields conditioned on observed frames.Recurrent-free predictors and generative video models are outside the work’s scope.Source: §2 · p. 3

• A smooth function S on Q defines an exact Lagrangian submanifold graph(dS), with the canonical one-form pulling back to dS.Source: §3 · p. 3
• Hamiltonian flow preserves the symplectic form and Lagrangian exactness, but may not preserve a graph representation.Source: §3 · p. 3
• The phase-space viewpoint motivates using Lagrangian graphs to organize spatial locations, channel values, and their local spatial sensitivities together.Source: §3 · p. 3

4 LAGRANGIAN–HAMILTONIAN FLOW MATCHING

• The shared geometric construction supports conditional flow matching and deterministic video prediction, with application-specific initial conditions, conditioning, losses, and numerical methods.Source: §4 · p. 3

4.1 IMAGE SPACE AND CONTINUOUS REPRESENTATION

• Each image is extended from its pixel grid to a smooth continuous map, providing the representation used in the geometric construction.Source: §4.1 · p. 3

4.2 LAGRANGIAN IMAGE REPRESENTATION

• The exact Lagrangian graph combines pixel values with their local spatial sensitivities in one geometric object.The auxiliary probe a acts linearly on channel values; the graph’s covector components encode channel values and spatial response changes.Source: §4.2 · pp. 4–5

4.3 HAMILTONIAN DYNAMICS

• Hamiltonian transport of the prescribed image graphs is equivalent to a Hamilton–Jacobi condition, and induces image velocities parameterized by transport and source fields.The transport field moves spatial locations while the source field changes channel values along those trajectories; the split is not unique and does not restrict image velocities.Source: §4.3 · pp. 5–6

5 MODELS

• Transport moves image structures, while the source changes channel values along that motion.Figure 3 illustrates the complementary roles with transport moving a digit's spatial structure and source changing its color.Source: §5 · p. 6
• At t = 0.5, pixelwise interpolation blends misaligned features, while transport–source interpolation aligns features as their appearance changes.The comparison motivates explicit spatial transport for capturing structural changes.Source: §5 · p. 6, Figure 4 · p. 7

5.1 LHFM-I: FLOW MATCHING FOR IMAGE GENERATION

• LHFM-I uses the standard independent-endpoint regression objective with a transport–source parameterization of the predicted image velocity.For sampling, it solves the velocity ODE from a prior image using EMA parameters and a fixed-step Heun solver.Source: §5.1 · p. 7

5.2 LHFM-V: DETERMINISTIC VIDEO PREDICTION

• LHFM-V recurrently updates an image state from predicted transport and source fields, recomputing them at each half-frame step.The source is integrated together with transport over each interval, and the fields are predicted from the updated recurrent state.Source: §5.2 · pp. 7–8
• For video prediction, LHFM-V uses second-order upwind spatial discretization and holds predicted transport and source fields fixed over each half-frame interval.Its Moving MNIST implementation uses ten observed and ten future frames, with two steps per future frame.Source: §5.2 · p. 8

6.1 IMAGE GENERATION

• At 64, 128, and 256 NFE, LHFM-I lowers matched I-CFM FID by 0.28, 0.28, and 0.24, respectively.Figure 5 reports these reductions for matched 150k-update models using a fixed-step Heun solver.Source: §6.1 · p. 8, Figure 5 · p. 9
• 3.5809 FID for LHFM-I versus 3.8241 for matched I-CFM at 150k updates; LHFM-I's FID is 6.36% lower.Published results are included for context, not as matched-protocol comparisons.Source: Table 2 · p. 8, §6.1 · p. 8

6.2 VIDEO PREDICTION

• The video evaluation predicts ten future Moving MNIST frames from ten observed frames, with Table 3 providing context rather than protocol-matched comparisons.The table compares recurrent video predictors; its baseline numbers come from the original papers.Source: §6.2 · p. 8, Table 3 · p. 9
• Among recurrent models in Table 3, LHFM-V has the lowest reported FLOP count (13.1G FLOPs), corresponding to its image-update mechanism rather than repeatedly reconstructing frames from latent features.Source: §6.2 · p. 9

7 CONCLUSION

• LHFM-V attains the lowest FLOP count among compared recurrent predictors at similar prediction accuracy, while a matched image-generation experiment improves over its baseline.The conclusion also says the model has not undergone systematic architecture or training-strategy optimization, leaving refinement of the accuracy–efficiency trade-off for future validation.Source: §7 · p. 9
• The current model has not undergone systematic optimization of its architecture or training strategy, and expected accuracy–efficiency improvements remain unvalidated.The conclusion identifies further refinement as future work.Source: §7 · p. 9
• The paper reports use of public image and video benchmarks, without human participants or private data.It notes that downstream deployment may reproduce dataset biases or enable misleading synthetic content, requiring application-specific evaluation and safeguards.Source: ETHICS STATEMENT · p. 10

APPENDIX CONTENTS

• The appendix includes mathematical proofs, supplementary experiments, and examples for LHFM-I and LHFM-V.It also records architectures, hyperparameters, numerical implementation, and evaluation protocols.Source: APPENDIX CONTENTS · p. 13

A MATHEMATICAL BACKGROUND AND PROOFS

• The appendix organizes proofs around interpolation and recovery, exact Lagrangian image graphs, graph-transporting Hamiltonians, and compatibility of continuous and discrete dynamics.Its opening identifies these as the results needed for the image representation and Hamiltonian dynamics in Section 4.Source: A MATHEMATICAL BACKGROUND AND PROOFS · p. 14

A.1 FULL-BAND INTERPOLATION AND EXACT RECOVERY

• Full-band DCT interpolation gives a smooth, even, 2-periodic image function that exactly recovers the pixel samples while retaining all coefficients, including the constant component.The even, 2-periodic boundary convention is part of the representation, not an assumption about the unknown continuum data distribution; the statements concern fixed resolution.Source: §A.1 · p. 14

A.2 EXACT LAGRANGIAN IMAGE GRAPHS

• Each encoded image is an exact Lagrangian graph, and the encoder is injective because the image samples can be recovered from the graph.The representation retains the image family’s nN^2 degrees of freedom and is not a compression to n + 2 scalar coordinates.Source: §A.2 · p. 15

A.3 HAMILTONIANS TRANSPORTING IMAGE GRAPHS

• A Hamiltonian transports the prescribed family of exact Lagrangian graphs if and only if it satisfies the Hamilton–Jacobi condition.The resulting characteristic fields separate spatial transport from channel-value change along that motion.Source: §A.3 · pp. 15–16

A.4 COMPATIBLE CONTINUOUS AND DISCRETE IMAGE DYNAMICS

• For an image path and spatial field with the stated smoothness and flow-existence assumptions, the compatible Hamiltonian flow transports the initial image graph to the evolving graph.The construction sets R_t = F_N(v_t) + DF_{J_t}U_t and H_t = ξᵀU_t − aᵀR_t.Source: §A.4 · p. 16
• For DCT-based dynamics, the compatible source agrees with the learned source on the grid, but need not equal its DCT extension; video updates use frozen fields intervalwise under spatial-flow existence assumptions.The implemented truncated-series updates approximate the exact grid evolutions, and the CFM objective alone does not uniquely identify transport and source fields.Source: §A.4 · p. 17

B SUPPLEMENTARY EXPERIMENTS

• The supplementary experiments train LHFM-I, LHFM-V, and the matched I-CFM baseline from random initialization without pretrained components, distillation, or additional fine-tuning.The supplementary materials provide architecture, training, and evaluation details.Source: B SUPPLEMENTARY EXPERIMENTS · p. 18

B.1 IMAGE-GENERATION ARCHITECTURES AND HYPERPARAMETERS

• LHFM-I uses two transport and three source channels, adding 2,306 parameters to the matched I-CFM model’s three image-velocity channels.Both models share a time-dependent U-Net backbone and training settings; Table 4 reports settings for the matched comparison.Source: §B.1 · p. 18, Table 4 · p. 18
• LHFM-I bounds each grid transport component by 0.125t and suppresses transport near the noise endpoint, while retaining the independent conditional flow-matching objective.The matched models use independent endpoint sampling with σ = 0; the I-CFM baseline uses the local matched backbone, not the upstream TorchCFM U-Net or a pretrained checkpoint.Source: §B.1 · p. 18
• Both models are initialized from scratch and sample training images uniformly with replacement.The EMA decay at update k is min{0.9999, (1 + k)/(10 + k)}.Source: §B.1 · p. 18

B.2 VIDEO-PREDICTION ARCHITECTURES AND HYPERPARAMETERS

• LHFM-V processes observed frames, frame differences, and coordinates, then uses recurrent-state feedback and predicted transport and source fields to produce future frames.Figure 6 depicts two transport–source half-steps per future frame; training uses the full rollout without teacher forcing, and field-prediction modules share parameters across the rollout.Source: §B.2 · p. 18, Figure 6 · p. 19
• LHFM-V is trained from scratch with AdamW using a two-phase OneCycle schedule, resuming the complete 400,000-update checkpoint before cosine decay to update 1,250,000.The learning rate is halved at resumption without additional warmup; neural training uses BF16, while image integration, recurrent states, and optimization use FP32.Source: §B.2 · pp. 18, 20

B.3 NUMERICAL IMPLEMENTATION

• For image generation, the full-band DCT spatial Jacobian is evaluated on the pixel grid and J evolves by dot J_t = v_theta(t, J_t) with 128 Heun steps (256 NFE), without explicitly evaluating the Hamiltonian during sampling.Source: §B.3 · p. 20
• Video prediction advances each future frame with two half-frame transport–source updates, using a second-order upwind stencil for −U · ∇F and zero values outside the grid.The fields are frozen over each half-step and reused across internal intervals; gradients propagate through A(w) and rnet, while scaling q and subdivision count L are constants during differentiation.Source: §B.3 · pp. 20–21

B.4 EVALUATION PROTOCOLS

• Image-generation FID uses 50,000 generated images against all 50,000 CIFAR-10 training images, with fixed-step Heun128 sampling at 256 NFE.Published Table 2 baselines use adaptive DOPRI5 and retain their original protocols, so they are not matched-control comparisons.Source: §B.4 · p. 21
• Video evaluation selects the EMA checkpoint with lowest validation MSE, then reports metrics on the official 10,000-sequence test set not used for selection.MSE and MAE use unclipped predictions; SSIM and PSNR use predictions clipped to [0, 1].Source: §B.4 · p. 21

C.1 LHFM-I: IMAGE GENERATION

• CIFAR-10 samples come from a 150k-update EMA checkpoint using 128 Heun steps (256 NFE), shown in their original 64-image order.Pixels are clipped only for evaluation, with no intermediate solver clipping.Source: §B.4 · p. 21, Figure 7 · p. 22, §C.1 · p. 22

C.2 LHFM-V: VIDEO PREDICTION

• Figures show the first six official Moving MNIST test sequences in original order, with observed frames, ground-truth futures, and deterministic predictions arranged by time.Predictions use the validation-selected 1.25M-update EMA checkpoint.Source: Figure 8 · p. 23, §C.2 · p. 23, Figure 9 · p. 24