Neural ODEs with stochastic vector field mixtures

Neural ODEs

• NODEs interpret neural-network residual dynamics as continuous ODE flows whose solver-generated trajectories transform inputs into outputs.The hidden state follows small vector-field steps across the integration interval, with parameters optimized for data likelihood.Source: §1 · p. 1

Baseline limitations

• Even highly flexible NODE vector fields cannot model arbitrary problems, including a simple one-dimensional task.This limitation was demonstrated in prior work [6].Source: §1 · pp. 1–2
• NODE advections preserve topology, forbidding crossing, while NODE and ANODE models also cannot map one input to multiple targets or represent system variance.ANODEs address crossing by lifting representations into an augmented space, but splitting and scaling remain unresolved.Source: §1 · p. 2

Motivation

• Behavioural forecasting requires handling crossing paths, latent-intent splitting, and variable speeds or directions from identical starting conditions.These cases motivate probabilistic trajectory models for indoor movement and related human behaviour.Source: §1 · p. 2, Figure 1 · p. 2

Contribution

• The paper introduces stochastic vector field mixtures and trajectory-focused losses, using uncertainty in fields and component selection to solve the three tasks where baselines fail.The proposed methods are developed for forecasting and also demonstrated on classical classification tasks.Source: §1 · p. 2

Vector field mixtures

• A K-component vector field mixture represents component membership as a probability distribution π(t) over the K-simplex.The mixture combines component-specific NODE units through a latent gate.Source: §2.1 · pp. 2–3

Dynamic component selection

• Forward filtering propagates component-membership uncertainty through transition and emission dynamics over the integration interval.The resulting belief update can also be interpreted as a probabilistically growing recurrent network.Source: §2.1 · p. 3

Stochastic vector fields

• Stochastic vector fields model uncertainty separately in direction and length, sampling these variables and transforming them into vector fields.The decomposition supports directional preservation and imposes length uncertainty in the log domain.Source: §2.2 · p. 3

Trajectory losses

• Transportation, variance, and combined TVLoss regularize path length and directional variation, encouraging straighter trajectories that can require fewer function evaluations.The minimum path distance is achieved when vector-field evaluations share a direction.Source: §2.3 · p. 4

Forecasting losses

• Forecasting loss penalizes deviations along observed measurement sequences, while mixture density loss reconciles stochastic mixture outputs with their predicted density.The forecasting formulation uses interpolated targets and a model-dependent secondary loss.Source: §2.3 · p. 4

Sampling

• SVFM dynamically samples one of multiple vector fields at each evaluation position, producing branching trajectories through the component distribution π(t_i).The approach considered here is myopic: each step is based on a sample from π(t_i), while particle filtering and reinforcement learning are identified as future alternatives.Source: §3 · p. 4

SVFM relationships

• SVFM units combine stochastic vector-field behavior with mixture-based selection, but discontinuous samples require careful scheduling for ODE solver guarantees.Standard RK4 and Dormand–Prince evaluation assumptions do not directly hold because SVFMs are discontinuous everywhere.Source: §3 · p. 5, Figure 5 · p. 5
• Uncertainty over vector fields and component selection enables SVFM models to represent scaling and splitting cases that NODE and ANODE models cannot solve.Mixture distributions and uncertainty are identified as key factors for these capabilities.Source: §3 · p. 5
• The experiments primarily evaluate SVFM rather than A-SVFM because the study focuses on forecasting, while augmentation lifts representations beyond the original feature space.A-SVFM is described as the most complete model because it incorporates all three modeling components.Source: §3 · p. 5

Theory

• Theoretical analysis proves that NODE and ANODE cannot model splitting and scaling, while characterizing minimal-NFE vector fields motivates variance and transportation losses.The resulting minimal-NFE vector fields are trivial and generally not useful, but the analysis supports the proposed loss design.Source: §3 · p. 5

Illustrative examples

• SVFM produces linear transformations on nested circles, avoiding the nonlinear transport required by baseline models and thereby reducing function evaluations.Different particles can follow independent trajectories when assigned to different vector-field components.Source: §4 · pp. 5–6

Cyclic continuation

• Cyclic VF continuation extrapolates periodic behaviour far beyond the training interval while preserving faithful intermediate evaluations, unlike traditional vector-field parameterisations.The demonstration queries t*=10π after training on 0≤t≤2π.Source: §4 · p. 6

Forward evaluation analysis

• SVFM models deliver the greatest NFE savings across moons, circles, and XOR, surpassing both baseline VF and VF+TVLoss models.Per-instance comparisons on XOR show SVFM dominating VF, with supplementary distributions reporting more significant savings on the other datasets.Source: §4 · p. 7, Figure 9 · p. 7

Behavioural forecasting

• On residential movement data, SVFM samples four endpoint-specific trajectory families while preserving unique paths and representing kitchen endpoint variation of approximately two metres.The model uses directional and length uncertainty to generate varied trajectories around shared behavioural patterns.Source: §4 · pp. 7–8, Figure 10 · p. 8
• Time-conditioned SVFM forecasts shift toward landing endpoints at night, demonstrating sensitivity to temporal context rather than uniform room-entry probabilities.The synthetic night condition assigns landing probability 0.9 and kitchen probability 0.1.Source: §4 · p. 8
• Baseline NODE and ANODE models cannot adequately characterize the natural variation of walked paths relevant to high-entropy behavioural transitions.The paper connects this setting to distinguishing wandering behaviour from lower-entropy transitions in home environments.Source: §4 · p. 8

• Trajectory-based losses lead to models with lower effective complexity, while uncertainty in vector fields delivers simpler solutions.Source: §5 · p. 8
• Stochastic vector field mixtures facilitate improved performance and linearly transported solutions.Source: §5 · p. 8
• The proposed models capture rich variation in human behaviour and forecast human movement in behavioural modelling.Source: §5 · p. 8

NODE and ANODE fail cases

• NODEs cannot solve crossing, splitting, and scaling because ODE solutions cannot intersect, split, or remain distinct when identical initial conditions require divergent paths.The argument relies on existence and uniqueness of ODE solutions and the resulting homeomorphic feature mappings.Source: §A.1 · p. 10
• ANODEs retain NODEs’ homeomorphic feature mappings, so they cannot model the one-to-many splitting function gs.The paper explicitly states that augmentation resolves crossing but not splitting.Source: §A.1 · p. 10
• The proposed stochastic vector-field assignment overcomes scaling’s identical-initial-condition limitation by probabilistically assigning data to vector fields.This enables the scaling equation to be modelled despite paths that would otherwise remain identical.Source: §A.1 · p. 10

ODE solver analysis

• Aligned vector fields can make truncation error approach zero, allowing one-step solutions and motivating variance-reduction losses that reduce expected function evaluations.The limiting case is minimal in function evaluations, and experiments report savings with these losses.Source: §A.2 · p. 11
• Embedded Runge–Kutta solvers adapt their step size using truncation-error estimates to satisfy a specified tolerance.The Butcher tableau defines vector-field evaluation coefficients, temporal offsets, and weights.Source: §A.2 · p. 11

Synthetic data

• The synthetic evaluation covers moons, nested circles, and XOR classification datasets.Each dataset contains positive and negative classes.Source: §B.2.1 · p. 12
• Each synthetic dataset uses 1,000 datapoints for training, testing, and validation.The passage states the same sample count for each split.Source: §B.2.1 · p. 12
• The datasets’ configurations are illustrated with positive points in red and negative points in blue.The colour coding appears in the top row of Figure 12.Source: §B.2.1 · p. 12

Behavioural data collection

• A bespoke robotic wearable collected 2D LiDAR point clouds while a participant walked from one living-room origin to four household targets.The targets were the front door, study room, kitchen, and stair landing.Source: §B.2.2 · p. 12
• The collection protocol required selecting a target, walking there in roughly 5–10 seconds, and repeating each procedure 10 times.Every trial began at the same origin anchor.Source: §B.2.2 · p. 12

Data processing

• Two-dimensional SLAM extracted maps and locations, while ICP associated the locations so the walking tracks overlapped before modelling.Figure 11 shows maps for all four targets, and low-cost LiDAR and SLAM introduced some beginning-of-path variation.Source: §B.2.2 · pp. 12–13, Figure 11 · p. 13
• The raw and processed behavioural data are intended for release in a public repository after publication.This is stated as a planned release rather than an already available resource.Source: §B.2.2 · p. 13

C Supplementary results

• Figures 12 and 13 support and deepen the intuition for the main-paper results through self-contained visualizations and comparisons.The supplementary text identifies these figures as supporting the main discussion points and conclusions.Source: C Supplementary results · p. 13
• Figure 12 visualizes SVFM solution paths for moons, nested circles, and XOR, including the final predictive distribution.Rows show points through each solution path; the bottom row shows the predictive distribution after the final transformation.Source: Figure 12 · p. 14
• The moons visualization shows minimal total particle movement, while the nested-circles visualization shows overlapping classes splitting apart across successive rows.These examples illustrate distinct trajectory behaviors in the supplementary solutions.Source: Figure 12 · p. 14
• Figure 13 reports significant function-evaluation savings for TFLosses and SVFM models across all illustrative experiments.Compared with baseline approaches, the reported savings are approximately 3,000-13,000 function evaluations, or 3-13 per instance.Source: Figure 13 · p. 15