PaperlayerView exact revision on arXiv

Source-linked AI summary

SPNets: Differentiable Fluid Dynamics for Deep Neural Networks

Connor Schenck, Dieter Fox

arXiv:1806.06094v2cs.RO

TL;DR

Robotic liquid manipulation is difficult because liquids are unstructured and prior models were often limited in representation or differentiability. SPNets integrate particle-based fluid dynamics into deep networks through ConvSP and ConvSDF, enabling differentiable learning and control. The framework learns fluid parameters, controls liquids, learns manipulation policies, and improves liquid-state matching when combined with perception.

  • Problem

    Robots have made limited progress with liquids because liquid states are highly unstructured, while prior particle-based fluid models were not fully differentiable.

  • Method

    SPNets combine ConvSP and ConvSDF with standard neural-network layers to implement Position Based Fluids over unordered particles, using fluid parameters as network parameters.

  • Results

    SPNets learn fluid parameters, control liquids, learn liquid-control policies, and track liquids with perception; perception raises pouring-sequence IOU from 36.1% to 56.8%.

  • Takeaways & Limitations

    A fully differentiable fluid model can support model-based robotic tasks spanning parameter identification, liquid manipulation, policy learning, and perception-assisted tracking.

  • Takeaways & Limitations

    The authors identify residual modeling against real fluids and modeling substances such as sand or flour as future work.

Abstract

from arXiv · show

In this paper we introduce Smooth Particle Networks (SPNets), a framework for integrating fluid dynamics with deep networks. SPNets adds two new layers to the neural network toolbox: ConvSP and ConvSDF, which enable computing physical interactions with unordered particle sets. We use these lay- ers in combination with standard neural network layers to directly implement fluid dynamics inside a deep network, where the parameters of the network are the fluid parameters themselves (e.g., viscosity, cohesion, etc.). Because SPNets are imple- mented as a neural network, the resulting fluid dynamics are fully differentiable. We then show how this can be successfully used to learn fluid parameters from data, perform liquid control tasks, and learn policies to manipulate liquids.

1 Introduction

SPNets combine analytical fluid dynamics with deep networks to address the challenge of robotic liquid manipulation. The framework represents fluid interactions differentiably and supports learning fluid parameters, liquid control, and policy learning.

  • Robots have made less progress with liquids than rigid objects because liquid states are highly unstructured and difficult to interface with deep networks.
  • SPNets combine analytical fluid dynamics with deep neural networks to enable robots to interact with liquids.
  • ConvSP and ConvSDF let networks interface directly with unordered particle sets and compute particle-based physical interactions.
  • The network directly implements Position Based Fluids, with fluid parameters such as viscosity and cohesion serving as network parameters.
  • Because fluid dynamics are implemented as a neural network, SPNets provide full analytical gradients for learning and control.

2 Related Work

Prior liquid-robotics work used task-specific, coarse, grid-based, or nondifferentiable models. SPNets address this gap with a fully differentiable particle-based fluid model for several liquid-manipulation tasks.

  • Liquid manipulation research has examined pouring, perception, simulation, and tracking, but earlier approaches generally used task-specific models or coarse fluid dynamics.
  • Unlike the cited liquid simulator that was not differentiable, SPNets provide a fully differentiable fluid-dynamics model and apply it to several tasks.
  • Prior parameter-learning methods estimated fluid properties through action-based differences, handcrafted features, latent-state models, or simulator–deep-network combinations.
  • Grid-based representations enabled some prior methods to use standard neural-network operations, whereas SPNets target unordered particle representations.
  • PBF models incompressible fluids such as water, while SPH models compressible fluids such as air; SPNets are presented as the first fully differentiable particle-based fluid model.

3 Position Based Fluids

The paper uses Position Based Fluids as a particle-based approximation of incompressible fluid dynamics. Each timestep applies forces, solves pressure-related constraints, and updates viscosity to produce new particle positions and velocities.

  • PBF is a Lagrangian approximation of the Navier–Stokes equations for incompressible fluids, representing liquids as collections of particles.
  • The algorithm maps current particle locations P and velocities V to new locations P′ and velocities V′ using timestep duration ∆T.
  • For each timestep, the PBF update applies external forces, moves particles to solve constraints, and applies viscosity.
  • The three fluid constraints are pressure, cohesion, and surface tension, each producing a position correction for particles.
  • Pressure correction moves high-pressure particles away from one another to reduce pressure toward the constraint.
  • Pressure depends on particle density relative to rest density, with the pressure constant controlling the correction.

4 Smooth Particle Networks

SPNets reproduces Position Based Fluids inside a deep network by combining particle-particle and particle-object interaction layers with standard operations. The resulting network computes fluid updates from particle states while exposing fluid parameters for learning.

  • ConvSP: ConvSP computes particle-particle interactions by smoothing feature vectors over neighboring particles within a cutoff radius.The kernel weights interactions according to inter-particle distance and produces smoothed values for each particle.
  • ConvSP: ConvSP enables pressure, cohesion, surface-tension, and viscosity solutions within the PBF computation.The same layer structure supports all four corrections, with pressure computed by first estimating particle densities and pressures.
  • ConvSP: Pressure computation uses ConvSP(P, M) to estimate density, then derives pressure from density relative to ambient density and the pressure parameter.The resulting pressure values are unfolded into ConvSP operations to compute position corrections.
  • ConvSDF: ConvSDF computes particle-static-object interactions by evaluating signed distance functions around each particle with a convolutional kernel.Negative SDF values indicate particles inside objects; kernel offsets, weights, and dilation determine the sampled locations.
  • ConvSDF: A size-one ConvSDF kernel yields penetration distances, which are combined with surface directions to correct colliding particle positions.The correction multiplies each penetration distance by its direction and adds the result to the particle position.
  • SPNets: SPNets combines ConvSP, ConvSDF, and standard layers to reproduce PBF line by line from particle positions and velocities.Each timestep maps current positions P and velocities V to updated positions P′ and velocities V′; Figure 2 also depicts parameter-estimation training.

5 Evaluation & Results

The evaluation covers fluid-parameter estimation, liquid control, reinforcement-learning policy learning, and perception-assisted liquid-state tracking. Across these tasks, SPNets estimates parameters, controls liquids, catches or pours liquid accurately, and improves state tracking when perception is added.

  • Liquid Control: Analytical gradients let SPNets optimize controls for three liquid-control scenes using model predictive control.The evaluated scenes were plate, pouring, and catching tasks; controls were re-optimized over short finite horizons.
  • Learning Fluid Parameters: SPNets estimates cohesion and viscosity from simulated ladle sequences, converging to ground-truth values across all 9 sequences and both loss functions.The projection loss also converged correctly, showing that gradients through camera projection captured parameter-induced liquid changes.
  • Liquid Control: The plate controller placed 100% of the liquid into the target bowl for all but one bowl, where only a small number of particles missed.Successful trajectories dipped the plate, waited for momentum, and returned it upright, rather than merely dipping toward the target.
  • Liquid Control: Pouring errors stayed within 11g of the target in every case, with a 5g average difference across 11 runs and no liquid spilled onto the ground.The cup rotation emerged implicitly from a loss specifying only the desired liquid target.
  • Liquid Control: In the catching scene, the target cup caught the vast majority of liquid across evaluations, with only a small amount dropped during initial movement.The task shifted a ground cup beneath an arbitrarily moving source stream.
  • Combining SPNets with Perception: Perception increased liquid-state tracking IOU from 36.1% with SPNets alone to 56.8% across 12 pouring sequences.The perception-assisted system corrected simulated state using camera observations when model mismatch accumulated.

6 Conclusion & Future Work

SPNets embeds differentiable fluid dynamics and rigid-object interactions in a deep network, supporting fluid identification, control, policy learning, and perception-based tracking. The paper also outlines extensions toward residual modeling, other substances, and more complex manipulation tasks.

  • Conclusion: SPNets computes differentiable fluid dynamics and rigid-object interactions inside a deep network using ConvSP and ConvSDF layers.These layers compute particle-particle and particle-rigid-object interactions, while standard network layers implement the fluid dynamics.
  • Conclusion: The evaluation demonstrates applications to fluid-parameter identification, liquid control, policy learning, and perception-based liquid tracking.The authors present these applications as evidence of the broad applicability of the fully differentiable fluid model.
  • Conclusion: SPNets specifies identification and control in terms of desired liquid states, with controls determined by physical interactions with controllable objects.This contrasts with prior pouring approaches that manually specified relationships between controls and liquid states.
  • Future work: Future work includes residual models for real fluids, dynamics for sand or flour, and more complex tasks such as mixing or transferring materials between containers.The paper also mentions online identification and prediction of liquid behavior as a possible extension.
  • SPNet implementation: The network applies external forces, updates positions, iteratively solves pressure, cohesion, surface-tension, and collision constraints, then updates velocities and viscosity.The architecture computes one fluid timestep from particle positions and velocities to new positions and velocities.
  • SPNet implementation: The constraint-solve loop is shown with 3 iterations, although the network can use any number of iterations.Each iteration partially updates particle positions to better satisfy the constraints.

C Model Comparison

SPNets is compared with Nvidia FleX, an established implementation of the same Position Based Fluids algorithm, using matched parameters and particle-set overlap. The results show close agreement across two scenes.

  • Setup: SPNets and Nvidia FleX use the same Position Based Fluids algorithm and are compared with matched model parameters.The comparison initializes the particle states consistently before evaluating the simulations.
  • Results: 91.0% IOU is obtained for the scooping scene and 97.1% for the ladle scene.IOU is computed at each time point using a 2.5 cm neighborhood tolerance for matching particles.
  • Results: The comparison reports that SPNets is not identical to FleX but matches closely and produces stable fluid dynamics.The conclusion follows from the particle-set overlap results across both scenes.

D Evaluation Details

The evaluation-details section introduces the procedures used to assess the model on the control tasks from Section 5.

  • Evaluation: The authors provide additional details on evaluating SPNets for the control tasks described in Section 5.The supplied passage introduces the evaluation procedure without specifying its individual experiments.

D.1 Estimating Fluid Parameters

Fluid parameters are estimated by rolling SPNet forward on simulated data, comparing predictions with ground truth, and updating parameters through backpropagation and gradient descent. The ladle experiment varies cohesion and viscosity across nine sequences and evaluates both state-aware and projection losses.

  • Parameter estimation: Ground-truth sequences generated with Nvidia FleX are used to estimate fluid parameters by backpropagation and gradient descent until convergence.FleX implements Position Based Fluids, providing the data against which SPNet predictions are optimized.
  • Parameter estimation: Each iteration samples particle positions and velocities, rolls SPNet forward T timesteps, and computes loss against the corresponding future ground-truth state.The procedure uses batches of sampled states and predicted future positions and velocities.
  • Experiment setup: The ladle experiment contains 9 sequences covering cohesion λc ∈ {0.05, 0.1, 0.15} and viscosity λv ∈ {30, 60, 90}, with other parameters fixed.Each sequence lasts 620 frames; training uses batch size 8, T = 2, Adam, and learning rate 1e-2.
  • Evaluation losses: Evaluation uses both L1 loss over matched particle states and projection loss that avoids requiring particle data association.The projection loss simulates camera observations as binary pixel labels, making it applicable when particle correspondences are unavailable.

D.2 Liquid Control

SPNets uses differentiable rollouts and Model Predictive Control to optimize liquid-manipulation controls across three scenes: plate targeting, pouring, and catching.

  • Control optimization: Model Predictive Control repeatedly simulates a fixed horizon with SPNets, differentiates the accumulated loss with respect to controls, and re-optimizes at every timestep.The horizon is set to T=10, with velocity controls initialized to zero.
  • Plate Scene: The plate task tilts a liquid-filled plate about two axes so the particles fall into a selected target bowl.Eight evaluations use each bowl as the target once.
  • Pouring Scene: The pouring task optimizes cup rotation to move a desired amount of liquid toward the bowl, then encourages the cup to return upright.Eleven evaluations vary the desired poured amount from 75g to 275g.
  • Catching Scene: The catching task shifts a target cup horizontally to catch liquid released from an arbitrarily moving source cup before it reaches the ground.

D.3 Learning a Liquid Control Policy via Reinforcement Learning

The policy-learning procedure maps observations to controls, then trains policy parameters by backpropagating through differentiable fluid rollouts; experiments train on catching trajectories generated by MPC.

  • Policy representation: The policy computes each control as π(o_t, θ), where observations are derived from particle positions and velocities and θ denotes policy parameters.
  • Policy optimization: Because the fluid model is fully differentiable, the policy is rolled out for a fixed horizon and its parameters are updated by backpropagating loss gradients through the rollout.
  • Training data: Training samples random timesteps from eight MPC-generated catching sequences and starts each rollout from the corresponding particle state and velocities.The target cup’s initial x position is perturbed with Gaussian noise.
  • Policy architecture: The policy network uses a convolutional layer, rectified linear layers, a 100-unit hidden layer, and a final one-unit output passed through tanh.

E.1 Methodology

The perception methodology closes the loop between RGB observations and SPNets simulation by converting liquid pixels into particle corrections that limit accumulated state error.

  • Motivation: Open-loop simulation errors can compound over time, so perception corrections are used to keep the simulated liquid aligned with the real liquid.
  • Methodology: The method simulates liquid forward alongside real pouring interactions and corrects the simulated state at each timestep using RGB observations.It replaces the prior thermographic-camera setup with an inexpensive RGB camera and an LSTM-FCN.
  • Perception correction: SOLVEPERCEPTION treats perception correction as a constraint in the inner PBF loop and outputs particle displacements that better satisfy observed liquid labels.
  • SOLVEPERCEPTION: The method computes observed distance-field gradients from LSTM-FCN liquid labels, blends them with gradients from projected particles, and projects the result back onto 3D particles.
  • Training: The network is pretrained using particle placement and ground-truth liquid labels, with a loss combining particle-to-liquid accuracy and liquid-pixel coverage.Pretraining runs for 48,000 iterations with ADAM at learning rate 0.0001 and batch size 4.

E.2 Evaluation

The evaluation compares SPNets with perception against open-loop SPNets alone on real pouring sequences, measuring agreement between simulated and ground-truth liquid masks.

  • Experimental setup: The evaluation uses 12 real-robot pouring sequences with cup and bottle source containers and initial fill levels of 30%, 60%, or 90%.
  • Comparison: SPNets with perception is compared against SPNets alone, where SOLVEPERCEPTION is omitted and the liquid state is tracked open-loop.
  • Optimization rollout: The perception-network rollout diagram shows three forward dynamics timesteps followed by backward loss-gradient propagation to optimize θ.
  • Metric: Intersection-over-union is computed across all frames by comparing projected particle labels with ground-truth pixel labels from the thermal camera.Each projected particle is rendered as a circle of radius 5 before calculating IOU.
Loading 1806.06094v2…