Source-linked AI summary
Seismic Full-Waveform Inversion Using Deep Learning Tools and Techniques
Alan Richardson
TL;DR
FWI estimates subsurface properties from seismic data, and this paper recasts its forward modeling as a recurrent neural network for implementation with deep-learning software. It reports that minibatch Adam converges faster than SGD and full-dataset L-BFGS-B, while automatic differentiation yields the same gradient equations as the adjoint-state method.
Problem
FWI requires iterative nonlinear inversion of seismic data, with gradients used to update the subsurface model.
Source: §1 · p. 2Method
The paper implements wave-equation forward modeling as an RNN and applies automatic differentiation or the adjoint-state method to calculate gradients.
Source: §2.1 · p. 3, §2.3 · p. 5Results
Adam with minibatches converges significantly faster and more stably than SGD, and faster than L-BFGS-B using the entire dataset for cost and gradient calculations.
Source: §3.3 · p. 9, §5 · p. 13Takeaways & Limitations
The FWI RNN implementation is verified, and automatic differentiation calculates gradients with equations matching the traditional adjoint-state method.
Source: §2.3 · p. 5, §5 · p. 13Automatic differentiation stores wavefield time slices, and its memory requirements are likely prohibitive for realistic FWI applications.
Source: Automatic differentiation · pp. 10–11, §5 · p. 13
Abstract
from arXiv · showhide
I demonstrate that the conventional seismic full-waveform inversion algorithm can be constructed as a recurrent neural network and so implemented using deep learning software such as TensorFlow. Applying another deep learning concept, the Adam optimizer with minibatches of data, produces quicker convergence toward the true wave speed model on a 2D dataset than Stochastic Gradient Descent and than the L-BFGS-B optimizer with the cost function and gradient computed using the entire training dataset. I also show that the cost function gradient calculation using reverse-mode automatic differentiation is the same as that used in the adjoint state method.
1 Introduction
Full-waveform inversion (FWI) estimates subsurface properties from surface seismic data through iterative nonlinear inversion. The paper proposes expressing FWI as a neural network so deep-learning software and methods, including minibatch Adam, can be applied.
1 Introduction
- Deep learning trains neural networks by adjusting layer-transformation parameters using reverse-mode automatic differentiation to calculate cost-function gradients.The parameters are iteratively updated so the network produces the desired output.Source: §1 · p. 1
- FWI iteratively estimates subsurface properties from surface seismic data by minimizing differences between recorded and forward-modeled data.Its gradient is determined using the adjoint state method.Source: §1 · p. 2
- Expressing FWI as a deep neural network makes deep-learning software usable for implementation, including CPU/GPU execution and distributed-memory parallelism.The paper also identifies access to ideas such as Adam as a benefit.Source: §1 · p. 2
- The paper hypothesizes that FWI can be implemented with deep-learning software and may benefit from minibatches and the Adam optimizer.Source: §1 · p. 2
2 Materials and Methods
The methods describe FWI's forward-modeling and gradient requirements, then construct wave propagation as a recurrent neural network and outline automatic-differentiation and adjoint-state gradients. Experiments use a SEAM-derived dataset with training and development shots.
2 Materials and Methods
- FWI requires forward modeling to create synthetic data and a cost function, plus gradients to update the model; the implementation uses TensorFlow.The section outlines RNN forward modeling, two gradient approaches, and the SEAM model used in experiments.Source: §2 · pp. 2–3
2.1 Forwarding modeling
- Each RNN unit represents a finite-difference time step, carrying two adjacent wavefield time steps and a PML auxiliary field, with source amplitude as input.The implementation uses convolution for spatial derivatives and outputs the full wavefield, which is useful for adjoint-state gradients.Source: §2.1 · pp. 3–4, Figure 1 · p. 4
2.2 Automatic differentiation
- Because the wave-propagation operations are differentiable, automatic differentiation can compute cost gradients from the forward model and specified cost function.The equations for automatic differentiation of the FWI cost function are derived in Appendix A.Source: §2.2 · p. 4
2.3 Adjoint state method
- The adjoint-state gradient backpropagates the modeled-versus-recorded data difference, using forward and backward propagation through the same RNN.The paper states that its adjoint-state and automatic-differentiation equations are the same, distinguishing manual gradient implementation from software autodifferentiation.Source: §2.3 · p. 5
2.4 SEAM dataset
- The experiments use a 2D SEAM Phase I Vp section on a 100 m grid, with ten shots reserved for development and eighty for training.Development-set cost values are used to select hyperparameters with reduced risk of overfitting.Source: §2.4 · pp. 5–6
3 Results
The experiments compare gradient calculations, tune minibatch hyperparameters on a development dataset, and test Adam and SGD against full-dataset L-BFGS-B. Adam converges more quickly than both alternatives in this experiment.
3 Results
- The optimizer comparison uses minibatches for Adam and SGD, full-dataset cost and gradient calculations for L-BFGS-B, and development-data-selected hyperparameters for Adam and SGD.Source: §3 · p. 6
3.1 Gradient comparison
- Automatic-differentiation and adjoint-state gradients are similar to finite-difference gradients on the 1D test, with small edge differences unlikely to affect optimization significantly.The adjoint-state method slightly overestimates gradients.Source: §3.1 · p. 7, Figure 2 · p. 7
3.2 Using a development dataset for hyperparameter selection
- For Adam, the lowest tested development loss used a learning rate of 45.4 and batch size 2; the best tested SGD combination was 733.35 and 5.The study could not determine whether batch size affected Adam’s result.Source: §3.2 · p. 8, Figure 3 · p. 8
3.3 Adam and SGD minibatch updates compared to L-BFGS-B
- Adam converges faster than L-BFGS-B and SGD on development loss, and its model better recovers sediment and parts of the salt body than SGD’s.L-BFGS-B’s model changed little after 1,200 shot evaluations; Adam’s loss flattened after about one training-set pass, even counting hyperparameter-selection evaluations.Source: §3.3 · pp. 9–10, Figure 4 · p. 10
4 Discussion
The discussion weighs the convenience of deep-learning tools and minibatch optimization against memory and data-selection constraints. It also distinguishes this conventional-FWI implementation from learned seismic-to-model approaches.
Automatic differentiation
- Automatic differentiation reduces development time and eases experimentation, but storing every wavefield time slice makes adjoint-state gradients likely necessary for realistically sized FWI models.FWI backpropagates over the full time sequence, unlike the thousands-layer networks uncommon in typical deep-learning applications.Source: Automatic differentiation · pp. 10, 12
- FWI memory-saving options include not storing every time slice, lossy compression, disk storage, and regenerating wavefields during backpropagation.Deep-learning software generally lacks these strategies, while manual adjoint-state gradients provide control to implement some of them.Source: Automatic differentiation · p. 11
Adam
- Even including hyperparameter trials, Adam converged faster than L-BFGS-B in this example, though that may not always hold and minibatch L-BFGS may be more competitive.Adam’s momentum smooths gradients across minibatches and may explain its greater stability than SGD in the SEAM experiment.Source: Automatic differentiation · p. 12
Development datasets
- Development-set tuning worked well for this example but does not show that the full dataset could not achieve similar results; regularization tuning has a clearer overfitting concern.Seismic datasets often have fewer samples relative to model parameters than deep-learning datasets.Source: Automatic differentiation · p. 12
Other applications of deep learning to FWI
- Alternative deep-learning inversion can train a network to map seismic data to a model, enabling later application without wave propagation for tasks such as salt reconstruction.The passage also identifies low-wavenumber starting-model generation for traditional FWI as a potential task.Source: Automatic differentiation · p. 13
5 Conclusion
The paper verifies that conventional FWI can be implemented as an RNN using deep-learning software. On the example dataset, Adam with minibatches converged faster than full-dataset L-BFGS-B and performed better than SGD, while automatic differentiation may be too memory-intensive for realistic applications.
5 Conclusion
- Adam with minibatches converged significantly faster than full-dataset L-BFGS-B and performed better than SGD on the example dataset.The comparison used L-BFGS-B with cost and gradient calculated using the entire dataset.Source: §5 · p. 13
- FWI was verified implementable as an RNN in deep-learning software, with automatic-differentiation gradients similar to traditional adjoint-state gradients.Source: §5 · p. 13
- Development-set hyperparameters appeared suitable for the example, but automatic differentiation’s memory requirements are likely prohibitive for realistic applications.Source: §5 · p. 13
A Derivation of automatic differentiation procedure
The derivation shows that reverse-mode automatic differentiation accumulates the cost-function gradient backward through time and yields the same equation as the adjoint state method. It illustrates the procedure for a simplified one-shot, one-receiver case using squared-error cost.
A Derivation of automatic differentiation procedure
- The derivation simplifies the calculation to a discretized 1D scalar wave equation with one shot and one receiver, excluding non-reflecting PML regions.Source: A Derivation of automatic… · p. 15
- The derivation uses a sum-of-squared-errors cost and expresses its dependence on wave speed through the propagated wavefield.With a second-order time finite-difference scheme, each wavefield step depends on wave speed and the two preceding steps.Source: A Derivation of automatic… · pp. 15–16
- Reverse-mode differentiation accounts for dependencies among wavefield time steps by specifying which variables are held constant in each partial derivative.The derivation notes that a derivative with respect to one time step includes effects on later wavefields that depend on it.Source: A Derivation of automatic… · p. 16
- Automatic differentiation accumulates wave-speed gradient terms by applying the chain rule backward through time steps.Source: A Derivation of automatic… · p. 16
- The derived gradient equation is the same as the one used in the adjoint state method.Source: A Derivation of automatic… · p. 18