PaperlayerView exact revision on arXiv

Source-linked AI summary

When Deep Learning Meets Digital Image Correlation

S. Boukhtache, K. Abdelouahab, F. Berry, B. Blaysat, M. Grediac, F. Sur

arXiv:2009.03993v1eess.IVcs.AIcs.LG

TL;DR

CNN use for displacement and strain measurement from speckle images had not previously been explored, and existing optical-flow datasets do not match the task’s deformed-speckle setting. The paper develops StrainNet and tailored speckle datasets; on Star images, its accuracy is similar to DIC at low spatial frequencies and better at high frequencies.

  • Problem

    CNNs had not previously been used to retrieve dense subpixel displacement fields from speckle-image pairs, and existing optical-flow datasets largely cover natural images and rigid or quasi-rigid motion.

    Source: §1 · p. 1, §4.1 · p. 5

  • Method

    The authors develop StrainNet and train it using purpose-built synthetic speckle datasets with known displacement fields.

    Source: §1 · p. 1, §6.2.1 · p. 18

  • Results

    On Star images, StrainNet’s accuracy is similar to DIC at low spatial frequencies and better at high spatial frequencies.

    Source: §6.2.2 · p. 19

  • Takeaways & Limitations

    The Star-image comparisons support StrainNet as an alternative to DIC when high-frequency displacement accuracy matters.

    Source: §6.2.2 · p. 19

    The 8 × 8-region training dataset limits generalization, particularly for Star displacements with high spatial frequencies.

    Source: §6.2.1 · pp. 16–17

Abstract

from arXiv · show

Convolutional Neural Networks (CNNs) constitute a class of Deep Learning models which have been used in the recent past to resolve many problems in computer vision, in particular optical flow estimation. Measuring displacement and strain fields can be regarded as a particular case of this problem. However, it seems that CNNs have never been used so far to perform such measurements. This work is aimed at implementing a CNN able to retrieve displacement and strain fields from pairs of reference and deformed images of a flat speckled surface, as Digital Image Correlation (DIC) does. This paper explains how a CNN called StrainNet can be developed to reach this goal, and how specific ground truth datasets are elaborated to train this CNN. The main result is that StrainNet successfully performs such measurements, and that it achieves competing results in terms of metrological performance and computing time. The conclusion is that CNNs like StrainNet offer a viable alternative to DIC, especially for real-time applications.

1. Introduction

DIC provides full-field displacement and strain measurements, but its iterative computation and low-pass behavior limit dense and high-spatial-frequency measurements. This paper investigates whether CNNs can retrieve dense subpixel displacement fields from speckle-image pairs.

1. Introduction

  • DIC’s iterative procedure uses significant computational resources and its low-pass filtering blurs displacement and strain fields, including high-spatial-frequency fields.The paper focuses on subpixel displacements, which differ from the larger rigid-body movements commonly addressed in computer vision.Source: §1 · p. 1
  • The study investigates whether a CNN can retrieve dense subpixel displacements from reference and deformed random-speckle images, a route the authors say had not previously been explored.The task also requires thorough estimation of measurement errors because it is intended for metrology.Source: §1 · p. 1
  • The paper develops StrainNet and specific training datasets, then tests the network on speckle-image pairs from the DIC Challenge and a previous study.The numerical experiments can be reproduced with the provided Matlab and Pytorch code and datasets.Source: §1 · p. 2

2. A short primer on deep learning

The section introduces neural-network layers, CNN operations, and supervised training. It also explains how pre-trained networks can be adapted when training a deep CNN from scratch is impractical.

2. A short primer on deep learning

  • In this regression task, two images form the input and the displacement field is the output; output neurons have no activation function.Hidden and output neurons combine weighted inputs using an activation function, commonly ReLU.Source: §2 · p. 2
  • CNNs constrain weighted sums to convolutions, reducing the number of weights; stride-two convolutions down-sample feature layers, and transposed convolutions up-sample predictions.The feature extraction part becomes progressively narrower, while up-sampling is used to predict displacement.Source: §2 · p. 2
  • Supervised training adjusts network weights to reduce a loss over expected and predicted outputs, commonly using mini-batch stochastic gradient descent across many epochs.The learning rate sets the magnitude of each weight update.Source: §2 · pp. 2–3
  • When training a deep CNN requires infeasible data or computation, pre-trained networks can be adapted by fine-tuning weights or changing part of the architecture through transfer learning.In transfer learning, the remaining weights are kept constant while the corresponding new weights are learned.Source: §2 · p. 3

3. A brief review of CNN-based methods for optical flow estimation

CNNs estimate optical flow through feature extraction and flow prediction, but existing designs trade accuracy, model size, speed, and training complexity. The paper adapts this line of work to small, deformational subpixel displacements and metrological error assessment.

3. A brief review of CNN-based methods for optical flow estimation

  • FlowNetS concatenates the two input images, while FlowNetC processes them separately and combines feature maps with a correlation layer; both output quarter-resolution flow requiring bilinear upscaling.The FlowNet-f schematic shows feature extraction followed by up-sampling and displacement prediction, with features concatenated across levels.Source: §3 · pp. 3–4, Figure 1 · p. 4
  • Hu et al.’s recurrent spatial-pyramid network has comparable performance to FlowNetS with 90 times fewer weights, but runs twice as slowly because of recurrent operations.Its initial flow is upscaled and refined recurrently on a spatial pyramid.Source: §3 · p. 4
  • FlowNet 2.0 reduces estimation error by more than 50% versus FlowNet, but has over 160 million weights, runs four times slower, and requires sequential sub-network training [27].Its larger model and slower inference accompany its accuracy improvement.Source: §3 · p. 4
  • SpyNet is about 26 times smaller than FlowNet and has similar running speed, but is less accurate than FlowNetS and FlowNet 2.0 and harder to train [28].Its pyramid levels are trained independently and use previous levels to initialize image warping.Source: §3 · p. 4
  • PWC-Net outperforms previous optical-flow methods on MPI Sintel and KITTI 2015, with about 17 times fewer weights and half the inference time of FlowNet 2.0 [30].It combines pyramidal processing, warping, and cost volume.Source: §3 · p. 5
  • The proposed StrainNet task assumes planar deformation of a randomly speckled surface and targets small subpixel displacements with metrological error assessment.Earlier CNN applications to particle-image velocimetry reported promising results but did not outperform classical methods in accuracy.Source: §3 · p. 5

4. Dataset

Existing optical-flow datasets do not match the paper’s target of tiny displacements in deformed speckle images. The authors therefore construct synthetic speckle datasets with known displacement fields for training and validation.

4.1. Existing datasets

  • Almost all existing optical-flow datasets cover large rigid-body or quasi-rigid motions in natural images, rather than deformed speckles.The new dataset is designed to mimic reference and deformed speckle pairs with associated deformation fields.Source: §4.1 · pp. 5–6

4.2. Developing a speckle dataset

  • Speckle dataset 1.0 was developed to support estimation of displacements around 10^-2 pixel and contains 36,663 image pairs with subpixel ground-truth fields.The paper also develops a second version, Speckle dataset 2.0.Source: §4.2 · p. 6
  • Reference images were rendered as 256 × 256 pixel frames of randomly overlaid black disks; 363 visually screened frames were retained as speckle references.The generator was chosen to produce patterns similar to experimental-mechanics speckles.Source: §4.2 · p. 6
  • Dataset 1.0 displacement fields assign random corner displacements to adjacent 8 × 8 pixel regions, interpolate within each region, and set boundary displacements to zero.Imposed displacements ranged from -1 to +1 pixel; Figure 2 illustrates the speckles and x/y displacement fields.Source: §4.2 · p. 7, Figure 2 · p. 7
  • Each reference frame generated 101 deformations: 36,300 pairs formed the training set and 363 pairs the validation set, evaluated using average endpoint error.AEE is the pixel-averaged Euclidean distance between predicted and ground-truth flow vectors; bicubic interpolation was used to deform images as a trade-off between computing time and quality.Source: §4.2 · p. 7

5. Fine-tuning networks of the literature

The paper fine-tunes optical-flow CNNs for subpixel displacement estimation, then assesses their performance against DIC on test images and a synthetic Star displacement. FlowNetS-ft is the strongest of the pre-existing networks, but its initial accuracy remains short of the expected 0.01-pixel resolution.

5. Fine-tuning networks of the literature

  • Because the dataset differed from the networks’ original training data, fine-tuning updated weights in all layers; the selected models prioritized accuracy and computational efficiency.The study considered FlowNet, PWC-NET, and LiteFlowNet models.Source: §5 · p. 8

5.1. Training strategy

  • Fine-tuning used batch size 16, a learning rate of 10^-4 halved every 40 epochs, and 300 epochs; each network took three to four days on an NVIDIA Tesla V100 GPU.The 300-epoch duration was chosen because the loss stabilized.Source: §5.1 · p. 8

5.2. Obtained results

  • 0.1 pixel was the best average AEE among the fine-tuned networks, still above the expected 0.01-pixel resolution; FlowNetS had the lowest average AEE.Train and test average AEE values were about the same across the networks, suggesting no overfitting; the figure reports test-set AEE.Source: §5.2 · p. 8, Figure 3 · p. 9

5.3. Assessing the results with a reference Star displacement field

  • The synthetic Star displacement uses a sine wave whose period increases toward the right, enabling assessment of frequency-dependent attenuation in the estimated displacement.The off-center green region was used for numerical evaluation and emphasizes high spatial frequencies.Source: §5.3 · p. 9
  • The Star test compared four fine-tuned CNNs with subset-based DIC using 11- and 21-pixel subsets and first-order subset shape functions.The subset shift was one pixel, putting DIC at its best performance level regardless of computing time.Source: §5.3 · pp. 9–10
  • FlowNetS-ft gave the best CNN results on the Star displacement, with a displacement map similar to DIC using an 11-pixel subset; FlowNetC performed worst.DIC with an 11-pixel subset had the lowest MAE, followed by FlowNetS-ft.Source: §5.3 · pp. 10–12
  • At high spatial frequencies FlowNetS-ft was similar to DIC, while at low frequencies DIC was smoother and more accurate.The evaluation region excludes boundary effects; MAE measures the vertical displacement error there.Source: §5.3 · pp. 10–11

6. Tailoring FlowNetS to estimate displacement fields

This section explores network-architecture and training-dataset changes to improve FlowNetS displacement estimation. It moves from higher-resolution outputs to StrainNet variants and a revised speckle dataset.

6.1. Improvement of the network

  • Because FlowNetS predicts quarter-resolution optical flow and upsamples it by a factor of four, the proposed improvement is to output higher-resolution flow directly without interpolation.Source: §6.1 · pp. 12–13

6.1.1. First approach: adding one or two levels

  • Retraining all weights after adding one level worsens average AEE from 0.107 to 0.141 and yields a Star-displacement MAE of 0.3334.Adding another level and retraining all weights does not improve the results.Source: §6.1.1 · p. 13, Figure 7 · p. 14
  • Updating only added levels reduces average AEE by more than 50% versus FowNetS-ft on 363 random test fields, but does not improve Star-displacement accuracy.For the Star field, the mean absolute error is nearly the same as FlowNetS-ft; the column-wise error advantage over DIC (2M + 1 = 11 pixels) occurs only at medium frequencies and reverses for x > 600.Source: §6.1.1 · pp. 13–14, Table 4 · p. 14

6.1.2. Second approach: simplifying the architecture of FlowNetS

  • StrainNet-f and StrainNet-h outperform previous networks on average AEE and MAE, with better results than DIC at high spatial frequencies and comparable results at low frequencies.StrainNet-f uses four down-samplings and four up-samplings; StrainNet-h uses five down-samplings and four up-samplings.Source: §6.1.2 · p. 15, Table 5 · p. 15

6.2.1. Procedure

  • Speckle dataset 2.0 uses 4-to-128-pixel regions, bicubic interpolation, and heteroscedastic noise (v = 0.0342 × s + 0.2679) to address high- and low-frequency needs and limit interpolation bias.Source: §6.2.1 · p. 18

6.2.2. Results obtained with the Star images

  • Training on Speckle dataset 2.0 improves StrainNet-h and StrainNet-f results on Star images; they match DIC at low frequencies and outperform it at high frequencies.Source: §6.2.2 · p. 19
  • With noisy Star images, StrainNet-h and StrainNet-f outperform DIC (2M + 1 = 11 pixels) at both high and low spatial frequencies, with lower high-frequency bias and smoother displacement.Source: §6.2.2 · p. 19

7. Spatial resolution and metrological performance indicator

StrainNet has lower spatial resolution values than first-order DIC, and its column-wise error is also lower than second-order DIC. Its metrological indicator falls between first- and second-order DIC, while pattern-induced bias affects StrainNet but appears less pronounced than for DIC.

7.1. Spatial resolution

  • The spatial-resolution estimate d uses 10% attenuation along the star's symmetry axis; it does not account for DIC's additional off-axis bias.For CNNs, d is estimated numerically or graphically because no transfer function has been identified.Source: §7.1 · p. 20, Figure 17 · p. 22, Figure 19 · p. 23
  • StrainNet-f and -h have smaller spatial resolution d than first-order DIC, and lower mean absolute error per column than second-order DIC.The second-order DIC comparison uses a 21-pixel subset; its d is nearly the same as StrainNet's, but the column-wise error is higher.Source: §7.1 · pp. 20–21, Figure 18 · p. 23

7.2. Metrological performance indicator

  • The metrological indicator α is nearly identical for StrainNet-f and -h and lies between first- and second-order DIC.At nearly equal d to second-order DIC, StrainNet has higher noise; a slight noise-induced bias appears at the highest spatial frequencies.Source: §7.2 · p. 24, Figure 20 · p. 24, Figure 21 · p. 25

7.3. Pattern-induced bias

  • Pattern-induced bias affects StrainNet, but its effect seems less pronounced than for DIC in the two pattern-and-noise experiments.StrainNet-f and -h were trained on Speckle dataset 2.0, whose deformed images used interpolation; the authors propose testing an interpolation-free generator and a larger dataset to investigate the observed bias.Source: §7.3 · pp. 24–26, Figure 22 · p. 26

8. Assessing the generalization capability

The authors test generalization using synthetic speckles from a different generator and real wood-specimen images. StrainNet produces displacement and strain maps in both examples, including maps with relatively high spatial frequencies on the real speckles.

8. Assessing the generalization capability

  • The generalization tests use speckles unlike those generated for Speckle dataset 2.0, including both TexGen synthetic images and real images from a wood compression test.The reference Star images and training-set references share a generator, so the authors include these two examples to test beyond that shared source.Source: §8 · p. 27, §8.1 · p. 27, §8.2 · p. 28

8.1. Example 1: Sample 14 of the DIC Challenge 1.0

  • StrainNet trained on Speckle dataset 2.0 determines the imposed displacement on synthetic Sample 14 speckles generated with TexGen, a different generator.The example uses a 0.1-pixel horizontal sine displacement with increasing frequency; StrainNet maps are similar to DIC using an 11-pixel first-order subset.Source: §8.1 · p. 27, Figure 23 · p. 28

8.2. Example 2: Compression test on a wood specimen

  • On real wood-specimen speckles, StrainNet extracts displacement and strain fields with relatively high spatial frequencies, although the displacement exceeds one pixel over most of the specimen.The authors split the displacement into integer and subpixel components and process it piecewise across 11 horizontal bands; slight straight lines appear at some band borders in the strain maps.Source: §8.2 · pp. 28–29, Figure 25 · p. 30

9. Computing time

The paper reports GPU computing times and points of interest per second for two image examples. StrainNet's inference rate is described as compatible with real-time measurement, while a fair comparison with GPU-based DIC remains for future work.

9. Computing time

  • The paper reports computing time and POI/s for a Star-image pair and a Sample 14 pair, normalizing throughput across techniques and frame sizes.POI/s is the number of measurement points produced per second; Table 7 gives the reported values.Source: §9 · p. 31, Table 7 · p. 31
  • StrainNet-h's POI/s is about ten times lower than StrainNet-f's, while its pre-interpolation resolution is four times lower.The cited GPU-DIC comparison is nearly two orders of magnitude slower, but it used a less powerful GPU than the study's Tesla V100.Source: §9 · p. 31
  • The authors conclude that StrainNet provides pixel-wise displacement maps at a rate compatible with real-time measurements.They note that a fair computing-time comparison with GPU-based DIC remains necessary and must account for the heavy calculations needed to train CNNs.Source: §9 · p. 31

10. Conclusion

The paper presents StrainNet as a CNN for full-field displacement and strain measurement and demonstrates accurate pixelwise subpixel measurement in examples. The authors identify dataset quality, network redesign, and displacements larger than one pixel as areas for further work.

10. Conclusion

  • StrainNet demonstrates the feasibility of accurate pixelwise subpixel measurement over full displacement fields in the examples presented.The paper describes short computing time as the main benefit of CNNs for this measurement task.Source: §10 · p. 31
  • The training dataset influences result quality; the developed dataset produced valuable results on different speckles, including experimental ones, but broader speckle coverage is needed to investigate noise and bias.The authors also suggest using an interpolation-free generator for dataset images.Source: §10 · pp. 31–32
  • Accurate subpixel estimation for displacements larger than one pixel remains an open target, and the authors propose training on larger-displacement images.They also call for a complete network redesign, particularly to simplify the network and reduce training and processing times.Source: §10 · pp. 31–32
Loading 2009.03993v1…