Deep speckle correlation: a deep learning approach towards scalable imaging through scattering media

• Scattering makes deterministic characterization difficult; transmission matrices grow quadratically with transferred pixel number and require recalibration after decorrelation, while prior diffuser networks were tested on the same diffuser.The limited range of the memory effect also constrains the SBP of approaches that rely on it.Source: §1 · pp. 3–5
• The one-to-all CNN learns across diffusers with different microstructures but the same macroscopic parameter, rather than learning one diffuser’s transmission matrix.The intended model captures statistical variation across a diffuser class and invariant information in correlated or decorrelated speckles.Source: §1 · p. 4
• In the shift-variant experiment, a defocused diffuser limited the isoplanatic region to about one speckle size while objects spanned roughly 300×300 speckle sizes, and the CNN interpreted nonlinear intensity-only measurements.The multilayer CNN learns to interpret phaseless measurements without the interferometric measurements often used for transmission-matrix calibration.Source: §1 · p. 5
• At approximately 256×256-pixel SBP using up to 2400 training pairs, the CNN predicted objects through unseen diffusers and generalized to new object types.The authors describe the approach as improving information throughput and adaptability by increasing SBP and robustness to speckle decorrelations.Source: §1 · pp. 5–6
• CNN activation-map correlations and speckle analysis indicate that the model learns invariant features across patterns from different diffusers, including seemingly decorrelated speckles.The authors attribute the model’s ability to exploit these hidden invariant features to its representation power.Source: §1 · pp. 5–6

2.1 Experimental setup

• A defocused 220-grit glass diffuser produces spatially variant scattering, with measurements showing an isoplanatic range of about one speckle size.A scanned 3×3-pixel point object decorrelates rapidly beyond that range, where the correlation coefficient plateaus near 0.3; the speckle size is about 16µm. The SLM object and camera image are relayed through a 4F system, with an iris controlling speckle size.Source: §2.1 · p. 6, Figure 2 · p. 7

2.2 Data acquisition

• Data from up to 4 of 9 diffusers train the CNN, while the other 5 diffusers are reserved for testing; Quickdraw objects are also used only for testing.The four test groups assess seen objects through unseen diffusers, unseen same-type objects through unseen diffusers, new object types through unseen diffusers, and unseen objects through a seen training diffuser. Training uses 600 objects and up to 2400 speckle images; data collection spans about 8 weeks.Source: §2.2 · pp. 8–9

2.3 Data preprocessing

• Input speckles and target images are downsampled by 2×2 binning, and speckles are normalized to the 0–1 range.Binning can reduce speckle contrast and mix object and background pixels, but the authors report little effect on final results. Binary targets threshold nonzero object pixels to 1, while grayscale targets retain normalized binned intensities; the background channel is the complement of the object.Source: §2.3 · p. 9

2.4 CNN implementation

• The CNN uses a modified U-Net with dense blocks and skip connections, and predicts complementary object and background channels for binary detection or grayscale reconstruction.Averaged cross-entropy is used as the loss to promote sparse predictions; grayscale outputs are set to the SLM’s 8-bit level.Source: §2.4 · pp. 10–11

• The reported results show binary images, while grayscale predictions had similar performance; the CNN is designed for sparse objects, and nonsparse imaging is more challenging.The paper leaves imaging nonsparse objects for future work.Source: §3 · p. 11
• For seen objects through unseen diffusers, the CNN made high-quality predictions despite different speckle patterns, with slight pixel-wise prediction variations.The task requires per-pixel input-output learning that adapts to statistical variations across diffusers.Source: §3 · p. 11
• The CNN made high-quality binary predictions for unseen objects of the same class as its training data through unseen diffusers.The tested object classes included handwritten digits and letters.Source: §3 · p. 12
• Our four-diffuser CNN matched state-of-the-art Task 4 performance at average PCC 0.626 [29], while a single-diffuser CNN could not reliably generalize to other diffusers.Task 4 used unseen objects on the same diffuser for training and testing; the generalization test used a different unseen diffuser.Source: §3 · p. 12
• JI and PCC improved with more training diffusers and larger datasets; 4 diffusers with 800 training examples slightly outperformed 2 diffusers with 1600.The PCC comparison is reported for the seen-objects-through-unseen-diffusers task; the cited table identifies the metric but supplies no cell values in the provided passage.Source: §3 · pp. 12–13, Table 1 · p. 13
• For unseen objects of new types through unseen diffusers, predictions remained high-quality, but JI was lower than for seen objects and varied by object type.The evaluation covered six object types using a CNN trained with four diffusers and 2400 training examples; the passage suggests broader object-type coverage in training may improve results.Source: §3 · pp. 12–13

4 analysis

• The analysis seeks statistical invariant features across speckles from different diffusers as a basis for a statistical mapping learned by the CNN.The paper frames this as the deep-learning principle of learning statistical invariants across a large dataset.Source: §4 · pp. 13–14
• Activation maps for the same object through different testing diffusers become more similar through the encoder-decoder, with corresponding-layer PCC generally increasing with CNN depth.The trend is similar across objects and is shown for all combinations of the five testing diffusers.Source: §4 · p. 14
• Task 4 speckles have mean PCC 0.307, Task 1 has mean PCC 0.221, and Tasks 2, 3, and 5 have correlation around 0.207.These values quantify decorrelation for different objects through the same diffuser, the same object through different diffusers, and different objects through different diffusers, respectively.Source: §4 · p. 14
• Same-object speckle intensity cross-correlations across training and testing diffusers resemble autocorrelation references despite low PCC, while similar objects show distinct patterns.The authors interpret these patterns as learnable, physically meaningful invariance that the CNN can exploit even when PCC classifies speckles as decorrelated.Source: §4 · p. 14
• Learned invariance in speckle correlation patterns may support scalability to decorrelations and generalize to measurements from unseen objects through unseen diffusers.The authors base this suggestion in part on their results for unseen objects and unseen diffusers.Source: §4 · p. 15

5 conclusion and discussion

• The “one-to-all” framework fits one model to scattering media within the same class, improving resilience to speckle decorrelations and space-bandwidth-product.The authors contrast this with traditional “one-to-one” techniques, where a model works only for one fixed medium.Source: §5 · p. 15
• The technique could be adapted to train, classify, and image through biological samples characterized by macroscopic scattering parameters.Examples include absorption and scattering coefficients and transport mean-free-path.Source: §5 · p. 15
• The demonstrated thin-diffuser imaging resembles aberrations from a single scattering layer [28], while volumetric multiple scattering remains a future challenge.The approach may enable real-time aberration compensation without expensive hardware, expanded fields of view, and improved tolerance to changing aberrations; adapting it to volumetric multiple scattering is future work [30] [48] [49].Source: §5 · p. 15

Funding

• The work was supported by the National Science Foundation (NSF) under award 1711156.Source: Funding · p. 15

Test on seen objects through unseen diffusers

• The CNN trained with four diffusers made consistently reliable predictions for the same trained objects through five previously unseen diffusers, despite visibly different speckle patterns.Source: Figure 4 · p. 21

Generalization to unseen objects

• The CNN predicted unused handwritten digits and letters through unseen diffusers, and was also tested on QuickDraw objects from a class never used during training.For the new object type, performance was quantified by mean Jaccard index across objects and five testing diffusers; error bars show the standard deviation for each type.Source: Figure 5 · p. 22, Figure 6 · p. 23, Figure 9 · p. 26

Single-diffuser comparison

• A CNN trained on one diffuser predicted unseen objects well through that same diffuser but failed when the objects were viewed through a different unseen diffuser.Source: Figure 7 · p. 24

Activation-map analysis

• Activation maps from the same object’s speckles through different testing diffusers became increasingly similar in deeper layers, with Pearson correlation generally rising across layers for four objects.Source: Figure 10 · p. 26

Robustness to speckle decorrelation

• Average speckle decorrelation was 0.307 for different objects on the same diffuser, 0.221 for the same object across diffusers, and 0.207 for different objects across diffusers.Patterns from the same object through different diffusers showed invariant structure, a possible source of weak correlation information used by the CNN.Source: Figure 11 · p. 27