Memory-free scattering imaging based on ultrafast convolutional neural networks

Spatial light modulator is a kind of dynamic component which can modulate the amplitude, phase and polarization state of incident light in real time under the control of external signal. The application of spatial light modulator in the field of Scattering Imaging can not only be used to generate pseudo-thermal light field instead of traditional ground glass, but also can be used as a target object for scattering imaging research. The application of spatial light modulator can realize initiative and maneuverability in the regulation of scattered light field.
Thesis information:

Optical memory effects have been a key basis for macroscopic and microscopic imaging methods in complex scattering media, including cloudy tissues and speckled layers. However, image reconstruction in strongly scattering media without optical memory effect has not been successful. To this end, we demonstrate the ability to reconstruct images in a scattering layer without optical memory effects by developing a multistage convolutional optical neural network (ONN) that integrates multiple parallel cores operating at the speed of light. Based on Fourier optics, parallel, one-step convolution ONN training, features are extracted directly to achieve memory-free image reconstruction, and the Field Of View is expanded by up to 271 times. The device can be dynamically reconfigured for ultra-fast multi-task image reconstruction with a computing power of 1.57peta operations per second (POPS), successfully establishing an ultra-fast and efficient optical machine learning platform for graphics processing.
The following are part of the experimental process and results:
The experimental setup of the convolutional ONN is shown in Figure 1. In the experiment, a reflective intensity modulator SLM0 (FSLM-HD70-A/P, pixel size 8μm, 1920×1080) is used to generate the programmable pixel-level input of the convolutional ONN. The light field modulated by SLM0 is then transmitted through a 4f system to the input plane of the convolution ONN. The 4f system consists of two lenses with Focal Lengths of 100 mm (L1) and 50 mm (L2) to provide 0.5x magnification for objects encoded as SLM0. The input plane is then sent into the three-layer convolutional ONN for further processing. The convolution ONN consists of three phase modulator SLMs (our model FSLM-4K70-P02). The phase SLMs has 4094×2400 pixels and the pixel size is 3.74μm×3.74μm. The first convolution layer is encoded as SLM1, and the distance between the input plane and SLM1 is 10cm. After being modulated by SLM1, the beam is transmitted to SLM2 by reflection from the beam splitters BS2 and BS3. The second convolution layer is encoded as SLM2, and SLM1 and SLM2 are 20cm. The beam modulated by SLM2 is then transmitted to SLM3. A third fully connected layer is coded into SLM3, with a distance of 10 cm between SLM2 and SLM3. The beam modulated by SLM3 is propagated to the output plane of the convolution ONN by reflection from the beam splitter BS4, with a distance of 20cm between SLM3 and the output plane. The acquisition camera (pixel size 4.8μm, 1280×1024) was placed on the output plane of the convolutional ONN to record the inference results. It is important to note that misalignment between the different layers of a convolution ONN can significantly degrade its performance.

FIG. 1 Schematic diagram of experimental setup of three-layer convolutional network ONN. (A) Convolve the parameters and coordinates of each layer of the ONN. (B) Experimental apparatus for convolution ONN. M, mirror; POL, linear polarizer; BS, beam splitter; L, lens; SLM, spatial light modulator.

FIG. 2 The mechanism of memory-free image reconstruction through scattering layer stacking. (A) Scattering diagram through multiple scatters. Each scatterer can be modeled as a thin scatterer layer, with N representing the number of scatterers and d representing the spacing between scatterers. (B) The pattern of speckles produced under different scattering conditions. The figure on the left compares spots with optical memory effects (N = 1, d = 0) and spots without optical memory effects (N > 1, d > 0). The figure on the right shows the angular correlation curves in each case, with virtual lines representing the angular correlation curves for a single flat glass layer. The half height and width of these curves determine the field of view of the image. (C) The concept of memory-free image reconstruction in three-layer convolutional ONN.

FIG. 3 Principle of optical convolutional neural networks. (A) Three-layer convolutional ONN consists of two optical convolutional layers and one optical fully connected layer. (B) The structure of the first convolution layer contains nine different types of nuclei. Each core consists of three structures: vortex phase, random phase, and grating phase. (C) The phase structure of the second convolution layer is divided into 3×3 regions. The phase of each region is constructed by the same process as the first convolution layer, resulting in 81 nuclei. (D) Convolutional ONN performance trained in inference classification tasks based onMNIST and FashionMNIST datasets. The figure above shows the learning curve in each case. The following figure shows the experimental classification results on the convolutional ONN output plane, with the red dashed squares representing the detector region trained by the corresponding number.

FIG. 4 Experimental verification of memory-free image reconstruction. (A) Experimental reconstruction results of a trained convolution ONN with two scattering layers (N = 2). The first line shows the actual image of the object, and the second to fourth lines correspond to the reconstruction results of d = 3 cm, d = 4 cm, and d = 5 cm, respectively. (B) Experimental reconstruction results of trained convolution ONNs with multiple scattering layers (N > 2). The first through fourth lines correspond to cases where N = 2, 3, 4, and 5, respectively. (C) The effect of vortex phase on the convolution ONN is shown. The figure on the right compares the convolution kernel curves with and without vortex phase in scattering d = 5 cm and the reconstruction results. The figure on the left shows that different vortex phase structures can be used to extract different directional edge information of the input object.

Figure 5 demonstrates the dynamic and multitasking performance of trainable convolutional ONNs. (A) Dynamic inference process (S1 and S2) for memory-free image reconstruction using convolutional ONN. The input spot pattern is loaded onto the 60Hz SLM. (B) The convolutional ONN framework of two tasks is demonstrated in S3 for video frame multitasking inference to achieve memory-free image reconstruction. The outline of the third layer of fully connected raster is shown in Figure S17.

The parameters of the amplitude-type spatial light modulator used in this experiment are as follows:

The parameters of the phase-type spatial light modulator used in this experiment are as follows:

Write at the end:
In the field of computational imaging, optical neural networks have been widely used to solve problems in ghost imaging, digital holography, Fourier lamination microscopy and other fields. At the same time, the powerful data fitting ability and optimization solution ability of deep learning also play a huge role in the field of scattering imaging. With the fine modulation and precise control of spatial light modulator, the combination of spatial light modulator and optical neural network will produce more sparks.

Article information:
DOI: 10.1126/sciadv.adn2205