Deep Phase Decoder: Self-calibrating phase microscopy with an untrained deep neural network

Introduction

Quantitative phase microscopy (QPM) enables label-free imaging of transparent samples such as unstained cells and tissues , and non-absorbing micro-elements . QPM can use partially-coherent beams (in lieu of coherent ones ) to increase spatial resolution and light throughput with reduced speckle. Examples include through-focus , interferometric , and angle-scanning microscopes. The common design theme is to image distinct non-linear renderings of phase as intensity from which quantitative phase is numerically recovered. For a given method, the performance and image quality is intrinsically governed by the phase reconstruction step. .

Traditionally, the phase reconstruction problem is addressed by solving an inverse problem by minimizing least-squares loss that is based on the physics of the problem. The approach is fundamental to phase imaging and has been practically employed in various systems. An immediate advantage is that our prior assumptions on the images can be directly integrated through regularization. An example is to constrain the phase image to admit a sparse representation in the wavelet domain . Such regularizers work well for various phase microscopes and improve the reconstruction quality . Another major advantage of the physics-based formulation is the possibility to incorporate algorithmic self-calibration . It involves—in alternation with the phase retrieval step—minimizing the least-squares loss over unknown or partially-known system parameters such as pupil aberrations . The concept hence accounts for the model-mismatch in the imaging pipeline. This provides us with great flexibility and allows phase reconstruction from measurements that are not fully characterized. In designing self-calibrating algorithms, the need for regularization (i.e., prior models for phase) is emphasized since one simultaneously decouples the individual contributions of phase and aberrations to the measured images. However, typical regularization techniques are hand-crafted and require manual tuning of parameters even after the model is constructed.

More recently, deep neural networks (DNNs), typically trained in an end-to-end fashion on large datasets to directly map given intensities back to phase, have been used to obtain efficient phase retrieval algorithms. For phase microscopy, trained DNNs give state-of-the-art performance in holographic , lensless , ptychographic , and through-scattering-media configurations, among others . The results have validated the efficiency of properly trained DNNs to solve non-linear inverse problems and shifted the computational paradigm in QPM towards predominantly data-driven frameworks. However, for deep networks to work well, the proximity of training and experiment settings is critical as the performance is susceptible to variations in sample features, instrumentation, and acquisition parameters . Although improved DNN architectures have been proposed , training-based approaches fundamentally rely on the reconstructed phase image to come from a distribution that is close to the one of the training images and are thus sensitive to misfits.

In this paper, we propose a new QPM algorithm that is based on a deep network, but requires no ground-truth training data. Our approach is inspired by the idea of employing untrained generative DNNs as prior models for images, a concept that is pioneered by the so-called deep image prior . Specifically, Ulyanvov et al. fitted a noisy image via optimizing over the weights of a randomly initialized, over-parameterized autoencoder (i.e., an autoencoder with more weights than the number of image pixels), and observed that early stopping the regularization yields good denoising performance, an effect theoretically explained in . For denoising, regularization through early stopping is critical, since the network can in principle fit the noisy data perfectly. Subsequently, an under-parameterized (i.e., less parameters than the number of image pixels) image-generating network, named the deep decoder, that does not need early stopping or any other further regularization has been proposed . The framework acts as a concise image model that provides a lower-dimensional description of images, akin to the sparse wavelet representations, and thus regularizes through its architecture alone. Unfortunately, a naive application of the method to the problem at hand would not account for practical issues such as drift and sample-induced aberrations , which points to the need of properly incorporating our knowledge about optical physics to achieve self-calibration.

The key contribution of this paper is a DNN-based self-calibrating reconstruction algorithm for QPM that is training-free and recovers quantitative phase from raw images recorded without the explicit knowledge of aberrations. We specify the entire measurement formation as an untrained DNN whose weights are fitted to the recorded images. Leveraging the well-characterized system physics and non-linear forward model, our network combines a fully-connected layer that synthesizes aberrations from Zernike polynomials with the deep decoder that is used to generate phase. The proposed algorithm hence describes both the image and aberrations by a few weight coefficients, and as a consequence enables us to jointly retrieve the phase and individual aberration profile of each measurement without requiring any training data. We term our algorithm the deep phase decoder (DPD) and demonstrate it on a commercial widefield microscope.

Methods

Next, we describe the image formation process in our optical setup (Fig. 2), and then describe our reconstruction approach in more detail. We consider an optically-thin and transparent sample that is placed at the focal plane of the microscope’s objective. The sample’s complex-valued image (i.e., its transmission function) is characterized as

where $\phi$ represents the spatial distribution of phase over 2D coordinates $\mathbf{r}$. The LEDs are place sufficiently far away that their illumination can be modeled as a monochromatic plane wave at the sample plane. Thus, the irradiance of the beam impinging on the camera is given by

where $\mathbf{F}$ is the discrete Fourier transform matrix and $\mathbf{P}_{\rm circ}$ is the ideal and space-invariant exit pupil function, which is a circle with its radius determined by numerical aperture (NA) of the objective and wavelength $\lambda$.

Phase is recovered based on multiple images with some type of data diversity that translates phase information into intensity (e.g. defocus , illumination coding , pupil coding ). Here we adopt a pupil-coding scheme where the wavefront at the exit pupil is differently aberrated for each measurement. The pupil aberration is modeled as a weighted sum of Zernike polynomials, so it is parameterized by a small number of coefficients:

where the Zernike basis $\mathbf{Z}=\left[\mathbf{z}_{1}\,\mathbf{z}_{2}\,\dots\,\mathbf{z}_{M}\right]$ is composed of $M$ orthogonal modes in vectorized form and $\mathbf{c}$ contains the corresponding coefficients of each mode.

The microscope is probed with a known (or pre-calibrated) set of aberrations $\left\{\mathbf{P}\right\}_{n=1}^{N}$, where $N$ is the total number of intensity images. The inverse problem then aims to recover the sample’s transmission function as

This can be solved by gradient-descent (or an accelerated variation), which is closely related to the well-known Gerchberg-Saxton method . After solving for the complex-valued $\mathbf{o}^{\star}$, the phase image is its argument. The conventional phase recovery in (5) does not necessarily impose any regularization on the recovered phase and the aberrations must be known a priori. To address these without needing any training data, we introduce a deep network in the derived formulation.

At the core of our approach, we use a DNN that generates $N$ intensity images. The network, denoted by $G(\mathbf{W})$, reparameterizes the measurement formation in (3) in terms of a weight tensor $\mathbf{W}$ rather than pixels in complex-image space as in (5). The network is untrained and the weights, which are randomly initialized, are optimized by solving the following problem:

The aberration-generating network, $G_{\rm a}(\mathbf{W}^{a})$, relies on the parameterization in (4) represented as a fully-connected layer (i.e. linear combination of Zernike modes) and the matrix $\mathbf{W}^{a}$ contains the Zernike coefficients for all measuruments. In combining the outputs of $G_{\rm p}$ and $G_{\rm a}$ , we reproduce the physical image formation using (1), (3), and (4) in the network’s architecture. The framework is implemented in PyTorch, allowing us to solve (6) using gradient-based algorithms thanks to auto-differentation with respect to $\mathbf{W}=\{\mathbf{W}^{p},\mathbf{W}^{a}\}$. Once the optimal weights $\mathbf{W}^{\star}$ are obtained, the reconstructed phase is given by $G_{p}(\mathbf{W}^{p\,\star})$ where $\mathbf{W}^{p}=\{\mathbf{W}_{0}^{p},\ldots,\mathbf{W}_{d}^{p}\}$.

We now explain some implicit aspects of the our method. First, we see from (6) that $G(\mathbf{W}^{\star})$ replicates the recorded intensities as closely as possible in the least-squares sense. Therefore, regularization of phase is governed by the generative network’s architecture for the images have to lie in its range. Specifically, both $G_{p}$ and $G_{a}$ under-parametrize their corresponding outputs (fewer weights than the number of pixels in generated images), so DPD imposes regularization on phase and aberrations. Moreover, once $G$ is constructed, the strength of regularization is not hand-tuned, as is typically done (such as adjusting the sparsity level for wavelet-based methods). It is also noteworthy that the DPD performs the phase reconstruction from randomly initialized (as $G$ is untrained) aberrations as opposed to other self-calibrating schemes that use theoretical pupils as initialization .

Results

Conclusion

In summary, we derived a new phase imaging algorithm that uses an untrained neural network, and demonstrate it on a phase-from-defocus dataset. Our DPD method, unlike its deep learning counterparts that are supervised, is training-free and does not rely on closely-matching training and experiment conditions. Moreover, our method is self-calibrating, allowing us to directly reconstruct high quality phase without a priori knowledge of the system’s aberrations.

Acknowledgments

The authors thank Gautam Gunjala for help with Zernike polynomials. This work was supported by STROBE: A National Science Foundation Science and Technology Center under Grant No. DMR 1548924, and ONR grant. E. Bostan is supported by the Swiss National Science Foundation (SNSF) under grant P2ELP2 172278. M. Kellman is supported in part by the National Science Foundation (NSF) under grant DGE 1106400.

References