Deep Generative Adversarial Networks for Compressed Sensing Automates MRI

Introduction

Owing to its superb soft tissue contrast, magnetic resonance imaging (MRI) nowadays serves as the major imaging modality in clinical practice. Real-time MRI visualization is of paramount importance for diagnostic and therapeutic guidance for instance in next generation platforms for MR-guided, minimally invasive neurosurgery . However, the scan is quite slow, taking several minutes to acquire clinically acceptable images. This becomes more pronounced for high-resolution and volumetric images. As a result, the acquisition typically undergoes significant undersampling leading reconstruction to a seriously ill-posed linear inverse problem. To render it well-posed, the conventional compressed-sensing (CS) incorporates the prior image information by means of sparsity regularization in a proper transform domain such as Wavelet (WV), or, Total Variation (TV); see e.g., . This however demands running iterative optimization algorithms that are time and resource intensive. This in turn hinders real-time MRI visualization and analysis.

Last but not least, the advocated GANCS scheme tailors inverse imaging tasks appearing in a wide range of applications with budgeted acquisition and reconstruction speed. All in all, relative to the past work this paper’s main contributions are summarized as follows:

Propose GANCS as a data-driven regularization scheme for solving ill-posed linear inverse problems that appear in imaging tasks dealing with (global) aliasing artifacts

First work to apply GAN as a automated (non-iterative) technique for aliasing artifact suppression in MRI with state-of-the-art image diagnostic quality and reconstruction speed

Proposed and evaluated a novel network architecture to achieve better trade-offs between data-consistency (affine projection) and manifold learning

Extensive evaluations on a large contrast-enhanced MRI dataset of pediatric patients, with the reconstructed images rated by expert radiologists

The rest of this paper is organized as follows. Section 2 states the problem. Manifold learning using LSGANs is proposed in Section 3. Section 4 also reports the data evaluations, while the conclusions are drawn in Section 5.

Problem Statement

Manifold Learning via Generative Adversarial Networks

The inverse imaging solution is to find solutions of the intersection between two subspaces defined by acquisition model and image manifold. In order to effectively learn the image manifold from the available (limited number of) training samples we first need to address the following important questions:

How to ensure the trained manifold contains plausible images?

How to ensure the points on the manifold are data consistent, namely $\mathbf{y}\approx\boldsymbol{\Phi}\mathbf{x},~{}\forall\mathbf{x}\in\mathcal{M}$?

The output of G, namely $\check{\mathbf{x}}$, however may not be consistent with the data. To tackle this issue, we add another layer after G that projects onto the feasible set of $\mathbf{y}=\boldsymbol{\Phi}\mathbf{x}$ to arrive at $\hat{\mathbf{x}}=\boldsymbol{\Phi}^{\dagger}\mathbf{y}+(\mathbf{I}-\boldsymbol{\Phi}^{\dagger}\boldsymbol{\Phi})\check{\mathbf{x}}$. Alternatively, we can add a soft LS penalty when training the G network, as will be seen later in (P1). To further ensure that $\hat{\mathbf{x}}$ lies in the intersection of the manifold $\mathcal{M}$ and the space of data consistent images we can use a mutlilayer network that alternates between residual units and data consistency projection as depicted in Fig. 1 (b). We have observed that using only a couple of residual units may improve the performance of G in discarding the aliasing artifacts. The overall network architecture is depicted in Fig. 1 (a), where $\mathbf{P}_{\mathcal{N}}:=(\mathbf{I}-\boldsymbol{\Phi}^{\dagger}\boldsymbol{\Phi})$ signifies projection onto the nullspace of $\boldsymbol{\Phi}$.

The first LS fitting term in (P1.2) is a soft penalty to ensure the input to D network is data consistent. Parameters $\lambda$ and $\eta$ also control the balance between manifold projection, noise suppression and data consistency.

Lemma 1. For the noise-free scenario ($\mathbf{v}=\mathbf{0}$), suppose D and G have infinite capacity. Then, for a given generator network G, i) the optimal discriminator D is $\mathcal{D}^{*}(\boldsymbol{\Theta}_{d};\check{\mathbf{x}})=p_{x}(\check{\mathbf{x}})/(p_{x}(\check{\mathbf{x}})+p_{g}(\check{\mathbf{x}}))$; and ii) $p_{g}=p_{x}$ achieves the equilibrium for the game (P1).

Experiments

Effectiveness of the novel GANCS scheme is assessed in this section via tests for MRI reconstruction. A single-coil MR acquisition model is considered where for $n$-th patient the acquired $k$-space data abides to $y_{i,j}^{(n)}=[\mathcal{F}(\mathbf{X}_{n})]_{i,j}+v_{i,j}^{(n)},~{}~{}(i,j)\in\Omega$. Here, $\mathcal{F}$ is the 2D FT, and the set $\Omega$ indexes the sampled Fourier coefficients. As it is conventionally performed with CS MRI, we select $\Omega$ based on a variable density sampling with radial view ordering that tends to pick low frequency components from the center of $k$-space (see sampling mask in Fig. 4 (left) of the supplementary document). Throughout the test we assume $\Omega$ collects only $20\%$ of the Fourier coefficients, and we choose $\lambda=0.1$.

Dataset. High contrast abdominal image volumes are acquired for $350$ pediatric patients after gadolinium-based contrast enhancement. Each 3D volume includes contains $151$ axial slices of size $200\times 100$. Axial slices used as input images for training a neural network. $300$ patients ($45,300$ images) are considered for training, and $50$ patients ($7,550$ images) for test. All in vivo scans were acquired at the Stanford’s Lucile Packard Children’s Hospital on a 3T MRI scanner (GE MR750) with voxel resolution $1.07\times 1.12\times 2.4$ mm.

Under this setting, the ensuing parts address the following questions:

Q2. How much speed up and quality improvement one can achieve using GANCS relative to conventional CS?

Q3. What MR image features derive the network to learn the manifold and remove the aliasing artifacts?

Q4. How many samples/patients are needed to achieve a reasonable image quality?

To satisfy data consistency term, previous work in the context of image super-resolution used (hard) affine projection after the G network. However, the affine projection drifts $\hat{\mathbf{x}}$ away from the manifold landscape. As argued in Section 3, we instead use a multilayer succession of affine projection and convolutional residual units that project back $\hat{\mathbf{x}}$ onto the manifold. We can repeat this procedure a few times to ensure $\hat{\mathbf{x}}$ lies close to the intersection. This amounts to a soft yet flexible data consistency penalty.

The D network starts from the output of the G network with two channels. It is composed of $8$ convolutional layers. In all the layers except the last one, the convolution is followed by batch normalization, and subsequently ReLU activation. No pooling is used. For the first four layers, number of feature maps is doubled from $8$ to $64$, while at the same time convolution with stride $2$ is used to reduce the image resolution. Kernel size $3\times 3$ is adopted for the first 5 layers, while the last two layers use kernel size $1\times 1$. In the last layer, the convolution output is averaged out to form the decision variable for binary classification. No soft-max is used.

Adam optimizer is used with the momentum parameter $\beta=0.9$, mini-batch size $L_{b}=8$, and initial learning rate $\mu=10^{-5}$ that is halved every $5,000$ iterations. Training is performed with TensorFlow interface on a NVIDIA Titan X Pascal GPU, 12GB RAM. We allow $20$ epochs that takes around $6$ hours for training. The implementation is available online at .

As a figure of merit for image quality assessment we adopt SNR (dB), and SSIM that is defined on a cropped window of size $50\times 50$ from the center of axial slices. In addition, we asked Radiologists Opinion Score (ROS) regarding the diagnostic quality of images. ROS ranges from $1$ (worse) to $5$ (excellent) based on the overall images quality in terms of sharpness/blurriness, and appearance of residual artifacts.

2 Observations and discussion

CS-based scheme achieve higher SNR and SSIM, but they miss the high frequency textures as evidenced by Fig. 2. In addition, they demands iterative algorithms for solving non-smooth optimization programs that takes a few seconds for reconstruction using the optimized BART toolbox . In contrast, the elapsed time for GANCS is only about $10$ msec, which allows reconstructing $100$ frames per second, and thus a suitable choice for real-time MRI visualization tasks. Regarding the convergence, we empirically observe faster and more stable training by imposing more weight on the data consistency which restricts the search space for the network weights.

To assess the perceptual quality of resulting images we also asked the opinion of expert radiologists. We normalize the scores so as the gold-standard images are rated excellent (i.e., ROS=$5$). Statistical ROS is evaluated for the image quality, residual artifacts, and image sharpness. It is shown in the bar plot of Fig. 3, which confirms GANCS almost perceptually pleasing as the gold-standard scan. This demonstrates the superior diagnostic quality of GANCS images relative to the other alternatives.

For the sake of completeness, the evolution of different (empirical) costs associated with the generator cost in (P1.2) over batches are also depicted in Fig. 5. It is observed that the data consistency cost and GAN loss tend to improve alternatively to find the distribution at the intersection of manifold and dats consistency space.

Manifold landscape. We visualize what the discriminator learns by showing the feature maps in different layers as heat-maps superimposed on the original images. Since there are several feature maps per layer, we computed the Principle Component maps for each layer and visualize the first $8$ dominant ones. Fig. indicates that after learning from tens of thousands of generated MRI images by the G network and their gold standards including different organs, is able to detect anatomically valuable features. It is observed that the first layers reveal the edges, while the last layers closer to the classification output reveal more regions of interests that include both anatomy and texture details. This observation is consistent with the way expert radiologist inspect the images based on their diagnosis quality.

Performance with different number of patients We also experimented on the number of patients needed for training and achieving good reconstruction quality in the test phase. It is generally valuable for the clinicians how much training data is needed as in the medical applications, patient data is not easily accessible due to privacy concerns. Fig. 7 plots the normalized RMSE on a test set versus the percentage of patients used for training (normalized by the maximum patient number $350$). Note, the variance differences for different training may be due to the training with fewer samples has better convergence, since we are using the same epoch numbers for all the training cases. More detailed study is the subject of our ongoing research.

Conclusions and Future Work

This paper caters a novel CS framework that leverages the historical data for faster and more diagnosis-valuable image reconstruction from highly undersampled observations. A low-dimensional manifold is learned where the images are not only sharp and high contrast, but also consistent with both the real MRI data and the acquisition model. To this end, a neural network based on LSGANs is trained that consists of a generator network to map a readily obtainable undersmapled image to the gold-standard one. Experiments based on a large cohort of abdominal MR data, and the evaluations performed by expert radiologists confirm that the GANCS retrieves images with better diagnostic quality in a real-time manner (about $10$ msec, more than $100$ times faster than state-of-the-art CS MRI toolbox). This achieves a significant speed-up and diagnostic accuracy relative to standard CS MRI. Last but not least, the scope of the novel GANCS goes beyond the MR reconstruction, and tailors other image restoration tasks dealing with aliasing artifacts. There are still important question to address such as using 3D spatial correlations for improved quality imaging, robustifying against patients with abnormalities, and variations in the acquisition model for instance as a result of different sampling strategies.

References