Attention U-Net: Learning Where to Look for the Pancreas

Introduction

Automated medical image segmentation has been extensively studied in the image analysis community due to the fact that manual, dense labelling of large amounts of medical images is a tedious and error-prone task. Accurate and reliable solutions are desired to increase clinical work flow efficiency and support decision making through fast and automatic extraction of quantitative measurements.

With the advent of convolutional neural networks (CNNs), near-radiologist level performance can be achieved in automated medical image analysis tasks including cardiac MR segmentation and cancerous lung nodule detection . High representation power, fast inference, and filter sharing properties have made CNNs the de facto standard for image segmentation. Fully convolutional networks (FCNs) and the U-Net are two commonly used architectures. Despite their good representational power, these architectures rely on multi-stage cascaded CNNs when the target organs show large inter-patient variation in terms of shape and size. Cascaded frameworks extract a region of interest (ROI) and make dense predictions on that particular ROI. The application areas include cardiac MRI , cardiac CT , abdominal CT segmentation, and lung CT nodule detection . However, this approach leads to excessive and redundant use of computational resources and model parameters; for instance, similar low-level features are repeatedly extracted by all models within the cascade. To address this general problem, we propose a simple and yet effective solution, namely attention gates (AGs). CNN models with AGs can be trained from scratch in a standard way similar to the training of a FCN model, and AGs automatically learn to focus on target structures without additional supervision. At test time, these gates generate soft region proposals implicitly on-the-fly and highlight salient features useful for a specific task. Moreover, they do not introduce significant computational overhead and do not require a large number of model parameters as in the case of multi-model frameworks. In return, the proposed AGs improve model sensitivity and accuracy for dense label predictions by suppressing feature activations in irrelevant regions. In this way, the necessity of using an external organ localisation model can be eliminated while maintaining the high prediction accuracy. Similar attention mechanisms have been proposed for natural image classification and captioning to perform adaptive feature pooling, where model predictions are conditioned only on a subset of selected image regions. In this paper, we generalise this design and propose image-grid based gating that allows attention coefficients to be specific to local regions. Moreover, our approach can be used for attention-based dense predictions.

We demonstrate the implementation of AG in a standard U-Net architecture (Attention U-Net) and apply it to medical images. We choose the challenging CT pancreas segmentation problem to provide experimental evidence for our proposed contributions. This problem constitutes a difficult task due to low tissue contrast and large variability in organ shape and size. We evaluate our implementation on two commonly used benchmarks: TCIA Pancreas $CT$-$82$ and multi-class abdominal $CT$-$150$. The results show that AGs consistenly improve prediction accuracy across different datasets and training sizes while achieving state-of-the-art performance without requiring multiple CNN models.

CT Pancreas Segmentation: Early work on pancreas segmentation from abdominal CT used statistical shape models or multi-atlas techniques . In particular, atlas approaches benefit from implicit shape constraints enforced by propagation of manual annotations. However, in public benchmarks such as the TCIA dataset , Dice similarity coefficients (DSC) for atlas-based frameworks ranges from $69.6\%$ to $73.9\%$ . In a classification based framework is proposed to remove the dependency of atlas to image registration. Recently, cascaded multi-stage CNN models have been proposed to address the problem. Here, an initial coarse-level model (e.g. U-Net or Regression Forest) is used to obtain a ROI and then a cropped ROI is used for segmentation refinement by a second model. Similarly, combinations of 2D-FCN and recurrent neural network (RNN) models are utilised in to exploit dependencies between adjacent axial slices. These approaches achieve state-of-the-art performance in the TCIA benchmark ($81.2\%-82.4\%$ DSC). Without using a cascaded framework, the performance drops between $2.0\%$ and $4.4\%$. Recent work proposed an iterative two-stage model that recursively updates local and global predictions, and both models are trained end-to-end. Besides standard FCNs, dense connections and sparse convolutions have been applied to the CT pancreas segmentation problem. Dense connections and sparse kernels reduce computational complexity by requiring less number of non-zero parameters.

Attention Gates: AGs are commonly used in natural image analysis, knowledge graphs, and language processing (NLP) for image captioning , machine translation , and classification tasks. Initial work has explored attention-maps by interpreting gradient of output class scores with respect to the input image. Trainable attention, on the other hand, is enforced by design and categorised as hard- and soft-attention. Hard attention , e.g. iterative region proposal and cropping, is often non-differentiable and relies on reinforcement learning for parameter updates, which makes model training more difficult. Recursive hard-attention is used in to detect anomalies in chest X-ray scans. Contrarily, soft attention is probabilistic and utilises standard back-propagation without need for Monte Carlo sampling. For instance, additive soft attention is used in sentence-to-sentence translation and more recently applied to image classification . In , channel-wise attention is used to highlight important feature dimensions, which was the top-performer in the ILSVRC 2017 image classification challenge. Self-attention techniques have been proposed to remove the dependency on external gating information. For instance, non-local self attention is used in to capture long range dependencies. In self-attention is used to perform class-specific pooling, which results in more accurate and robust image classification performance.

2 Contributions

In this paper, we propose a novel self-attention gating module that can be utilised in CNN based standard image analysis models for dense label predictions. Moreover, we explore the benefit of AGs to medical image analysis, in particular, in the context of image segmentation. The contributions of this work can be summarised as follows:

We take the attention approach proposed in a step further by proposing grid-based gating that allows attention coefficients to be more specific to local regions. This improves performance compared to gating based on a global feature vector. Moreover, our approach can be used for dense predictions since we do not perform adaptive pooling.

We propose one of the first use cases of soft-attention technique in a feed-forward CNN model applied to a medical imaging task. The proposed attention gates can replace hard-attention approaches used in image classification and external organ localisation models in image segmentation frameworks .

An extension to the standard U-Net model is proposed to improve model sensitivity to foreground pixels without requiring complicated heuristics. Accuracy improvements over U-Net are experimentally observed to be consistent across different imaging datasets.

Methodology

Fully Convolutional Network (FCN): Convolutional neural networks (CNNs) outperform traditional approaches in medical image analysis on public benchmark datasets while being an order of magnitude faster than, e.g., graph-cut and multi-atlas segmentation techniques . This is mainly attributed to the fact that (I) domain specific image features are learnt using stochastic gradient descent (SGD) optimisation, (II) learnt kernels are shared across all pixels, and (III) image convolution operations exploit the structural information in medical images well. In particular, fully convolutional networks (FCN) such as U-Net , DeepMedic and holistically nested networks have been shown to achieve robust and accurate performance in various tasks including cardiac MR , brain tumours and abdominal CT image segmentation tasks.

Attention Gates for Image Analysis: To capture a sufficiently large receptive field and thus, semantic contextual information, the feature-map grid is gradually downsampled in standard CNN architectures. In this way, features on the coarse spatial grid level model location and relationship between tissues at global scale. However, it remains difficult to reduce false-positive predictions for small objects that show large shape variability. In order to improve the accuracy, current segmentation frameworks rely on additional preceding object localisation models to simplify the task into separate localisation and subsequent segmentation steps. Here, we demonstrate that the same objective can be achieved by integrating attention gates (AGs) in a standard CNN model. This does not require the training of multiple models and a large number of extra model parameters. In contrast to the localisation model in multi-stage CNNs, AGs progressively suppress feature responses in irrelevant background regions without the requirement to crop a ROI between networks.

We use additive attention to obtain the gating coefficient. Although this is computationally more expensive, it has experimentally shown to achieve higher accuracy than multiplicative attention . Additive attention is formulated as follows:

Attention Gates in U-Net Model: The proposed AGs are incorporated into the standard U-Net architecture to highlight salient features that are passed through the skip connections, see Figure 1. Information extracted from coarse scale is used in gating to disambiguate irrelevant and noisy responses in skip connections. This is performed right before the concatenation operation to merge only relevant activations. Additionally, AGs filter the neuron activations during the forward pass as well as during the backward pass. Gradients originating from background regions are down weighted during the backward pass. This allows model parameters in shallower layers to be updated mostly based on spatial regions that are relevant to a given task. The update rule for convolution parameters in layer $l-1$ can be formulated as follows:

The first gradient term on the right-hand side is scaled with $\alpha_{i}^{l}$. In case of multi-dimensional AGs, $\alpha_{i}^{l}$ corresponds to a vector at each grid scale. In each sub-AG, complementary information is extracted and fused to define the output of skip connection. To reduce the number of trainable parameters and computational complexity of AGs, the linear transformations are performed without any spatial support (1x1x1 convolutions) and input feature-maps are downsampled to the resolution of gating signal, similar to non-local blocks . The corresponding linear transformations decouple the feature-maps and map them to lower dimensional space for the gating operation. As suggested in , low-level feature-maps, i.e. the first skip connections, are not used in the gating function since they do not represent the input data in a high dimensional space. We use deep-supervision to force the intermediate feature-maps to be semantically discriminative at each image scale. This helps to ensure that attention units, at different scales, have an ability to influence the responses to a large range of image foreground content. We therefore prevent dense predictions from being reconstructed from small subsets of skip connections.

Experiments and Results

The proposed AG model is modular and independent of application type; as such it can be easily adapted for classification and regression tasks. To demonstrate its applicability to image segmentation, we evaluate the Attention U-Net model on a challenging abdominal CT multi-label segmentation problem. In particular, pancreas boundary delineation is a difficult task due to shape-variability and poor tissue contrast. Our model is compared against the standard 3D U-Net in terms of segmentation performance, model capacity, computation time, and memory requirements.

Evaluation Datasets: For the experiments, two different CT abdominal datasets are used: (I) 150 abdominal 3D CT scans acquired from patients diagnosed with gastric cancer ($CT$-$150$). In all images, the pancreas, liver, and spleen boundaries were semi-automatically delineated by three trained researchers and manually verified by a clinician. The same dataset is used in to benchmark the U-Net model in pancreas segmentation. (II) The second datasethttps://wiki.cancerimagingarchive.net/display/Public/Pancreas-CT ($CT$-$82$) consists of 82 contrast enhanced 3D CT scans with pancreas manual annotations performed slice-by-slice. This dataset (NIH-TCIA) is publicly available and commonly used to benchmark CT pancreas segmentation frameworks. The images from both datasets are downsampled to isotropic $2.00$ mm resolution due to the large image size and hardware memory limitations.

Implementation Details: In contrast to the state-of-the-art CNN segmentation frameworks , we propose a 3D-model to capture sufficient semantic context. Gradient updates are computed using small batch sizes of $2$ to $4$ samples. For larger networks, gradient averaging is used over multiple forward and backward passes. All models are trained using the Adam optimiser , batch-normalisation, deep-supervision , and standard data-augmentation techniques (affine transformations, axial flips, random crops). Intensity values are linearly scaled to obtain a normal distribution $N(0,1)$. The models are trained using the Sorensen-Dice loss defined over all semantic classes, which is experimentally shown to be less sensitive to class imbalance. Gating parameters are initialised so that attention gates pass through feature vectors at all spatial locations. Moreover, we do not require multiple training stages as in hard-attention based approaches therefore simplifying the training procedure. Our implementation using PyTorch is publicly availablehttps://github.com/ozan-oktay/Attention-Gated-Networks.

Attention Map Analysis: The attention coefficients obtained from test images are visualised with respect to training epochs (see Figure 4). We commonly observe that AGs initially have a uniform distribution and pass features at all locations. This is gradually updated and localised towards the targeted organ boundaries. Additionally, at coarser scales AGs provide a rough outline of organs which are gradually refined at finer resolutions. Moreover, by training multiple AGs at each image scale, we observe that each AG learns to focus on a particular subset of organs.

Segmentation Experiments: The proposed Attention U-Net model is benchmarked against the standard U-Net on multi-class abdominal CT segmentation. We use $CT$-$150$ dataset for both training ($120$) and testing ($30$). The corresponding Dice scores (DSC) and surface distances (S2S) are given in Table 1. The results on pancreas predictions demonstrate that attention gates (AGs) increase recall values ($p=.005$) by improving the model’s expression power as it relies on AGs to localise foreground pixels. The difference between predictions obtained with these two models are qualitatively compared in Figure 2. In the second experiment, the same models are trained with fewer training images ($30$) to show that the performance improvement is consistent and significant for different sizes of training data ($p=.01$). For both approaches, we observe a performance drop on spleen DSC as the training size is reduced. The drop is less significant with the proposed framework. For kidney segmentation, the models achieve similar accuracy since the tissue contrast is higher.

In Table 1, we also report the number of trainable parameters for both models. We observe that by adding 8% extra capacity to the standard U-Net, the performance can be improved by 2-3% in terms of DSC. For a fair comparison, we also train higher capacity U-Net models and compare against the proposed model with smaller network size. The results shown in Table 2 demonstrate that the addition of AGs contributes more than simply increasing model capacity (uniformly) across all layers of the network ($p=.007$). Therefore, additional capacity should be used for AGs to localise tissues, in cases when AGs are used to reduce the redundancy of training multiple, individual models.

Comparison to State-of-the-Art: The proposed architecture is evaluated on the public TCIA CT Pancreas benchmark to compare its performance with state-of-the-art methods. Initially, the models trained on $CT$-$150$ dataset are directly applied to $CT$-$82$ dataset to observe the applicability of the two models on different datasets. The corresponding results (BFT) are given in Table 3. U-Net model outperforms traditional atlas techniques although it was trained on a disjoint dataset. Moreover, the attention model performs consistently better in pancreas segmentation across different datasets. These models are later fine-tuned (AFT) on a subset of TCIA dataset ($61$ train, $21$ test). The output nodes corresponding to spleen and kidney are excluded from the output softmax computation, and the gradient updates are computed only for the background and pancreas labels. The results in Table 3 and 4 show improved performance compared to concatenated multi-model CNN approaches due to additional training data and richer semantic information (e.g. spleen labels). Additionally, we trained the two models from scratch (SCR) with $61$ training images randomly selected from the $CT$-$82$ dataset. Similar to the results on $CT$-$150$ dataset, AGs improve the segmentation accuracy and lower the surface distances ($p=.03$) due to increased recall rate of pancreas pixels ($p=.09$).

Results from state-of-the-art CT pancreas segmentation models are summarised in Table 4 for comparison purposes. Since the models are trained on the same training dataset, this comparison gives an insight on how the attention model compares to the relevant literature. It is important to note that, post-processing (e.g. conditional random field) is not utilised in our framework as the experiments mainly focus on quantification of performance improvement brought by AGs in an isolated setting. Similarly, residual and dense connections can be used as in in conjunction with AGs to improve the segmentation results. In that regard, our 3D Attention U-Net model performs similar to the state-of-the-art, despite the input images are downsampled to lower resolution. More importantly, our approach significantly improves the results compared to single-model based segmentation frameworks (see Table 4). We do not require multiple CNN models to localise and segment object boundaries. Lastly, we performed $5$-fold cross-validation on the $CT$-$82$ dataset using the Attention U-Net for a better comparison, which achieved $81.48\pm 6.23$ DSC for pancreas labels.

Discussion and Conclusion

In this paper, we presented a novel attention gate model applied to medical image segmentation. Our approach eliminates the necessity of applying an external object localisation model. The proposed approach is generic and modular as such it can be easily applied to image classification and regression problems as in the examples of natural image analysis and machine translation. Experimental results demonstrate that the proposed AGs are highly beneficial for tissue/organ identification and localisation. This is particularly true for variable small size organs such as the pancreas, and similar behaviour is expected for global classification tasks.

Training behaviour of the AGs can benefit from transfer learning and multi-stage training schemes. For instance, pre-trained U-Net weights can be used to initialise the attention network, and gates can be trained accordingly in the fine-tuning stage. Similarly, there is a vast body of literature in machine learning exploring different gating architectures. For example, highway networks make use of residual connections around the gate block to allow better gradient backpropagation and slightly softer attention mechanisms. Although our experiments with residual connections have not provided any significant performance improvement, future research will focus on this aspect to obtain a better training behaviour. Lastly, we note that with the advent of improved GPU computation power and memory, larger capacity 3D models can be trained with larger batch sizes without the need for image downsampling. In this way, we would not need to utilise ad-hoc post-processing techniques to further improve the state-of-the-art results. Similarly, the performance of Attention U-Net can be further enhanced by utilising fine resolution input batches without additional heuristics. Lastly, we would like to thank to Salim Arslan and Dan Busbridge for their helpful comments on this work.

References