Boundary Guided Semantic Learning for Real-time COVID-19 Lung Infection Segmentation System

I Introduction

The outbreak of coronavirus disease (COVID-19) has created a global, disruptive, long-lasting, and unprecedented public health crisis. More than 452.22 million people have been reported to be infected by the COVID-19 globally and 6.02 million people have died, according to Reuters statistics. Moreover, the virus is constantly mutating (e.g., delta, omicron), and many new virus variants have appeared. To complement the Reverse Transcription-Polymerase Chain Reaction (RT-PCR) testing, chest X-ray (CXR) and computed tomography (CT) have been widely used as the auxiliary screening tools for COVID-19 infection, which can be further used to classify the confirmed cases and formulate corresponding treatment methods. In this paper, we utilize chest CT images as processing data to design an algorithm for automatically segmenting lung infections in COVID-19 cases.

Thanks to the powerful feature representation capabilities of deep learning, it has been widely used to address the computer vision task, such as enhancement , detection , super-resolution , and medical image processing including lung nodules segmentation , brain and brain-tumor segmentation , polyp segmentation , brain image synthesis , retinal image non-uniform illumination removal etc. For each different task, due to the differences in imaging equipment and disease characteristics, different segmentation models need to be designed separately. As far as COVID-19 diagnosis is concerned, a number of algorithms based on deep learning for CXR and CT images have been proposed . Among them, CT image is more widely used in clinical practice due to its higher sensitivity and clarity, such as COVID-19 classification and segmentation task. The COVID-19 lung infection segmentation from CT images aims to locate the infected regions and generate a pixel-wise segmentation mask. As can be seen from the examples given in Figure 1, this is a very challenging task, mainly manifested in:

On the one hand, missing and incomplete detection of infection regions are common problems in the existing methods. By observing all images in Figure 1, infection regions often not appear concentrated, but scattered in multiple locations of the image, leading to a missing and incomplete detection. Meanwhile, when the patients’ lungs have overwhelming infection, incomplete detection may be encountered, such as the third row of Figure 1. In fact, these scattered infection regions or internal larger range infection regions are still correlated in semantic attributes. Regarding this issue, we try to utilize the wealth of semantic information available at high-level encoder features to guide the feature learning in the decoder stage. Therefore, we propose a Dual-branch Semantic Enhancement (DSE) module to aggregate high-level encoder features, thereby modeling the global relation of different regions or different parts of regions. In addition, structure of human body causes backgrounds of the lung CT image (non-infected regions) to be complex, and thus the background interference is detrimental to precise targeting. Our DSE module can also benefit for suppressing complex backgrounds through semantic and category attributes.

On the other hand, the detailed boundaries of infected areas are not sharp and clear enough. As a tool of assistant COVID-19 diagnosis, boundary information plays an important role. The smooth boundaries may not have a positive effect on doctors’ diagnosis , such as the third row of Figure 1. As we all know, the low-level features have higher spatial resolution and more detailed boundaries, which can supplement the decoding process to achieve boundary guidance. However, directly transmitting the coarse low-level features may cause additional redundancy interference. Considering better suppression of unimportant features, we propose a Mirror-symmetric Boundary Guidance (MBG) module that can purify the features learned from the encoder and obtain more discriminative infection-related features.

In addition to the technical issues involved in model design, as described in recent and important review papers , some common-sense pitfalls and biases are waiting to be solved, including the data used for model development, the evaluation and reproducibility of designed model. We also step up efforts to these three areas, specifically as follows:

(1) Data. As described in , using a public dataset alone without additional new data may lead to community-wide overfitting on this dataset. Therefore, in order to ensure the generalization ability of the proposed model, we merge the two publicly available CT-based segmentation datasets to obtain 1018 CT images, which are further divided into 718 training images and 300 testing images. In this way, it also can avoid selection bias caused by COVID-19 images come from the same place. Furthermore, data augmentation is used during the training phase to alleviate the data shortage problem.

(2) Evaluation. For more comprehensive quantitative evaluation, we use six metrics, including Dice Similarity Coefficient, Sensitivity, Precision, Structure Measure, Enhance-alignment Measure, and Mean Absolute Error. These indicators are measured from multiple aspects such as segmentation accuracy, completeness, structural representation ability, etc. It is evident that our model achieves competitive performance across different metrics, indicating that the results produced by our model are validated and reliable.

(3) Reproducibility. In order to describe the details of our network structure succinctly and clearly, we provide a table of convolution parameters to list the details of each convolution block in the method introduction. In addition, as suggested in , the image resizing, cropping, and normalization are used before model input, and more training details (e.g., number of epochs, learning rate, and the optimizer) are also provided.

In summary, an end-to-end COVID-19 infection segmentation model is proposed, which focuses on semantic relation modeling and boundary details guidance. The good portability of our proposed method enable it can be easily transplanted to the existing intelligent diagnosis system such as the two intelligent diagnosis systems shown in Figure 1. On this basis, the quality of the infection segmentation results can be improved by updating the algorithm model without changing the hardware, thereby realizing the integrated application across different fields. The major contributions are summarized as follows.

We propose an end-to-end boundary guided semantic learning method for accurate and real-time COVID-19 lung infection segmentation, which can be easily transplanted to existing COVID-19 intelligent diagnosis system for algorithmic model upgrades. Our work belongs to the research of the underlying algorithm framework in the field of consumer electronics.

We design a DSE module to aggregate high-level features in a complementary dual-branch strategy, including the top-level semantic preservation and the progressive semantic integration, thereby modeling the semantic relations and forcing the generation of complete infection area segmentation.

We propose a MBG module to introduce the low-level boundary information in the feature decoding stage with a mirror-symmetric structure, which can ensure the complementarity and sufficiency of feature learning.

Comparing the proposed method with eleven state-of-the-art approaches, our method achieves the superior performance under six evaluation metrics. Besides, the model has a real-time inference speed of 44 FPS.

II Related Work

Convolutional neural networks became a popular machine learning algorithm for automated medical image analysis due to the breakthrough of deep learning for computer vision. Most of the medical image segmentation methods are based on U-Net structure or its modifications, such as UNet++ , Attention_UNet , ResNet34_UNet .

Currently, thin-slice chest scans have become indispensable in thoracic radiology, but the huge amount of data also substantially increases the load of radiologists. As a result, automated chest CT image segmentation has become a popular auxiliary technique for lung disease diagnosis. For example, Shen et al. designed an automated lung segmentation system to boost the segmentation accuracy by utilizing the bidirectional chain code. Compared to classical machine learning methods, the deep learning algorithms can extract features from the perspective of semantic relation, which helps to segment nodule regions accurately from the similar visual background. Wang et al. proposed a central focused CNN to segment lung nodules in chest CT images, and designed a weighted sampling to facilitate the model training. Jin et al. utilized GAN-synthesized data to improve the training of a discriminative model for pathological lung segmentation.

II-B COVID-19 Lung Infection Segmentation

So far, many COVID-19 lung infection segmentation methods from CT Images based on deep learning have been proposed, and promising performance has been obtained . Zhou et al. integrated the spatial and channel attention mechanisms to automatically segment the infection area. Fan et al. presented the parallel partial decoder, reverse attention, and edge-attention specifically for COVID-19 to improve the performance, and also provided a semi-supervised framework to alleviate the shortage of labeled data. Wang et al. proposed a noise-robust learning framework based on self-ensembling of convolutional neural network. Paluru et al. developed an anamorphic depth embedding-based lightweight convolutional neural network to segment anomalies in COVID-19 chest CT slices.

Wu et al. proposed a novel joint classification and segmentation system to perform real-time and explainable chest CT diagnosis. Because the high intra-class variation and inter-class indistinction in COVID-19 infection appearance, Wang et al. employed the autofocus and panorama modules for integrating the peer- and cross-level contexts. Yan et al. introduced a feature variation block which adaptively adjusts the global properties of the features for segmenting COVID-19 infection. Kitrungrotsakul et al. proposed an interactive attention refinement network and an automatic seed point generation technique for the training.

III Proposed Method

As illustrated in Figure 2, an end-to-end network named BSNet is proposed for COVID-19 lung infection segmentation in CT images, following an encoder-decoder architecture. The overall framework can be divided into the encoder stage and decoder stage. Specifically, our backbone encoder extractor consists of five sequentially-stacked convolutional blocks, thereby obtaining multi-level features $\{X^{1},X^{2},X^{3},X^{4},X^{5}\}$. For clarity, we list the details of each convolution block in Table I, where Res2Net is used as our backbone feature extractor, and RFB is a receptive field block used to enhance features learned from Res2Net.

Considering the important role of semantic relations and boundary constraints on the segmentation task, we design the Dual-branch Semantic Enhancement (DSE) module and Mirror-symmetric Boundary Guidance (MBG) module to highlight the global semantics and sharp boundaries during the decoding process. The high-level features from the last three convolutional blocks contain abundant semantic information and the corresponding low-level features from the first two convolutional blocks contain more detailed boundary information due to higher spatial resolution. In order to comprehensively utilize the rich semantic information of the last three convolutional blocks, we design the DSE module in a complementary two-branch way to generate a global semantic mask, and use it for semantic refinement. It is well known that clear boundaries are essential for diagnosis, so making full use of the boundary information is the key to obtaining competitive segmentation results. Thus, in addition to the commonly used boundary supervision constraints, we also incorporate the boundary information into the decoding process through the designed MBG module to achieve more in-depth and comprehensive boundary optimization and guidance. To strengthen ability to express boundary information, the features of $X^{2}$ are fed to a total of four convolutional layers to extract boundary information, thereby generating a one-channel mask called boundary map. At the same time, explicit supervision is used between the generated boundary map and the boundary ground truth obtained by the boundary extractor (e.g., Canny) to guarantee the effectiveness of learning.

Finally, we utilize the features obtained by the third decoder layer to generate the final prediction of lung infection regions through additional Sigmoid activation function.

III-B Dual-branch Semantic Enhancement Module

Due to the overwhelming background context redundancies and scattered distribution of the infection regions in the chest CT images, it is difficult to segment COVID-19 lung infection with accurate location and complete structure. For this problem, resorting to semantic comprehension is a feasible solution. The high-level features have been proven to be rich in semantic information, which can construct relationship not only between different scattered regions, but also between complex background and infected regions. Thus, we propose a DSE module to aggregate high-level features in a complementary dual-branch strategy, thereby generating a semantic attention mask $A_{SE}$ to highlight the important regions.

Different from the existing methods , we start with different spatial resolutions and information contents of different high-level features, and design a dual-branch structure, as shown in the lower-right corner of Figure 2. On one hand, the top-level features contain the richest channel semantic features, but their spatial resolution is the lowest. Therefore, the most intuitive way of upsampling and filtering is adopted to maintain the pure top-level semantic information. We directly up-sample $X^{5}$ four times to retain the pure highest-level semantic information. Referring to , we then employ spatial attention mechanism to obtain attention map $A_{t}$. On the other hand, we design a progressive multi-scale fusion strategy, taking into account the information of the three high-level features at the same time, and the final resolution is unified on the scale of the third layer. In this way, the complementary relationship between different high-level features can be learned from a more comprehensive perspective and the spatial resolution sampling distortion can be alleviated. We first implement dilated convolution layers with different dilated rates on $X^{3}$, $X^{4}$, and $X^{5}$ respectively. Then, we progressively up-sample $X^{5}$ and fuse the deep features with shallow features to achieve adequate high-level features aggregation. The fused features are transformed to attention map $A_{p}$ by a convolutional layer.

Our final attention map $A_{SE}$ is obtained by adding two attention maps $A_{t}$ and $A_{p}$ generated by two complementary attention calculation methods.

where $\sigma$ denotes the sigmoid function, $conv_{n\times n}$ represents a convolution operation with the filter size of $n\times n$, and $\hat{\theta}_{n\times n}$ is the learnable parameters of the corresponding convolution operation.

(2) Progressive Semantic Integration. Although we regard the third, fourth, and fifth layers as high-level feature layers, the features they extract are still different. Therefore, in order to obtain more comprehensive semantic information, effectively fusing them is a reasonable solution. In addition, since the size of the infected area varies greatly, in order to allow the model to obtain a robust and stable segmentation result for different regions, we first enhance the features of each layer before fusion to perceive a larger receptive field. Concretely, three dilated convolution layers with the dilated rate of $8$, $4$, and $2$ are separately applied to the input features $X^{3}$, $X^{4}$, and $X^{5}$, thereby generating multi-scale features $F_{dc}^{3}$, $F_{dc}^{4}$, and $F_{dc}^{5}$:

where $conv_{d=n}$ represents a $3\times 3$ convolution operation with the dilated rate of $n$, and $\hat{\omega}_{d=n}$ denotes the learnable parameters.

Then, in order to reduce the resolution blur distortion caused by upsampling as much as possible in the fusion process, we adopt a progressively fusion strategy. We first concatenate the upsampled features $F_{d}^{5}$ with $F_{d}^{4}$ and employ a $3\times 3$ convolution layer to generate the fusion features $F_{4,5}$, which has the same spatial resolution with $X^{4}$. Similarly, the fused features $F_{4,5}$ are further upsampled and combined with features $F^{3}_{d}$, thereby obtaining the global semantic features $F_{3,4,5}$. After that, we employ a convolution layer with filter size of $1\times 1$ to generate the semantic integration attention map $A_{p}$. Formally, the above fusion processes can be formulated as:

where $up_{2}(\cdot)$ is the $\times 2$ spatial upsampling.

Finally, these two attention maps are aggregated to produce the final semantic enhancement attention map $A_{SE}$, which is computed as:

where $\oplus$ represents the element-wise summation. The attention map $A_{SE}$ is used to further refine decoding process after the MBG module guidance of each stage.

III-C Mirror-symmetric Boundary Guidance Module

As we all know, low-level features include rich detailed information (e.g., boundaries) with a larger spatial resolution, which are conducive to refine boundaries of the lesion regions accurately. In order to highlight the important boundaries in the feature decoding, we design a MBG module to introduce the boundary guidance of low-level features from the perspective of feature integration, as shown in Figure 3. Instead of simply combining the features through concatenation or addition operation, the MBG module is designed as a mirror-symmetric structure to combine the corresponding encoder features and the boundary information from the second layer of encoder stage. In other words, the boundary features $X^{2}$ (also denoted as $F_{b}$) and the corresponding encoder features $X^{s}$ are worked as the basic features of each other, and other features are used for guidance. In the right branch generating $F_{fb}^{s}$, we modify the encoder features by using the mask derived from boundary features, aiming to reinforce important boundary locations in the encoder features. However, compared to features from the last three convolutional blocks ($X^{3}$, $X^{4}$, and $X^{5}$), $F_{b}$ may contain much redundant information. Therefore, we generate the corresponding high-level mask to refine the boundary features in the mirror-symmetric left branch. Finally, the two refined features are concatenated to form the final boundary-guided feature output. This mirror-symmetrical strategy can ensure the complementarity and sufficiency of feature learning, so as to maximize the interference suppression capability of the multiplication fusion.

where $W^{s}=up_{n}(conv_{3\times 3}(X^{s}))$, $n=2^{s-2}$ is the upsampling scale factor, $\odot$ denotes the element-wise multiplication, $\delta$ denotes the ReLU activation function, and $s$ indexes the feature level.

The other is a mirror symmetry method with the first one, which modifies the encoder features by using the mask derived from boundary features. Specifically, we obtain the boundary mask by performing an additional $3\times 3$ convolution operation on the boundary features ${F}_{b}$, and multiply it by the upsampled features $X^{s}$:

where $up(\cdot)$ upsamples $X_{s}$ to the same resolution of ${F}_{b}$. Then, we concatenate $F_{fb}^{s}$ and $F_{bf}^{s}$ to obtain the output of MBG module i.e., $F_{B}^{s}$. Each MBG module is followed by a channel-wise concatenation to integrate the upsampled decoder features from the previous stage and boundary-guided features. After that, the features after concatenation are fed into a $3\times 3$ convolution layer for compressing to the original channel number. Finally, a skip connection is employed to generate the current decoder features $F_{d}^{s}$:

where $s\in{\{5,4,3\}}$ indexes the decoder stage. Note that, for the calculation of the top-level decoder features $F_{d}^{5}$, since there is no previous decoder layer, we directly use the semantic enhancement attention map instead, i.e., $F_{d}^{6}=A_{SE}$.

III-D Loss Function

We design a hierarchical loss function on the side outputs of different scales (i.e., $S_{3}$, $S_{4}$, and $S_{5}$) and semantic attention map $A_{SE}$ by weighted IoU loss and weighted BCE loss. Following , , compared with the traditional IoU loss and BCE loss, the weights in the weighted IoU/BCE loss pay more attention on hard pixels and assign larger weights to them. In addition, local structure information is encoded into the weighted BCE loss, which may help the model focus on a larger receptive field rather than on a single pixel. Specifically, the weighted BCE loss and weighted IoU loss are defined as:

where $W$ and $H$ are the width and height of the input image, $\textbf{1}(\cdot)$ is the indicator function, $l=\{0,1\}$ indicates two kinds of labels, $\gamma$ is a hyper parameter, $p_{ij}^{k}$ and $gt_{ij}^{k}$ are the prediction and ground truth of the pixel at location $(i,j)$ in the image $k$, $k=\{S_{3},S_{4},S_{5},A_{SE}\}$, $\Psi$ represents all the parameters of the model, $\textbf{Pr}(p_{ij}^{k}=l\mid\Psi)$ denotes the predicted probability, and $\alpha_{ij}^{k}=\left|\frac{\sum_{(m,n)\in{A_{ij}}}gt_{mn}^{k}}{\sum_{(m,n)\in{A_{ij}}}1}-gt_{ij}^{k}\right|$ is a pixel importance indicator, which is calculated by the difference between the center pixel and its surrounding pixel set $A_{ij}$. For simplicity, we do not distinguish the importance of these two losses in the final loss function, and the weighting coefficients of weighted BCE loss and weighted IoU loss are both set to 1.

Moreover, we use the standard binary cross-entropy on the boundary map as the boundary-aware loss function. Therefore, the total loss can be defined as:

where $l^{b}_{bce}$ is the boundary loss using the standard binary cross-entropy.

IV Experiments

Benchmark Dataset. Research shows that there are currently fewer public COVID-19 lung CT datasets used for infection segmentation. To be able to train the model better, we merge the two publicly available CT-based segmentation datasets to obtain 1018 CT images, which are further divided into 718 training images and 300 testing images. The design of our method requires boundary supervision information, so the Canny operator is used to extract the boundaries of the infection mask. Each CT slice contains the original image, the corresponding infection mask, and the corresponding infection boundary.

Evaluation Metrics. We use seven metrics for quantitative evaluation, i.e., Dice Similarity Coefficient (DSC) , Sensitivity (Sen.) , Structure Measure ($S_{\alpha}$) , Enhance-alignment Measure ($E_{\phi}$) , Mean Absolute Error (MAE) , Precision (Prec.) , and Hausdorff Distance (HD) . The Prec. is the proportion of positive samples in the samples that are predicted to be positive, and the Sen. assesses the ratio of correctly identified positive cases to all positive cases.

The DSC is used to evaluate the overlap ratio between the predicted segmentation map $S_{p}$ and the corresponding ground truth $G$, which is calculated by:

The $S_{\alpha}$ measures the structural similarity between the segmentation map $S_{p}$ and the ground truth $G$:

where $\alpha$ is set to 0.5 for assigning equal contribution to both region similarity $S_{r}$ and object similarity $S_{o}$.

The $E_{\phi}$ is used to evaluate both local and global similarity between two binary maps, which is formulated as:

where $w$ and $h$ are the width and height of the image, $(x,y)$ denotes the coordinate of each pixel in $S_{p}$ and $G$, and $\phi$ is the enhanced alignment matrix.

The MAE measures the pixel-level error between the prediction map $S_{p}$ and the ground truth $G$, which is defined as:

The HD explicitly measures the boundary performance, which is defined as:

where $p$ and $q$ represent the pixel in the boundary prediction set $B$ and boundary ground truth set $G_{B}$, respectively.

Among these indicators, Sen. and $S_{\alpha}$ can reflect the segmentation integrity, HD measures the boundary effects, and the DSC, Prec., $E_{\phi}$ and MAE can evaluate the overall performance. Moreover, in addition to the MAE and HD, the larger the value, the better the performance.

IV-B Implementation Details

We implement the proposed model via the PyTorch toolbox and train it on an RTX 2080Ti GPU in an end-to-end manner. We also implement our network by using the MindSpore Lite toolhttps://www.mindspore.cn/. Referring to Inf-Net , the Res2Net-50 pretrained on ImageNet is employed as the backbone feature extractor in the experiment. In addition to the reason for fair comparison, the reason why we choose Res2Net-50 as the backbone network also benefits from its own advantages. First, compared to the classical ResNet and VGG , the Res2Net can construct hierarchical residual-like connections within one single residual block, represent multi-scale features at a granular level, and increase the range of receptive fields for each network layer. In addition, the Res2Net consumes less parameters and computing resources, which is also very friendly to improve the real-time efficiency of our model. Due to limited computing resources, all input images are resized to $352\times 352$, and a multi-scale training strategy is used to train the network. Our BSNet is trained by using the Adam optimizer for 90 epochs, the batch size and learning rate are set to 8 and $1e^{-4}$ respectively. We choose the model according to the determined epoch number. As can be seen from the training curve shown in Figure 4, our network can converge after training to 90 epochs.

IV-C Comparison with the State-of-the-art Methods

In order to demonstrate the effectiveness of BSNet, we compare it with eleven state-of-the-art methods, including UNet , UNet++ , Attention_UNet , ResNet34_Unet , CeNet , fcn8s , CopleNet , JCS , FFR , AnamNet , and Inf-Net . To ensure the fairness of the experiment, all state-of-the-art methods are retrained on the same dataset as our BSNet under the default parameters.

Qualitative Comparison. Figure 5 shows the visual comparison results of different methods. We can see that our method more accurately and completely locates the COVID-19 lung infection regions than other competing methods. On the whole, the classic segmentation methods (e.g., UNet , UNet++ , Attention_UNet , and ResNet34_UNet ) tend to have weak background interference suppression capabilities, leading to erroneous prediction results. By contrast, the existing COVID-19 segmentation methods (e.g., Inf-Net , CopleNet , and AnamNet ) achieve better detection results, but these methods fail to completely suppress the background, thereby leading to false detection and missing detection to some extent. For example, the background regions in the middle of the fifth image are not effectively suppressed, and there is the missing detection phenomenon in the lower area of the image. However, our proposed method exhibits stronger advantages in terms of accurate positioning, background suppression, and detection integrity. In addition, the lower right area of the last image, only our method can locate the infected regions clearly, accurately, and completely. In general, our method has a more complete structure and clearer boundaries, which benefits from the full use of high-level semantic information and edge information for modeling.

Quantitative Comparison. The numerical indexes, including Dice Similarity Coefficient (DSC), Sensitivity (Sen.), Precision (Prec.), Structure Measure ($S_{\alpha}$), Enhance-alignment Measure ($E_{\phi}$), Mean Absolute Error (MAE), and Hausdorff Distance (HD), are reported in Table II. It is evident that our model achieves competitive performance across different metrics. To be specific, our method achieves the best performance in terms of all measures on the merged dataset.

Compared with the classical deep learning-based segmentation methods for natural imsges (i.e., UNet , UNet++ , Attention_UNet , and ResNet34_UNet ), most of the segmentation methods specifically designed for COVID-19 exhibit the superior performance, demonstrating the particularity and challenges of COVID-19 infection segmentation. It’s worth mentioning that AnamNet is an embedding-based lightweight network with about one-sixth parameter consumption of the classic U-Net, which is why its performance is not as good as some classic medical segmentation algorithms. In summary, due to the delicately designed modules, our BSNet ranks first in all evaluation metrics. For example, compared with the classic UNet method, the percentage gain of our method reaches 9.52% in terms of the DSC, and 30.00% in terms of the MAE score. Compared with the second best method, the percentage gain reaches 1.43% in terms of the DSC, 1.40% in terms of Prec., and 6.67% in terms of MAE, respectively. In addition, similar to the visualization results, some metrics, such as Sen. and $S_{\alpha}$, reflecting the detection completeness also demonstrate the advantages of our method. In terms of these two indicators, our method achieves the best results compared to other competitors. Specifically, compared to the second best method, the percentage gain reaches 0.35% in terms of the Sen., and 0.80% in terms of $S_{\alpha}$. Likewise, our method achieves the best performance in boundary effect evaluation by using the HD score. For example, our method wins a minimum percentage gain of 0.9%, and a maximum percentage gain of 56.4% against the comparison methods. All these clearly demonstrate the superior performance of the proposed model in COVID-19 lung infection segmentation.

In order to make the inference speed comparison with other methods succinctly and clearly, we provide the FPS-DSC map in Figure 6. Generally speaking, more than 30 FPS can be considered real-time, and our model reaches 44 FPS, which meets the real-time requirement. While some algorithms are fast at inference (e.g., AnamNet , CopleNet ), their performance is not as good as ours. In other words, our method strikes a trade-off between performance and efficiency.

IV-D Ablation Study

We conduct ablation experiments to verify the effectiveness of the each key module of our proposed model, including the MBG module and DSE module. The quantitative results are shown in Table III.

In model 2, the DSE module is first added to the baseline to deal with background context redundancies and scattered distribution of the infection regions in the chest CT image. Compared to baseline (model 1), the high-level features are not directly used as global information for guidance, but refined by DSE module. Compared with the baseline (model 1), all indicators in model 2 increase obviously, especially for the DSC and MAE. Specifically, the DSC is boosted from 0.764 to 0.838 with the percentage gain of 9.69%, and the MAE is improved from 0.024 to 0.017 with the percentage gain of 29.17%. It further shows that the DSE proposed in this paper can more effectively use high-level semantic information to provide the global guidance. We present the qualitative comparison results in Figure 7. Compared with the Baseline model shown in Figure 7(c), it can be found that some irreverent interferences and complex backgrounds around the infection region are well suppressed by introducing the DSE module (e.g., the leftmost region in the second image and the small upper right region in the third image).

We discuss the impact of the MBG module on the entire network and design the ablation experiments. The model 3 (full model) is to verify the importance of the boundary information, that is, add the MBG module to aforementioned architecture. It can be seen from Table III that the introduction of boundary information is effective and can bring performance gains. Compared with the model 2, the DSC and Sen. scores of model 3 are improved by 1.30% and 1.20%, respectively. For the visualization evaluation of the boundary effect, we can start from the following two aspects: (1) Boundary accuracy means that the detected boundary structure is complete, the location is accurate, and the degree of coincidence with the boundary GT is high. For example, compared with column (d) in Figure 7, the tiny regions in the upper right corner of the first image and the bottom right corner of the third image are all detected by our model after introducing the MBG unit. (2) Boundary sharpness refers to the boundaries of the detection result are sharp, clear and non-blurred, which is of great significance for diagnosis and treatment. For example, the boundaries of the large infected regions in the lower half of the second image are obviously sharper than the results in the column (d) of Figure 7. In order to observe the changes of boundary effects more clearly and intuitively, we provide the HD scores below the visualization results. It can be seen that the boundary effect has been improved after introducing the MBG module. In addition, in order to verify the design of mirror-symmetric structure in MBG module, we design an additional ablation experiment with only the left branch or right branch, as shown in Table IV. We can clearly see that the absence of any branch in the MBG module will lead to inferior performance, which illustrates the necessity of our mirror-symmetric design.

IV-D2 Validation of different boundary guidance methods

To verify how boundary supervision and boundary features are generated, we design two ablation experiments.

First, the boundary supervision map is directly processed on the binarized segmentation ground truth, which is a very simple computational task. Theoretically, although different boundary extractors result in slightly different boundary ground truths, there is no significant difference in general boundary locations. To verify this, we design an ablation experiment with three boundary GT extraction methods, including Canny operator, Sobel operator, and Roberts operator, as shown in Table V. From it, we can see that there is almost no difference in the final segmentation results obtained by different boundary GT generation algorithms, which confirms our previous conjecture.

Second, as demonstrated in the existing works , the low-level features (e.g., $X^{1}$, $X^{2}$) have a larger spatial resolution and include rich detailed information such as boundaries, which are conducive to refine boundaries of the lesion regions accurately. In our method, $X^{2}$ is chosen to provide boundary guidance for the MBG module, because $X^{1}$ contains a lot of unimportant and indistinguishable information, which is disruptive or burdensome for feature purifying. To this end, we add an ablation study to verify the effect of different boundary features. As shown in Table V, the top-level features $X^{5}$ containing rich semantic information are the worst, which also verifies the validity and rationality of our use of low-level features as boundary features. Furthermore, the boundary guidance from the features of $X^{2}$ achieves better performance than using $X^{1}$, which illustrates the effectiveness of our setup.

IV-D3 Validation of different output stages

The final output of the network is derived from features at the third decoding stage, mainly based on the following two points. First, we design the MBG module to make full use of the features of the second encoder layer ($X^{2}$) to supplement boundary information for high-level encoder features ($X^{3}$, $X^{4}$, and $X^{5}$). Under such a model framework, we do not embed the MBG module in shallow layers, so we also do not perform decoding at these stages. Second, in order to achieve a real-time inference speed, generating the final map from higher stage with lower resolution will consume fewer computing resources. In order to verify the difference in performance of different decoding stages, we design an ablation experiment. For fair comparison, we implement the same structure as the third decoding stage on the first and second decoding stage, which contains the MBG module, a channel-wise concatenation and a $3\times 3$ convolution layer. As shown in Table VI, it can be found that the third decoding layer achieves better performance than using the first and second decoder features.

V Conclusion and Future Work

This paper proposes a boundary guided semantic learning network for automatically segmenting COVID-19 lung infections from CT images by studying how to capture the infection area from the perspective of semantic relation and boundary guidance. The DSE module models semantic relations through complementary dual-branch strategies, and MBG module adopts mirror symmetry structure to ensure the complementarity and sufficiency of feature learning. Experiments show that our BSNet outperforms the state-of-the-art competitors and achieves the real-time effects.

Although our algorithm achieves a more complete structure and more accurate details, it is very challenging for COVID-19 infection segmentation due to the scattered infected regions over the chest slice. It just so happens that Transformer (e.g., ViT and Swin Transformer ), which can model long-term dependencies in data through self-attention mechanism, have been widely used in medical image segmentation . Therefore, we believe that exploring Transformer-based COVID-19 infection segmentation task is a worthy future research direction. In addition, considering the contradiction between the model and data, on the one hand, global scientific research institutions can be called on to open source relevant data in accordance with relevant regulations. On the other hand, we can try to use domain adaptation to better transfer the model trained on the normal medical segmentation dataset to the COVID-19 infection segmentation, which technically makes up for the lack of training data.

References