Federated Incremental Semantic Segmentation

Introduction

Federated learning (FL) is a remarkable decentralized training paradigm to learn a global model across distributed local clients without accessing their private data. Under privacy preservation, it has achieved rapid development in semantic segmentation by training on multiple decentralized local clients to alleviate the constraint of data island that requires enormous finely-labeled pixel annotations . As a result, federated learning-based semantic segmentation (FSS) significantly economizes annotation costs in data-scarce scenarios via training a global segmentation model on private data of different clients .

However, existing FSS methods unrealistically assume that the learned foreground classes are static and fixed over time, which is impractical in real-world dynamic applications where local clients receive streaming data of new categories consecutively. To tackle this issue, existing FSS methods typically enforce local clients to store all samples of previously-learned old classes, and then learn a global model to segment new categories continually via FL. Nevertheless, it requires large computation and memory overhead as new classes arrive continuously, limiting the application ability of FSS methods . If local clients have no memory to store old classes, existing FSS methods significantly degrade segmentation behavior on old categories (i.e., catastrophic forgetting ) when learning new classes incrementally. In addition, the pixels labeled as background in the current learning task may belong to old classes from old tasks or new foreground classes from future tasks. This phenomenon is also known as background shift that heavily aggravates heterogeneous forgetting speeds on old categories. More importantly, in practical scenarios, new local clients receiving new categories incrementally may join in global FL training irregularly, thus further exacerbating catastrophic forgetting to some extent.

To surmount the above real-world scenarios, we propose a novel practical problem called Federated Incremental Semantic Segmentation (FISS), where local clients collect new categories consecutively according to their preferences, and new local clients collecting unseen novel classes participate in global FL training irregularly. In the FISS settings, the class distributions are non-independent and identically distributed (Non-IID) across different clients, and training data of old classes is unavailable for all local clients. FISS aims to train a global incremental segmentation model via collaborative FL training on local clients while addressing catastrophic forgetting. In this paper, we use medical lesions segmentation as an example to better illustrate FISS, as shown in Figure 1. Hundreds of hospitals, as well as newly joined ones, collect unseen/new medical lesions continuously in clinical diagnosis. Considering privacy preservation, it is desired for these hospitals to learn a global segmentation modal via FL without accessing each other’s data .

A naive solution for FISS problem is to directly integrate incremental semantic segmentation and FL together. Nevertheless, such a trivial solution requires global server to have strong human prior about which and when local clients can collect new categories, so that global model learned in the latest old task can be stored by local clients to address forgetting on old classes via knowledge distillation . Considering privacy preservation in the FISS, this privacy-sensitive prior knowledge cannot be shared between local clients and global server. As a result, this naive solution severely suffers from intra-client heterogeneous forgetting on different old classes caused by background shift , and inter-client heterogeneous forgetting across different clients brought by Non-IID class distributions.

To overcome the above-mentioned challenges, we develop a novel Forgetting-Balanced Learning (FBL) model, which alleviates heterogeneous forgetting on old classes from intra-client and inter-client perspectives. Specifically, to tackle intra-client heterogeneous forgetting caused by background shift, we propose an adaptive class-balanced pseudo labeling to adaptively generate confident pseudo labels for old classes. Under the guidance of pseudo labels, we propose a forgetting-balanced semantic compensation loss to rectify different forgetting of old classes with background shift via considering balanced gradient propagation of local clients. In addition, a forgetting-balanced relation consistency loss is designed to distill underlying category-relation consistency between old and new classes for intra-client heterogeneous forgetting compensation. Moreover, considering addressing heterogeneous forgetting from inter-client aspect, we develop a task transition monitor to automatically identify new classes without any human prior, and store the latest old model from global perspective for relation consistency distillation. Experiments on segmentation datasets reveal large improvement of our model over comparison methods. We summarize the main contributions of this work as follows:

We propose a novel practical problem called Federated Incremental Semantic Segmentation (FISS), where the major challenges are intra-client and inter-client heterogeneous forgetting on old categories caused by intra-client background shift and inter-client Non-IID distributions.

We propose a Forgetting-Balanced Learning (FBL) model to address the FISS problem via surmounting heterogeneous forgetting from both intra-client and inter-client aspects. As we all know, in the FL field, this is a pioneer attempt to explore a global continual segmentation model.

We develop a forgetting-balanced semantic compensation loss and a forgetting-balanced relation consistency loss to tackle intra-client heterogeneous forgetting across old classes, under the guidance of confident pseudo labels generated via adaptive class-balanced pseudo labeling.

We design a task transition monitor to surmount inter-client heterogeneous forgetting by accurately recognizing new classes under privacy protection and storing the latest old model from global aspect for relation distillation.

Related Work

Federated Learning (FL) aggregates local-client model parameters to optimize a global model under privacy protection. enforces local model to approximate the global ones via a proximal term. To minimize computation cost, employs a layer-wise parameter aggregation strategy. Inspired by above FL methods, apply FL to semantic segmentation , which has achieved rapid developments in medical analysis and autonomous driving . considers adversarial framework to tackle domain adaptation problem in the FL field. proposes a federated class-incremental learning model via considering global and local forgetting. However, the above-mentioned methods cannot segment new foreground classes continuously under the FISS settings.

Incremental Semantic Segmentation (ISS) considers class-incremental learning in semantic segmentation. The key challenges of ISS are catastrophic forgetting and background shift , as claimed in . ILT proposes to distill latent features and probabilities between old and new models. PLOP utilizes multi-scale pooling distillation to maintain past experience. SDR considers feature consistency by prototype matching and contrastive learning . RCIL decouples the network into branches to overcome forgetting. Considering tackling background shift, propose pseudo labeling to annotate old classes labeled as background pixels. Nevertheless, these ISS methods cannot be effectively applied to address the FISS problem, due to their strong prior knowledge to access privately-sensitive information (i.e., when and which local clients receive new classes).

Problem Definition

We then extend the settings of incremental semantic segmentation (ISS) to Federated Incremental Semantic Segmentation (FISS). Denote global server as $\mathcal{S}_{g}$ and $L$ local clients as $\{\mathcal{S}_{l}\}_{l=1}^{L}$. In the FISS, at the $r$-th ($r=1,\cdots,R$) global round, we randomly select some local clients to aggregate gradients. When we choose the $l$-th local client to learn the $t$-th segmentation task, the latest global model $\Theta^{r,t}$ is distributed to $\mathcal{S}_{l}$, and trained on private training data $\mathcal{T}_{l}^{t}=\{\mathbf{x}_{li}^{t},\mathbf{y}_{li}^{t}\}_{i=1}^{N_{l}^{t}}\sim\mathcal{P}_{l}$ of $\mathcal{S}_{l}$. $\mathbf{x}_{li}^{t}$ and $\mathbf{y}_{li}^{t}\in\mathcal{Y}_{l}^{t}$ denote the images and labels of the $l$-th client. $\{\mathcal{P}_{l}\}_{l=1}^{L}$ are non-independent and identically distributed (i.e., Non-IID) across local clients. The label space $\mathcal{Y}_{l}^{t}\subset\mathcal{Y}^{t}$ of $\mathcal{S}_{l}$ in the $t$-th task is composed of $K_{l}^{t}$ new classes ($K_{l}^{t}\leq K^{t}$) that belongs to a subset of $\mathcal{Y}^{t}=\cup_{l=1}^{L}\mathcal{Y}_{l}^{t}$. Following ISS methods , we consider background shift in the FISS and also annotate $K_{l}^{o}=\sum_{i=1}^{t-1}K_{l}^{i}\subset\cup_{j=1}^{t-1}\mathcal{Y}_{l}^{j}$ old categories from $t-1$ old tasks and other foreground categories from future learning tasks as background. After getting global model $\Theta^{r,t}$ and performing local training on $\mathcal{T}_{l}^{t}$, $\mathcal{S}_{l}$ obtains a updated local model $\Theta_{l}^{r,t}$. Then global server $\mathcal{S}_{g}$ aggregates local models of selected clients as the global model $\Theta^{r+1,t}$ for the training of next global round.

In the $t$-th task, motivated by , all local clients $\{\mathcal{S}_{l}\}_{l=1}^{L}$ are divided into three categories: $\{\mathcal{S}_{l}\}_{l=1}^{L}=\mathbf{S}_{o}\cup\mathbf{S}_{c}\cup\mathbf{S}_{n}$. Specifically, $\mathbf{S}_{o}$ is composed of $L_{o}$ local clients that have accumulated past experience for previous tasks but cannot collect new training data of the $t$-th task; $\mathbf{S}_{c}$ consisting of $L_{c}$ local clients can receive new training data of current task and has learning experience for old classes; $\mathbf{S}_{n}$ includes $L_{n}$ new local clients with unseen novel classes but without past learning experience of old classes. These local clients are randomly determined in each incremental task. New clients $\mathbf{S}_{n}$ are added randomly at any global round in the FISS, increasing $L=L_{o}+L_{c}+L_{n}$ gradually as continuous tasks. More importantly, we don’t have prior knowledge about the class distributions $\{\mathcal{P}_{l}\}_{l=1}^{L}$, quantity and order of segmentation tasks, when and which local clients receive new classes. In this paper, FISS aims to learn a global model $\Theta^{R,T}$ to segment new categories continuously while surmounting heterogeneous forgetting on old categories brought by background shift, under the privacy preservation of local clients.

The Proposed Model

Figure 2 presents the overview of our model to address the FISS problem. Our FBL model overcomes intra-client heterogeneous forgetting via a forgetting-balanced semantic compensation loss (Section 4.2) and a forgetting-balanced relation consistency loss (Section 4.3), under the guidance of adaptive class-balanced pseudo labeling (Section 4.1) to mine pseudo labels for old classes with background shift. Meanwhile, it addresses inter-client heterogeneous forgetting via a task transition monitor (Section 4.4) to recognize new classes and store old model for relation distillation.

As aforementioned, in the FISS settings, local client $\mathcal{S}_{l}$ has no memory to store $K^{o}$ old classes, while background pixels may belong to $K^{o}$ old classes, other foreground classes from future tasks or real background (i.e., background shift ). As a result, it enforces the updating of local model $\Theta_{l}^{r,t}$ (i.e., Eq. (1)) to suffer from intra-client heterogeneous forgetting among different old classes brought by background shift, after $\mathcal{S}_{l}$ receives the global model $\Theta^{r,t}$ from $\mathcal{S}_{g}$ for local training. To this end, as shown in Figure 2, we develop an adaptive class-balanced pseudo labeling to adaptively mine confident pseudo labels for old classes labeled as background pixels in the $t$-th segmentation task. Different from existing ISS methods that only utilize a constant probability threshold to select pseudo labels for all classes, our FBL model considers class balance to mine pseudo labels for old classes via introducing class-specific entropy threshold for each old class, which are determined as continual learning process. These class-balanced pseudo labels of $K^{o}$ old classes are essential to alleviate heterogeneous forgetting of old classes.

where $(\hat{\mathbf{y}}_{li}^{t})_{j}$ is pseudo label of the $j$-th pixel from $\hat{\mathbf{y}}_{li}^{t}$. $\mathbf{P}_{l}^{t-1}(\mathbf{x}_{li}^{t},\Theta^{t-1})_{j}$ is softmax probability of the $j$-th pixel from $\mathbf{P}_{l}^{t-1}(\mathbf{x}_{li}^{t},\Theta^{t-1})$. $h_{j}=\mathcal{H}(\mathbf{P}_{l}^{t}(\mathbf{x}_{li}^{t},\Theta_{l}^{r,t})_{j})$ represents entropy of the $j$-th pixel, and $\mathcal{H}(\mathbf{p})=\sum_{i}\mathbf{p}_{i}\log\mathbf{p}_{i}$ is entropy measure function. $\{\gamma_{l}^{k}\}_{k=1}^{K^{o}}$ denote class-specific entropy threshold to adaptively select class-balanced pseudo labels with high confidence. As shown in Eq. (2), in the $t$-th task $\mathcal{T}_{l}^{t}$, when the $j$-th pixel belongs to background label space $\mathcal{Y}_{l}^{b}$ (i.e., $(\mathbf{y}_{li}^{t})_{j}\in\mathcal{Y}_{l}^{b}$) and the entropy $h_{j}$ is less than $\gamma_{l}^{k}$, pseudo label is determined by $(\hat{\mathbf{y}}_{li}^{t})_{j}=\arg\max\mathbf{P}_{l}^{t-1}(\mathbf{x}_{li}^{t},\Theta^{t-1})_{j}$. If the $j$-th pixel is not labeled as background (i.e., $(\mathbf{y}_{li}^{t})_{j}\notin\mathcal{Y}_{l}^{b}$), we consider pseudo label as new foreground classes: $(\hat{\mathbf{y}}_{li}^{t})_{j}=(\mathbf{y}_{li}^{t})_{j}$. Otherwise, $(\hat{\mathbf{y}}_{li}^{t})_{j}=0$ denotes real background for the $j$-th pixel of $\hat{\mathbf{y}}_{li}^{t}$.

2 Forgetting-Balanced Semantic Compensation

where $\mathbf{P}_{l}^{t}(\mathbf{x}_{li}^{t},\Theta_{l}^{r,t})_{j}^{k}$ is probability of the $k$-th class at the $j$-th pixel of $\mathbf{x}_{li}^{t}$, and $k=(\hat{\mathbf{y}}_{li}^{t})_{j}$ denotes pseudo label of the $j$-th pixel in $\mathbf{x}_{li}^{t}$. Considering that intra-client heterogeneous forgetting of old classes changes dynamically as continual learning tasks, we expect gradient scalar $\Gamma_{ij}^{t}$ of old classes to be adaptive in the FISS, and reformulate Eq. (3) as:

where $\mathcal{Y}_{l}^{b}$ is background label space of the $l$-th local client $\mathcal{S}_{l}$. When pseudo label $(\hat{\mathbf{y}}_{li}^{t})_{j}$ of the $j$-th pixel in $\mathbf{x}_{li}^{t}$ belongs to old classes from previous $t\!-\!1$ tasks, $\bar{\Gamma}_{ij}^{t}=|\Gamma_{ij}^{t}|^{{K_{l}^{o}}/{(K_{l}^{o}+K_{l}^{t})}}$; otherwise, $\bar{\Gamma}_{ij}^{t}=|\Gamma_{ij}^{t}|$ for new classes and background.

As a result, given mini-batch samples $\{\mathbf{x}_{li}^{t},\hat{\mathbf{y}}_{li}^{t}\}_{i=1}^{B}\in\mathcal{T}_{l}^{t}$ in the $t$-th segmentation task, we denote gradient propagation means $\Gamma_{b}$ and $\Gamma_{\eta}$ for the background and foreground classes learned from the $\eta$-th ($1\leq\eta\leq t$) task as follows:

3 Forgetting-Balanced Relation Consistency

The intrinsic relations between old and new classes are immutable in purely semantic space, independent of background shift and availability of training data of old classes. In light of this, consistent semantic relations between old model $\Theta^{t-1}$ and current local model $\Theta_{l}^{r,t}$ plays an important role in tackling intra-client heterogeneous forgetting on old classes. However, most existing ISS methods only consider underlying relationships among old classes via performing knowledge distillation on an individual sample, which can be severely affected by noisy predictions on old classes brought by background shift. In addition, forgetting heterogeneity of old classes within local clients enforces most ISS methods to suffer from heterogeneous inter-class relations distillation, thus aggravating imbalanced gradient propagation across incremental tasks.

In summary, the major objective of the $l$-th local client $\mathcal{S}_{l}$ to learn the $t$-th segmentation task $\mathcal{T}_{l}^{t}$ is expressed as:

4 Task Transition Monitor

5 Optimization Procedure

Experiments

We utilize two benchmark datasets: Pascal-VOC 2012 and ADE20k under various experimental settings to analyze effectiveness of our FBL model. For fair comparisons with baseline ISS methods under the FISS settings, we follow them to set exactly the same incremental tasks and class order, while using the identical segmentation backbone (i.e., Deeplab-v3 with ResNet-101 pretrained on ImageNet dataset ). As claimed in , background pixels in the current task may belong to old classes or new classes from future tasks (i.e., background has some overlap with new foreground classes in the future tasks). In the FISS, we consider more challenging settings by assigning more incremental segmentation tasks with overlapped foregrounds. On Pascal-VOC 2012 , 15-1, 4-4, and 8-2 settings with overlapped foregrounds respectively consist in 15 classes followed by 1 classes 5 times ($T=6$), learning 4 classes followed by 4 classes 4 times ($T=5$), and 8 classes followed by 2 classes 6 times ($T=7$). Likewise, on ADE20k , 100-10 setting with overlapped foregrounds means 100 classes followed by 10 classes 5 times ($T=6$).

We employ SGD optimizer with initial learning rate as $1.0\times 10^{-2}$ to train the first base task and $1.0\times 10^{-3}$ to learn incremental tasks. Considering the limitation of GPU overhead, we set initial local clients as $10$, and add $4$ new local clients for each task. We choose $4$ local clients randomly to perform local training with $6$ epochs for VOC and 12 epoches for ADE20k . On VOC dataset , we randomly select $40\%$ images for each client in each segmentation task under 15-1 setting; otherwise, we randomly sample $50\%$ classes from current label space $\mathcal{Y}^{t}$, and assign $60\%$ samples from these classes to selected local clients under the 4-4 and 8-2 settings. For the 100-10 setting in ADE20k , we randomly choose $70\%$ classes from $\mathcal{Y}^{t}$, and distribute them to selected clients. Following ISS methods , we employ mean Intersection over Union (mIoU) as metric, and evaluate mIoU of all classes after learning the last segmentation task (i.e., $t=T$). This metric evaluates the effectiveness to address heterogeneous forgetting and the ability to segment new classes continually.

2 Comparison Performance

3 Ablation Studies

4 Analysis of Task-Wise Comparisons

As presented in Table 6, we introduce task-wise comparison results to analyze the effectiveness of our model to address FISS settings. Our model outperforms baseline ISS methods for most task-wise comparisons under the overlapped 4-4 setting. The proposed FBL model encourages local clients to learn a global incremental segmentation model cooperatively under privacy preservation. Comparisons in Table 6 show large mIoU improvements of our model to address the FISS problem over other ISS methods. When segmenting new foreground classes consecutively, our model can effectively tackle intra-client and inter-client heterogeneous forgetting on different old classes.

Conclusion

In this work, we propose a Federated Incremental Semantic Segmentation (FISS) problem, and develop a novel Forgetting-Balanced Learning (FBL) model to address intra-client and inter-client heterogeneous forgetting on old classes. To tackle intra-client heterogeneous forgetting, we design a forgetting-balanced semantic compensation loss and a forgetting-balanced relation consistency loss, under the guidance of adaptive class-balanced pseudo labeling. Meanwhile, we propose a task transition monitor to address inter-client heterogeneous forgetting. It can automatically recognize new classes and store the latest old global model for distillation. Comparison results demonstrate the superiority of our model to tackle the FISS problem. In the future, we will consider using only few samples of new classes to address intra-client and inter-client forgetting.

References

Appendix A Appendix

A.2 Ablation Studies

In this subsection, we present qualitative ablation studies to verify the effectiveness and superiority of our proposed modules. As shown in Table 7, when removing one of the designed modules, the performance in terms of mIoU heavily degrades about $3.9\%\sim 11.6\%$. Specifically, when compared with Ours, Ours-w/oAPL decreases $3.9\%\sim 9.0\%$ mIoU, which validates the effectiveness of the proposed adaptive class-balanced pseudo labeling to mine confident pseudo labels of old classes. These pseudo labels provide strong guidance for two forgetting-balanced losses to address intra-client heterogeneous forgetting on old classes. Moreover, Ours significantly outperforms Ours-w/oFSC by a large margin of $3.9\%\sim 6.5\%$ mIoU. This significant performance improvement verifies that our FBL model could effectively tackle forgetting heterogeneity of different old classes within each local client via the forgetting-balanced semantic compensation loss. In addition, Ours-w/oFRC degrades the segmentation performance of $7.0\%\sim 11.6\%$ mIoU, compared with Ours. This phenomenon illustrates the effectiveness and superiority of the proposed forgetting-balanced relation consistency loss to compensate heterogeneous relation distillation gains. More importantly, the performance degradation illustrates that all designed modules are effective to collaboratively learn a global incremental segmentation model under the practical FISS settings.