Overcoming Catastrophic Forgetting by Incremental Moment Matching

Introduction

Catastrophic forgetting is a fundamental challenge for artificial general intelligence based on neural networks. The models that use stochastic gradient descent often forget the information of previous tasks after being trained on a new task . Online multi-task learning that handles such problems is described as continual learning. This classic problem has resurfaced with the renaissance of deep learning research .

Recently, the concept of applying a regularization function to a network trained by the old task to learning a new task has received much attention. This approach can be interpreted as an approximation of sequential Bayesian . Representative examples of this regularization approach include learning without forgetting and elastic weight consolidation . These algorithms succeeded in some experiments where their own assumption of the regularization function fits the problem.

Here, we propose incremental moment matching (IMM) to resolve the catastrophic forgetting problem. IMM uses the framework of Bayesian neural networks, which implies that uncertainty is introduced on the parameters in neural networks, and that the posterior distribution is calculated . The dimension of the random variable in the posterior distribution is the number of the parameters in the neural networks. IMM approximates the mixture of Gaussian posterior with each component representing parameters for a single task to one Gaussian distribution for a combined task. To merge the posteriors, we introduce two novel methods of moment matching. One is mean-IMM, which simply averages the parameters of two networks for old and new tasks as the minimization of the average of KL-divergence between one approximated posterior distribution for the combined task and each Gaussian posterior for the single task . The other is mode-IMM, which merges the parameters of two networks using a Laplacian approximation to approximate a mode of the mixture of two Gaussian posteriors, which represent the parameters of the two networks.

In general, it is too naïve to assume that the final posterior distribution for the whole task is Gaussian. To make our IMM work, the search space of the loss function between the posterior means needs to be smooth and convex-like. In other words, there should not be high cost barriers between the means of the two networks for an old and a new task. To make our assumption of Gaussian distribution for neural network reasonable, we applied three main transfer learning techniques on the IMM procedure: weight transfer, L2-norm of the old and the new parameters, and our newly proposed variant of dropout using the old parameters. The whole procedure of IMM is illustrated in Figure 1.

Previous Works on Catastrophic Forgetting

One of the major approaches preventing catastrophic forgetting is to use an ensemble of neural networks. When a new task arrives, the algorithm makes a new network, and shares the representation between the tasks . However, this approach has a complexity issue, especially in inference, because the number of networks increases as the number of new tasks that need to be learned increases.

Another approach studies the methods using implicit distributed storage of information, in typical stochastic gradient descent (SGD) learning. These methods use the idea of dropout, maxout, or neural module to distributively store the information for each task by making use of the large capacity of the neural network . Unfortunately, most studies following this approach had limited success and failed to preserve performance on the old task when an extreme change to the environment occurred . Alternatively, Fernando et al. proposed PathNet, which extends the idea of the ensemble approach for parameter reuse within a single network. In PathNet, a neural network has ten or twenty modules in each layer, and three or four modules are picked for one task in each layer by an evolutionary approach. This method alleviates the complexity issue of the ensemble approach to continual learning in a plausible way.

The approach with a regularization term also has received attention. Learning without forgetting (LwF) is one example of this approach, which uses the pseudo-training data from the old task . Before learning the new task, LwF puts the training data of the new task into the old network, and uses the output as pseudo-labels of the pseudo-training data. By optimizing both the pseudo-training data of the old task and the real data of the new task, LwF attempts to prevent catastrophic forgetting. This framework is promising where the properties of the pseudo training set is similar to the ideal training set. Elastic weight consolidation (EWC), another example of this approach, uses sequential Bayesian estimation to update neural networks for continual learning . In EWC, the posterior distribution trained by the previous task is used to update the new prior distribution. This new prior is used for learning the new posterior distribution of the new task in a Bayesian manner. EWC assumes that the covariance matrix of the posterior is diagonal and there are no correlations between the nodes. Though this assumption is fragile, EWC performs well in some domains.

EWC is a monumental recent work that uses sequential Bayesian for continual learning of neural networks. However, updating the parameter of complex hierarchical models by sequential Bayesian estimation is not new . Sequential Bayes was used to learn topic models from stream data by Broderick et al. . Huang et al. applied sequential Bayesian to adapt a deep neural network to the specific user in the speech recognition domain . They assigned the layer for the user adaptation and applied MAP estimation to this single layer. Similar to our IMM method, Bayesian moment matching is used for sum-product networks, a kind of deep hierarchical probabilistic model . Though sum-product networks are usually not scalable to large datasets, their online learning method is useful, and it achieves similar performance to the batch learner. Our method using moment matching focuses on continual learning and deals with significantly different statistics between tasks, unlike the previous method.

Incremental Moment Matching

In incremental moment matching (IMM), the moments of posterior distributions are matched in an incremental way. In our work, we use a Gaussian distribution to approximate the posterior distribution of parameters. Given $K$ sequential tasks, we want to find the optimal parameter $\mu^{*}_{1:K}$ and $\Sigma^{*}_{1:K}$ of the Gaussian approximation function $q_{1:K}$ from the posterior parameter for each $k$th task, $(\mu_{k},\Sigma_{k})$.

$q_{1:K}$ denotes an approximation of the true posterior distribution $p_{1:K}$ for the whole task, and $q_{k}$ denotes an approximation of the true posterior distribution $p_{k}$ over the training dataset $(X_{k},y_{k})$ for the $k$th task. $\theta$ denotes the vectorized parameter of the neural network. The dimension of $\mu_{k}$ and $\mu_{1:k}$ is $D$, and the dimension of $\Sigma_{k}$ and $\Sigma_{1:k}$ is $D\times D$, respectively, where $D$ is the dimension of $\theta$. For example, a multi-layer perceptrons (MLP) with has the number of nodes, $D$ = 1917610 including bias terms.

Next, we explain two proposed moment matching algorithms for the continual learning of modern deep neural networks. The two algorithms generate two different moments of Gaussian with different objective functions for the same dataset.

Mean-IMM averages the parameters of two networks in each layer, using mixing ratios $\alpha_{k}$ with $\sum^{K}_{k}\alpha_{k}=1$. The objective function of mean-IMM is to minimize the following local KL-distance or the weighted sum of KL-divergence between each $q_{k}$ and $q_{1:K}$ :

$\mu^{*}_{1:K}$ and $\Sigma^{*}_{1:K}$ are the optimal solution of the local KL-distance. Notice that covariance information is not needed for mean-IMM, since calculating $\mu^{*}_{1:K}$ does not require any $\Sigma_{k}$. A series of $\mu_{k}$ is sufficient to perform the task. The idea of mean-IMM is commonly used in shallow networks . However, the contribution of this paper is to discover when and how mean-IMM can be applied in modern deep neural networks and to show it can performs better with other transfer techniques.

Future works may include other measures to merge the networks, including the KL-divergence between $q_{1:K}$ and the mixture of each $q_{k}$ (i.e. KL(q_{1:K}||\mbox{\sum^{K}_{k}}\alpha_{k}q_{k})) .

2 Mode-based Incremental Moment Matching (mode-IMM)

Mode-IMM is a variant of mean-IMM which uses the covariance information of the posterior of Gaussian distribution. In general, a weighted average of two mean vectors of Gaussian distributions is not a mode of MoG. In discriminative learning, the maximum of the distribution is of primary interest. According to Ray and Lindsay , all the modes of MoG with $K$ clusters lie on $(K-1)$-dimensional hypersurface $\{\theta|\theta=(\sum^{K}_{k}a_{k}\Sigma^{-1}_{k})^{-1}(\sum^{K}_{k}a_{k}\Sigma^{-1}_{k}\mu_{k}),0<a_{k}<1$ and $\sum_{k}a_{k}=1\}$. See Appendix A for more details.

Motivated by the above description, a mode-IMM approximate MoG with Laplacian approximation, in which the logarithm of the function is expressed by the Taylor expansion . Using Laplacian approximation, the MoG is approximated as follows:

For Equation 8, we add $\epsilon I$ to the term to be inverted in practice, with an identity matrix $I$ and a small constant $\epsilon$.

Here, we assume diagonal covariance matrices, which means that there is no correlation among parameters. This diagonal assumption is useful, since it decreases the number of parameters for each covariance matrix from $O(D^{2})$ to $O(D)$ for the dimension of the parameters $D$.

For covariance, we use the inverse of a Fisher information matrix, following . The main idea of this approximation is that the square of gradients for parameters is a good indicator of their precision, which is the inverse of the variance. The Fisher information matrix for the $k$th task, $F_{k}$ is defined as:

Transfer Techniques for Incremental Moment Matching

In general, the loss function of neural networks is not convex. Consider that shuffling nodes and their weights in a neural network preserves the original performance. If the parameters of two neural networks initialized independently are averaged, it might perform poorly because of the high cost barriers between the parameters of the two neural networks . However, we will show that various transfer learning techniques can be used to ease this problem, and make the assumption of Gaussian distribution for neural networks reasonable. In this section, we introduce three practical techniques for IMM, including weight-transfer, L2-transfer, and drop-transfer.

Weight-transfer initialize the parameters for the new task $\mu_{k}$ with the parameters of the previous task $\mu_{k-1}$ . In our experiments, the use of weight-transfer was critical to the continual learning performance. For this reason, the experiments on IMM in this paper use the weight-transfer technique by default.

The weight-transfer technique is motivated by the geometrical property of neural networks discovered in the previous work . They found that there is a straight path from the initial point to the solution without any high cost barrier, in various types of neural networks and datasets. This discovery suggests that the weight-transfer from the previous task to the new task makes a smooth loss surface between two solutions for the tasks, so that the optimal solution for both tasks lies on the interpolated point of the two solutions.

To empirically validate the concept of weight-transfer, we use the linear path analysis proposed by Goodfellow et al. (Figure 2). We randomly chose 18,000 instances from the training dataset of CIFAR-10, and divided them into three subsets of 6,000 instances each. These three subsets are used for sequential training by CNN models, parameterized by $\theta_{1}$, $\theta_{2}$, and $\theta_{3}$, respectively. Here, $\theta_{2}$ is initialized from $\theta_{1}$, and then $\theta_{3}$ is initialized from $\theta_{2}$, in the same way as weight-transfer. In this analysis, each loss and accuracy is evaluated at a series of points $\theta=\theta_{1}+\alpha(\theta_{2}-\theta_{1})+\beta(\theta_{3}-\theta_{2})$, varying $\alpha$ and $\beta$. In Figure 2, the loss surface of the model on each online subset is nearly convex. The figure shows that the parameter at $\frac{1}{3}(\theta_{1}+\theta_{2}+\theta_{3})$, which is the same as the solution of mean-IMM, performs better than any other reference points $\theta_{1}$, $\theta_{2}$, or $\theta_{3}$. However, when $\theta_{2}$ is not initialized by $\theta_{1}$, the convex-like shape disappears, since there is a high cost barrier between the loss function of $\theta_{1}$ and $\theta_{2}$.

2 L2-transfer

L2-transfer is a variant of L2-regularization. L2-transfer can be interpreted as a special case of EWC where the prior distribution is Gaussian with $\lambda I$ as a covariance matrix. In L2-transfer, a regularization term of the distance between $\mu_{k-1}$ and $\mu_{k}$ is added to the following objective function for finding $\mu_{k}$, where $\lambda$ is a hyperparameter:

The concept of L2-transfer is commonly used in transfer learning and continual learning with large $\lambda$. Unlike the previous usage of large $\lambda$, we use small $\lambda$ for the IMM procedure. In other words, $\mu_{k}$ is first trained by Equation 10 with small $\lambda$, and then merged to $\mu_{1:k}$ in our IMM. Since we want to make the loss surface between $\mu_{k-1}$ and $\mu_{k}$ smooth, and not to minimize the distance between $\mu_{k-1}$ and $\mu_{k}$. In convex optimization, the L2-regularizer makes the convex function strictly convex. Similarly, we hope L2-transfer with small $\lambda$ help to find a $\mu_{k}$ with a convex-like loss space between $\mu_{k-1}$ and $\mu_{k}$.

3 Drop-transfer

Drop-transfer is a novel method devised in this paper. Drop-transfer is a variant of dropout where $\mu_{k-1}$ is the zero point of the dropout procedure. In the training phase, the following $\hat{\mu}_{k,i}$ is used for the weight vector corresponding to the $i$th node $\mu_{k,i}$:

where $p$ is the dropout ratio. Notice that the expectation of $\hat{\mu}_{k,i}$ is $\mu_{k,i}$.

There are studies that have interpreted dropout as an exponential ensemble of weak learners. By this perspective, since the marginalization of output distribution over the whole weak learner is intractable, the parameters multiplied by the inverse of the dropout rate are used for the test phase in the procedure. In other words, the parameters of the weak learners are, in effect, simply averaged oversampled learners by dropout. At the process of drop-transfer in our continual learning setting, we hypothesize that the dropout process makes the averaged point of two arbitrary sampled points using Equation 11 a good estimator.

We investigated the search space of the loss function of the MLP trained from the MNIST handwritten digit recognition dataset for with and without dropout regularization, to supplement the evidence of the described hypothesis. Dropout regularization makes the accuracy of a sampled point from dropout distribution and an average point of two sampled parameters, from 0.450 ($\pm$ 0.084) to 0.950 ($\pm$ 0.009) and 0.757 ($\pm$ 0.065) to 0.974 ($\pm$ 0.003), respectively. For the case of both with and without dropout, the space between two arbitrary samples is empirically convex, and fits to the second-order equation. Based on this experiment, we expect not only that the search space of the loss function between modern neural networks can be easily nearly convex , but also that regularizers, such as dropout, make the search space smooth and the point in the search space have a good accuracy in continual learning.

Experimental Results

We evaluate our approach on four experiments, whose settings are intensively used in the previous works . For more details and experimental results, see Appendix D. The source code for the experiments is available in Github repository https://github.com/btjhjeon/IMM_tensorflow.

Disjoint MNIST Experiment. The first experiment is the disjoint MNIST experiment . In this experiment, the MNIST dataset is divided into two datasets: the first dataset consists of only digits {0, 1, 2, 3, 4} and the second dataset consists of the remaining digits {5, 6, 7, 8, 9}. Our task is 10-class joint categorization, unlike the setting in the previous work, which considers two independent tasks of 5-class categorization. Because the inference should decide whether a new instance comes from the first or the second task, our task is more difficult than the task of the previous work.

We evaluate the models both on the untuned setting and the tuned setting. The untuned setting refers to the most natural hyperparameter in the equation of each algorithm. The tuned setting refers to using heuristic hand-tuned hyperparameters. Consider that tuned hyperparameter setting is often used in previous works of continual learning as it is difficult to define a validation set in their setting. For example, when the model needs to learn from the new task after learning from the old task, a low learning rate or early stopping without a validation set, or arbitrary hyperparameter for balancing is used . We discover hyperparameters in the tuned setting not only to find the oracle performance of each algorithm, but also to show that there exist some paths consisting of the point that performs reasonably for both tasks. Hyperparam in Table 1 denotes hyperparameter mainly searched in the tuned setting. Table 1 (Top) and Figure 3 (Left) shows the experimental results from the disjoint MNIST experiment.

In our experimental setting, the usual SGD-based optimizers always perform less than 50%, because the biases of the output layer for the old task are always pushed to large negative values, which implies that our task is difficult. Figure 4 also shows that mode-IMM is robust with $\alpha$ and the optimal $\alpha$ of mean-IMM is larger than $1/2$ in the disjoint MNIST experiment.

Shuffled MNIST Experiment. The second experiment is the shuffled MNIST experiment of three sequential tasks. In this experiment, the first dataset is the same as the original MNIST dataset. However, in the second dataset, the input pixels of all images are shuffled with a fixed, random permutation. In previous work, EWC reaches the performance level of the batch learner, and it is argued that EWC overcomes catastrophic forgetting in some domains. The experimental details are similar to the disjoint MNIST experiment, except all models are allowed to use dropout regularization. In the experiment, the first dataset is the same as the original MNIST dataset. However, in the second and the third dataset, the input pixels of all images are shuffled with a fixed, random permutation, respectively. Therefore, the difficulty of the three datasets is the same, though a different solution is required for each dataset.

Table 1 (Bottom) and Figure 3 (Middle) shows the experimental results from the shuffled MNIST experiment. Notice that accuracy of drop-transfer ($p$ = 0.2) alone is 96.86 ($\pm$ 0.21) and L2-transfer ($\lambda$ = 1e-4) + drop-transfer ($p$ = 0.4) alone is 97.61 ($\pm$ 0.15). These results are competitive to EWC without dropout, whose performance is around 97.0.

ImageNet to CUB Dataset. The third experiment is the ImageNet2CUB experiment , the continual learning problem from the ImageNet dataset to the Caltech-UCSD Birds-200-2011 fine-grained classification (CUB) dataset . The numbers of classes of ImageNet and CUB dataset are around 1K and 200, and the numbers of training instances are 1M and 5K, respectively. In the ImageNet2CUB experiment, the last-layer is separated for the ImageNet and the CUB task. The structure of AlexNet is used for the trained model of ImageNet . In our experiment, we match the moments of the last-layer fine-tuning model and the LwF model, with mean-IMM and mode-IMM.

Figure 3 (Right) shows that mean-IMM moderately balances the performance of two tasks between two networks. However, the balanced hyperparameter of mode-IMM is far from $\alpha$ = 0.5. We think that it is because the scale of the Fisher matrix $F$ is different between the ImageNet and the CUB task. Since the number of training data of the two tasks is different, the mean of the square of the gradient, which is the definition of $F$, tends to be different. This implies that the assumption of mode-IMM does not always hold for heterogeneous tasks. See Appendix D.3 for more information including the learning methods of IMM where a different class output layer or a different scale of the dataset is used.

Our results of IMM with LwF exceed the previous state-of-the-art performance, whose model is also LwF. This is because, in the previous works, the LwF model is initialized by the last-layer fine-tuning model, not directly by the original AlexNet. In this case, the performance loss of the old task is not only decreased, but also the performance gain of the new task is decreased. The accuracies of our mean-IMM ($\alpha$ = 0.5) are 56.20 and 56.73 for the ImageNet task and the CUB task, respectively. The gains compared to the previous state-of-the-art are +1.13 and -1.14. In the case of mean-IMM ($\alpha$ = 0.8) and mode-IMM ($\alpha$ = 0.99), the accuracies are 55.08 and 59.08 (+0.01, +1.12), and 55.10 and 59.12 (+0.02, +1.35), respectively.

Lifelog Dataset. Lastly, we evaluate the proposed methods on the Lifelog dataset . The Lifelog dataset consists of 660,000 instances of egocentric video stream data, collected over 46 days from three participants using Google Glass . Three class categories, location, sub-location, and activity, are labeled on each frame of video. In the Lifelog dataset, the class distribution changes continuously and new classes appear as the day passes. Table 2 shows that mean-IMM and mode-IMM are competitive to the dual-memory architecture, the previous state-of-the-art ensemble model, even though IMM uses single network.

Discussion

A Shift of Optimal Hyperparameter of IMM. The tuned setting shows there often exists some $\alpha$ which makes the performance of the mean-IMM close to the mode-IMM. However, in the untuned hyperparameter setting, mean-IMM performs worse when more transfer techniques are applied. Our Bayesian interpretation in IMM assumes that the SGD training of the $k$-th network $\mu_{k}$ is mainly affected by the $k$-th task and is rarely affected by the information of the previous tasks. However, transfer techniques break this assumption; thus the optimal $\alpha$ is shifted to larger than $1/k$. Fortunately, mode-IMM works more robustly than mean-IMM where transfer techniques are applied. Figure 4 illustrates the change of the test accuracy curve corresponding to the applied transfer techniques and the following shift of the optimal $\alpha$ in mean-IMM and mode-IMM.

Bayesian Approach on Continual Learning. Kirkpatrick et al. interpreted that the Fisher matrix $F$ as weight importance in explaining their EWC model. In the shuffled MNIST experiment, since a large number of pixels always have a value of zero, the corresponding elements of the Fisher matrix are also zero. Therefore, EWC does work by allowing weights to change, which are not used in the previous tasks. On the other hand, mode-IMM also works by selectively balancing between two weights using variance information. However, these assumptions on weight importance do not always hold, especially in the disjoint MNIST experiment. The most important weight in the disjoint MNIST experiment is the bias term in the output layer. Nevertheless, these bias parts of the Fisher matrix are not guaranteed to be the highest value nor can they be used to balance the class distribution between the first and second task. We believe that using only the diagonal of the covariance matrix in Bayesian neural networks is too naïve in general and that this is why EWC failed in the disjoint MNIST experiment. We think it could be alleviated in future work by using a more complex prior, such as a matrix Gaussian distribution considering the correlations between nodes in the network .

Balancing the Information of an Old and a New Task. The IMM procedure produces a neural network without a performance loss for $k$th task $\mu_{k}$, which is better than the final solution $\mu_{1:k}$ in terms of the performance of the $k$th task. Furthermore, IMM can easily weigh the importance of tasks in IMM models in real time. For example, $\alpha_{t}$ can be easily changed for the solution of mean-IMM $\mu_{1:k}=\sum^{k}_{t}\alpha_{t}\mu_{t}$ . In actual service situations of IT companies, the importance of the old and the new task frequently changes in real time, and IMM can handle this problem. This property differentiates IMM from the other continual learning methods using the regularization approach, including LwF and EWC.

Conclusion

Our contributions are four folds. First, we applied mean-IMM to the continual learning of modern deep neural networks. Mean-IMM makes competitive results to comparative models and balances the information between an old and a new network. We also interpreted the success of IMM by the Bayesian framework with Gaussian posterior. Second, we extended mean-IMM to mode-IMM with the interpretation of mode-finding in the mixture of Gaussian posterior. Mode-IMM outperforms mean-IMM and comparative models in various datasets. Third, we introduced drop-transfer, a novel method proposed in the paper. Experimental results showed that drop-transfer alone performs well and is similar to the EWC without dropout, in the domain where EWC rarely forgets. Fourth, We applied various transfer techniques in the IMM procedure to make our assumption of Gaussian distribution reasonable. We argued that not only the search space of the loss function among neural networks can easily be nearly convex, but also regularizers, such as dropout, make the search space smooth, and the point in the search space have good accuracy. Experimental results showed that applying transfer techniques often boost the performance of IMM. Overall, we made state-of-the-art performance in various datasets of continual learning and explored geometrical properties and a Bayesian perspective of deep neural networks.

Acknowledgments

The authors would like to thank Jiseob Kim, Min-Oh Heo, Donghyun Kwak, Insu Jeon, Christina Baek, and Heidi Tessmer for helpful comments and editing. This work was supported by the Naver Corp. and partly by the Korean government (IITP-R0126-16-1072-SW.StarLab, IITP-2017-0-01772-VTT, KEIT-10044009-HRI.MESSI, KEIT-10060086-RISF). Byoung-Tak Zhang is the corresponding author.

References

APPENDIX A. Modes in the Mixture of Gaussian

According to Ray and Lindsay , all the critical points $\theta$ of a mixture of Gaussian (MoG) with two components are in one curve as the following equation with $0<\alpha<1$.

The proof is as follows. Imagine two Gaussian distribution $q_{1}$ and $q_{2}$, such as in Equation 2.

$D$ is the dimension of the Gaussian distribution. Mixture of two Gaussian $q_{1}$ and $q_{2}$ with the equal mixing ratio (i.e., 1:1) is $q_{1}/2+q_{2}/2$. The derivation of the MoG is as follows:

If we set Equation 15 to 0, to find all critical points, the following equation holds:

When $\alpha$ is set to $\frac{q_{2}}{q_{1}+q_{2}}$, Equation 12 holds.

Note that $\alpha_{k}$ is a function of $\theta$, so $\theta$ cannot be calculated in a closed-form from Equation 16. However, the optimal $\theta$ is in the set $\{\theta|\theta=((1-\alpha)\Sigma^{-1}_{1}+\alpha\Sigma^{-1}_{2})^{-1}((1-\alpha)\Sigma^{-1}_{1}\mu_{1}+\alpha\Sigma^{-1}_{2}\mu_{2}),0<\alpha<1\}$, which motivates our mode-IMM method.

In our study IMM uses diagonal covariance matrices, which means that there is no correlation between parameters. This diagonal assumption is useful, since it decreases the number of parameters for each covariance matrix from $O(D^{2})$ to $O(D)$. Based on this, the $\theta$ in Equation 12 is defined as follows:

$v$ denotes an index of the parameter vector. $\mu_{\cdot,v}$ and $\sigma^{2}_{\cdot,v}$ are scalar.

For MoG with two components in $K$ dimension, the number of modes can be at most $K+1$ . Therefore, it is hard to find all modes in high-dimensional Gaussian in general.

The property of critical points of a MoG with two components can be extended to the case of $K$ components. The following equation holds:

where $0<\alpha_{k}<1$ for all $k$ and $\sum_{k}\alpha_{k}=1$. There is no tight upper bound on the number of modes of MoG in general. There is a guess that, for all $D,K\geq 1$, the upper bound is ${}_{(D+K-1)}C_{D}$ .

APPENDIX B. Bayesian Neural Networks and Continual Learning

Bayesian Neural Networks. Bayesian neural networks (BNN) assume an uncertainty for the whole parameter in neural networks so that the posterior distribution can be obtained . Previous studies have argued that BNN regularizes better than NN, and provides a confidence interval for the output estimation of each input instance. Current research on BNN, to the best of our knowledge, uses Gaussian distributions as the posteriors of the parameters. In the Gaussian assumption, because tracking the entire information of a covariance matrix is too expensive, researchers usually use only the diagonal term for the covariance matrix, where the posterior distribution is fully factorized for each parameter. However, the methods using full covariance were also suggested recently . To estimate a covariance matrix most studies use stochastic gradient variational Bayes (SGVB), where a sampled point from the posterior distribution by Monte Carlo is used in the training phases . Alternatively, Kirkpatrick et al. approximated the covariance matrix as an inverse of a Fisher matrix. This approximation makes the computational cost of the inference of a covariance matrix cheaper when the update of covariance information is not needed in the training phase. Our method follows the approach using the Fisher matrix.

Elastic Weight Consolidation. We compare the work of Kirkpatrick et al. to the results of our framework. The mechanism of EWC follows sequential Bayesian estimation. EWC maximizes the following terms by gradient descent to get the solution $\mu_{1:K}$.

When moving to a third task, EWC uses the penalty term of both first and second network (i.e., $\mu_{1}$ and $\mu_{1:2}$). Although this heuristic works reasonably in the experiments in their paper, it does not match to the philosophy of Bayesian.

Learning without Forgetting. We compare the work of Li and Hoiem . Although LwF does not explicitly assume Bayesian, the approach can be represented nonetheless as follows:

Where $\hat{y_{k}}$ is the output from $\mu_{k}$ with input $X_{K}$. This framework is promising where the properties of a pseudo training set of $k$th task ($X_{K}$, $\hat{y}_{k}$) is similar to the ideal training set ($X_{k}$, $y_{k}$).

APPENDIX C. Example Algorithms of Incremental Moment Matching

Two moment matching methods: mean-IMM and mode-IMM, and three transfer learning techniques: weight-transfer, L2-transfer, and drop-transfer, are combined to make various continual learning algorithms in our study. Algorithm 1 describes mean-IMM and mode-IMM with weight-transfer and L2-transfer.

APPENDIX D. Experimental Details

Appendix D further explains following issues, 1) additional explanation of the untuned setting and tuned setting 2) techniques for IMM with a different class output layer for each task 3) other experimental details.

We first explain the untuned setting and the tuned setting in detail. The untuned setting refers to the most natural hyperparameter in the equation of each algorithm, whereas the tuned setting refers to using heuristic hand-tuned hyperparameters. For mean-IMM, it is most natural to evenly average K models and $1/K$ is the most natural $\alpha_{k}$ value for $K$ sequential tasks. For EWC, $1$ is the most natural $\lambda$ value in Equation 19, because EWC is derived from the equation of sequential Bayesian. For L2-transfer, there is no natural hyperparameter value in Equation 10, so we need to heuristically choose a $\lambda$ value, which is not too small but does not damage the performance of the new network for the new task.

In the SGD, the number of epochs for the dataset (epoch per dataset) for the second task corresponds to the hyperparameter. The unit is how much of the network is trained from the whole data at once. In the L2-transfer and EWC, $\lambda$ in Equations 10 and 19 corresponds to their hyperparameter. In the mean-IMM and mode-IMM, $\alpha_{K}$ in Equations 4 and 7 corresponds to the hyperparameter. In the drop-transfer, dropout ratio $p$ in Equation 11 corresponds to the hyperparameter.

All of the explained hyperparameters are devised to balance the information between the old and new tasks. If $\lambda/(1+\lambda)=1$ or $\alpha_{1}=1$, the final network of the algorithms is the same as the network for the first task. If $1/(1+\lambda)=1$ or $\alpha_{K}=1$, the final network is the same as the network for the last task.

We used multi-layer perceptrons (MLP) with as the number of nodes, ReLU as the activation function, and vanilla SGD as the optimizer for all of the experiments. We set the epoch per dataset to 10, unless otherwise noted. The entire IMM model uses weight-transfer to smooth the loss surface of the model. Without weight-transfer, our IMM model does not work at all. In our experiments, all models only use one 10-way softmax output layer. For only SGD, dropout is used as proposed in Goodfellow et al. , but dropout does not help much.

Each accuracy was measured by averaging the results of 10 experiments. In the experiment, IMM outperforms comparative models by a significant margin. In the tuned experiment, the performance of the IMM models exceeds 90%, and the performance increases more when more transfer techniques are applied. Among all the models, weight-transfer + L2-transfer + drop-transfer + mode-IMM performs the best and its performance is greater than 94%. However, the comparative models fail to reach greater than 90%. Existing regularizer including dropout does not improve the comparative models.

D.2 Shuffled MNIST Experiment

The second experiment is the shuffled MNIST experiment for three sequential tasks. For the hyperparameter of IMM, we set $\alpha_{1}$ and $\alpha_{2}$ as the same value, and tune only $\alpha_{3}$. Table 1 (Bottom) shows the experimental results. The performance of SGD + dropout and EWC + dropout comes from the report in . Changing only the epoch does not significantly increase the performance in SGD. The results show that our IMM paradigm performs similarly to EWC in a case where EWC performs well. Dropout regularization in the task makes both our models and comparative models perform better.

In our IMM framework, weight-transfer, L2-transfer, and drop-transfer all take $\mu_{k-1}$ as the reference models of the transfer for training $\mu_{k}$. In other words, weight-transfer initializes $\mu_{k}$ with $\mu_{k-1}$, L2-transfer uses a regularization term to minimize the Euclidean distance between $\mu_{k-1}$ and $\mu_{k}$, and drop-transfer uses a $\mu_{k-1}$ as the zero point of the dropout procedure. All three transfer techniques can be considered to change the reference point to, for example, $\mu^{mean}_{1:(k-1)}$ or $\mu^{mode}_{1:(k-1)}$, as previous works do . However, all these alternatives make performances worse in our shuffled MNIST experiment. We argued that our utilization of transfer techniques is devised not to minimize the distance between $\mu_{k-1}$ and $\mu_{k}$, but to help find a $\mu_{k}$ with a smooth and convex-like loss space between $\mu_{k-1}$ and $\mu_{k}$.

D.3 ImageNet to Other Image Datasets

When each task needs a different class output layer, IMM requires additional techniques. There is no counterpart weight matrix in the last-layer of the first network representing the second task, nor the second last-layer of the first network. To tackle this problem, we add the training process of the last-layer fine-tuning model to the IMM procedure; we match the moments of the last-layer fine-tuning model with the original new network for the new task. Last-layer fine-tuning is the model the last-layer is only fine-tuned for each new task; thus it does not make a performance loss for the first task, but does not often learn enough for new tasks.

The technique utilizing the last-layer fine-tuning model makes mean-IMM work in the case of different class output layers, but it is not enough for mode-IMM. It is not possible to calculate a proper Fisher matrix of the second last-layer in the first network for the first dataset. As the Fisher matrix is defined with the gradient from the loss of the first task, elements of the Fisher matrix have a value of zero. However, a zero matrix not only is what we do not want but also degenerates the performance of mode-IMM. To tackle this problem, we apply mean-IMM for the last-layer with a re-scaling. We change the mixing ratios $\alpha_{1}$ : $\alpha_{2}$ to $\hat{\alpha}_{1}$ : $\hat{\alpha}_{2}$ = $\alpha_{1}$ : $\alpha_{2}\cdot\frac{|\hat{w}_{1}|}{|\hat{w}_{1}|+|\hat{w}_{2}|}$ for the re-scaling, where $|\hat{w}_{1}|$ and $|\hat{w}_{2}|$ is the average of the whole element of weight matrix in the layer before the last-layer, in the first and the second task.

In our ImageNet2CUB experiment, the moments of the last-layer fine-tuning model and the LwF model are matched. Though LwF does not perform well in our previous experiments, it is known that LwF performs well when the size of a new dataset is small relative to the old dataset, as in the ImageNet2CUB experiment.

Figure 5 (Left) compares the performances of mode-IMM models with different assumptions on the Fisher matrix. In naïve mode-IMM, the Fisher matrix of the second last-layer of the first network is a zero matrix. In other words, the second last-layer of the final naïve mode-IMM is the second last-layer of the second network. Naïve mode-IMM does not yield a good performance as we expect.

In Figure 5, scaled mode-IMM denotes the results of mode-IMM re-plotted by the $\hat{\alpha}$ as we defined above. The result shows that re-scaled mode-IMM performs similarly to mean-IMM in the ImageNet2CUB experiment.

D.4 Lifelog Dataset

The Lifelog dataset is the dataset recorded from Google Glass over 46 days from three participants. The 660,000 seconds of the egocentric video stream data reflects the behaviors of the participants. The dataset consists of 10 days of training data and 4 days of test data in order of time for each participant respectively. In the framework of Lee et al. , the network can be updated every day, but a new network can be made for the 3rd, 7th, and 10th day, with training data of 3, 4, and 3 days, respectively. Following this framework, our network is made in the 3rd, 7th, and 10th day, and then merged to previously trained networks. Our IMM used AlexNet pretrained by the ImageNet dataset as the initial network. The experimental results on the Lifelog dataset are in Table 3, where the performance of models is from Lee et al. except IMM.