A Survey on Green Deep Learning

Chapter 1 Introduction

Deep learning, based on deep neural networks, is part of machine learning methods. In this chapter, we first introduce the development of deep learning in section 1.1. Then, we elucidate what is Green deep learning, why Green deep learning matters, and how to evaluate the “greenness” of deep learning in section 1.2.

While a decade ago, artificial intelligence (AI) mainly focuses on shallow models, like structure perceptrons (McDonald et al., 2010; Huang et al., 2012; Li & Ji, 2014) and conditional random fields (Ghosh et al., 2011; Sutton & McCallum, 2012; Zheng et al., 2015). These shallow models only require limited computations. Most AI approaches can be deployed on CPUs.

In recent years, powerful GPUs have become increasingly accessible, making it possible to deploy larger models, which accelerates the development of deep learning. The ideas of widely-used deep learning models have been proposed in the 1990s, such as convolutional neural networks (CNNs) (LeCun et al., 1998) and long-short term networks (LSTMs) (Hochreiter & Schmidhuber, 1997). Confined by hardware capacity and large-scale data resources, these models began to be popular until the past few years. Collobert et al. (2011) proposed the first systematic deep learning framework for NLP tasks. Krizhevsky et al. (2012) proposed a convolution-based deep network, which ranked the first in the image classification challenge. These studies are good pioneers that motivate AI participants to dive into deep learning.

Deep learning methods currently have become a prime choice for AI. The promising prospect of neural networks attracts more AI participants to engage with deep learning, in the meanwhile, deep learning is highly competitive in yielding profits when applied to real-world applications. The industry continuously develops more efficient hardware and launches better programming platforms, such as Theano, Caffe, MxNet, Tensorflow, and Pytorch. Advanced infrastructures further enable AI participants to develop stronger deep models. Therefore, deep learning takes a high-speed train since 2010. To visualize the transition process from shallow models to deep networks in the AI community, we analyze papers from a top-tier AI conference, ACL, starting from 2010 to 2020. We randomly select 50 works from each year and manually count the number of papers using deep learning. As we can see from Figure 1.1, the number of papers using neural networks is growing fast from 2012 to 2020. All papers adopt deep networks as backbone since 2020. From then, the AI field is fully entered into the age of deep learning.

In the age of deep learning, a hot direction is to obtain SOTA results. Following Schwartz et al. (2020a), we call such research trend as Red AI. Recently, researchers have noticed that it was harder to gain an advantage over SOTA results. For traditional AI fields, like CV and NLP, the improvements achieved by new AI models/algorithms are diminishing. Many popular research benchmarks are reaching their performance ceiling. Figure 1.2 and 1.3 list several examples showing how the returns of deep learning are diminishing over time.

The trend of red AI requires massive computations to achieve better results. For example, as reported in Schwartz et al. (2020a), the amount of computations used to train deep learning models has increased 300,000x in 6 years. These computations not only cause expensive financial costs but also contribute to an excessive carbon footprint. The former harms AI inclusiveness and the latter harms our environment. We classify the computation source required by deep learning into the following three categories: model size, parameter tuning, and training data.

1) With access to large-scale data resources, increasing model size is the simplest way to improve results. For example, on WMT English-German translation, the performance can be increased from 27.3 to 28.4 while the size of the machine translation models is increased from 60M (Transformer-base) to 180M (Transformer-large). To achieve better results, more and more AI participants would like to increase model size as much as possible, especially for rich organizations. For example, researchers from OpenAI first pre-trained a large-scale text generation model, called GPT-3 with 175B parameters. It shows that a super-large model can generate human-like texts. However, according to Strubell et al. (2019), training GPT-3 can emit almost 500M carbons, almost emissions of five cars in their lifetime. These studies are key milestones in a long run. However, we believe that larger models are not always better if we consider “invisible” computation cost. We are still concerned about “the crazy love” to super-large models no matter whether the increased computations bring significant benefits. In addition, bigger models largely increase the burden of inference serving. Amazon estimates that 90% of production ML infrastructure costs are for inference, not training (Jain et al., 2019).

2) Model experiments are also an overlooked computation consumer. To verify the effectiveness of a new model/algorithm, AI participants usually conduct massive experiments, including model/algorithm implementation, baseline re-implementation, and hyper-parameter tuning. First, baseline re-implementation is an redundant computation source. For example, the original Transformer paper has 18K citations. Assume each citation represents a single implementation. Each re-implementation takes 100 hours on a single GPU (following the cost of running a Transformer-base model on English-German translation). It means that only baseline re-implementation on a single dataset can take 1.8M GPU hours. In addition, hyper-parameter tuning is an overlooked computation source. We design a simple questionnaire to ask the ratio of experiments experiments for hyper-parameter tuning while developing a new model/algorithm, answered by 64 AI specialists, including researchers and engineers. All of the surveyed choose tuning hyper-parameters. To be specific, 10.7%, 32.1%, 35.7%, 21.4% individuals take 80%-100%, 50%-80%, 30%-50%, 0%-30% experiments to tune hyper-parameters. In sum, 42.8% individuals spend over 50% experiments for hyper-parameter tuning.

3) Starting from shallow models, it is popular to increase the amount of training data to achieve better generalization ability, especially in semi-supervised settings. One recent hot topic is pre-training a super-large model on billions of raw data. In the NLP field, ELMo (Peters et al., 2018) is the first well-known work to explore large-scale pre-training. Following ELMo, BERT pre-trains a Transformer encoder on 3 billion word pieces. Researchers from OpenAI recently proposed GPT-3, a generative model pre-trained on 45TB data. These massive training examples largely increase the training costs compared to previous shallow models.

In all, the trend of Red AI brings heavy computation costs. These computations exacerbate the research inequality, making it difficult to involve all researchers in Red AI. Furthermore, massive computation requirements bring huge carbon emissions. To address these problems, Green deep learning, or Green AI, was first proposed by Schwartz et al. (2020a) to encourage the AI community to focus more on energy costs.

2 Green Deep Learning

In this section, we mainly describe what is Green deep learning, how to evaluate “greenness” in deep learning, and why Green deep learning matters.

Green learning, a term first proposed by Schwartz et al. (2020a), is gaining mounting attention. Formally, Green deep learning, or Green AI, appeals to researchers to obtain novel results without increasing computational cost rather, ideally reducing it. Unlike Red AI pushing state-of-the-art results at any cost, Green deep learning encourages AI participants to achieve comparable or better results using as few computations as possible.

2.2 Measure

In Green deep learning, computations are important evaluation metrics. Currently, the whole community lacks a comprehensive and widely-accepted measure to evaluate computations because multiple aspects can attribute to computations, including model size, training examples, and so on. A comprehensive measure is expected for a fair comparison. Here we list several computation measures and discuss their merits and demerits.

Some studies adopt the total training time as a kind of computations measure. If all models/algorithms adopt the same hardware and software settings, it is the most natural measure to evaluate training/inference computations. However, since running time heavily relies on infrastructure settings, it is not suitable for comparing models running on different infrastructures. Even so, we still encourage AI participants to report the running time for an intuitive understanding.

Carbon emission is the most direct approach to evaluate environmental effects. In order to quantify carbon emissions, Lacoste et al. (2019) used $CO_{2}$-equivalents ($CO_{2}eq$) as the amount of $CO_{2}$ which would have the equivalent global warming impact. However, the main challenge of this measure lies in accurate estimation. First, computations via electricity consumption are easily influenced by local infrastructures. Furthermore, it is hard for AI participants to estimate the amount of $CO_{2}$ if they do not run experiments on well-known cloud platforms. Therefore, it is also not suitable as a standard metric to compare different models running on different regions and different computing infrastructures.

The model size is also an important factor in deciding training and inference costs. We encourage researchers to report model size to corporate with other measures in practice.

Floating-Point Operations (FLOPs) count the number of works required for running a model when executing a specific instance. Previous studies usually adopt this metric to evaluate efficiency. FLOPs are almost independent of hardware and software platforms, being the simplest measure to conduct a fair comparison between different models. However, FLOPs are theoretical values, and there is a gap between FLOPs and running time. In addition to the total amount of works (FLOPs), the degree of parallelism also affects the running time.

According to these measures, we summarize evaluation strategies towards fair comparison and intuitive understanding.

Generally speaking, AI participants prefer well-performing models/algorithms with fewer computations. Therefore, it is an important question to fairly compare computations required for training and inference. We strongly suggest reporting FLOPs during model training and inference. Last but not least, to evaluate the wasted and redundant computations required for developing a new model/algorithm, we also encourage researchers to report the total FLOPs during all experiments, including but not limited to parameter tuning and baseline implementation.

To increase the intuitively understanding about computations, we encourage researchers to report running time, model size, and carbon emission as optional results.

2.3 Broader Impact

First, Green deep learning can help deep learning empower real society applications better. In the past decade, deep learning continuously pushed state-of-the-art results on research benchmarks, like machine translation, image classification, and so on. In the research community, the cost of deep learning seems to be nothing compared to energy consumption of all human activities. Nowadays, deep learning is widely applied to real society tasks, such as auto-driving, face recognition, drug discovery, and so on. Once deep learning is involved in large-scale applications, the cost of deep learning will be multiplied hundreds of millions of times. Furthermore, some edge applications, like mobiles with extremely few computation resources, also require Green deep learning. Therefore, Green deep learning is a necessary research direction in the future.

Second, Green deep learning can largely improve AI inclusiveness. We note the contributions of rich organizations for pushing higher results on many downstream tasks. Meanwhile, we also notice the dilemma of researchers from academics and developing countries on engaging Red AI research. Most researchers only have limited computations, which could not support them to develop super-large models with state-of-the-art results. Unfortunately, compared to ideas with state-of-the-art results, novel and innovative ideas without state-of-the-art results are losing their sounds. For example, news media would like to report studies with state-of-the-art results. The attractive propaganda of No.1 also pushes the rich organizations to pour more money on super-big models. These cases may confuse researchers on how they engage in deep learning research without strong financial support. We argue that state-of-the-art results are good, but not the only criteria to evaluate the quality of new models/algorithms. We encourage rich organizations to continuously explore data and model boundaries, also encourage the AI community to pay attention to innovative ideas. In fact, deep learning struggles many years until it outperforms shallow models with feature engineers. We believe that the development of AI should be diverse. Green deep learning can improve AI inclusion and motivate more AI participants to explore deep learning possibilities.

3 Outline of the Survey

It is a long-term goal to develop tiny yet strong networks for all AI researchers and engineers. Driven by this target, several popular tiny networks have been proposed (Howard et al., 2017; Chollet, 2017; Tan & Le, 2019). For example, MobileNet proposed by Howard et al. (2017) is an efficient architecture based on depthwise separable convolution. Similar idea has been adopted at Xception (Chollet, 2017). Recently, to explore extremely tiny networks, advanced training/inference/network surgery methods have been proposed. For example, EdgeBERT (Tambe et al., 2021) is proposed to build an extremely tiny network that can run on IoT devices. It adopts advanced methods like quantization, pruning, early exit to further reduce model parameters and running computations.

In this survey, we give a systematic review of Green deep learning technologies. We first build a green technology taxonomy and then classify the related technologies into four categories, including compact networks, energy-saving training strategies, energy-saving inference, and efficient data usage. In each category, we review the current progress on Green technologies and explore potential issues.

It is important to note that building a Green technology taxonomy is challenging since there lacks a unified standard measurement. For example, BERT requires massive computations during training. If we only consider training costs, BERT can not be treated as a Green technology. However, BERT can improve downstream performance with fewer training examples. If we consider its transfer ability, BERT is absolutely a Green technology. Therefore, whether a technology is defined as Green or not is open to doubt. We will try our best to avoid giving a biased definition. If a technology has the potential to reduce the costs of deep learning, we will include it in the green technology taxonomy. We review Green deep learning technologies in the following categories:

Compact Architecture Design. This part focuses on small networks. We split this chapter into two sub-chapters, i.e., component design, and component assembling. The component design focuses on subtle components with competitive results but much fewer computations. Component assembling describes how to build a network efficiently.

Energy-efficient Training Strategies. Previous studies have proposed several efficient training approaches. In this survey, we classify these studies into four categories, including initialization, normalization, progressive training, and efficient AutoML.

Energy-efficient Inference. In this chapter, we describe approaches that aim to get a smaller yet comparable network from a larger network for efficient inference, including model pruning, low-rank factorization, quantization, distillation.

Efficient Data Usage. This chapter lists algorithms that leverage training data efficiently. We focus on two popular directions: active learning and pre-trained models as few-shot learners.

Chapter 2 Compact Architecture

Developing efficient neural networks has been a long-standing goal towards Green AI. In this survey, we define Green networks as neural networks that are efficient in terms of computational costs. We can generate compact networks via subtle design, model surgery, and network search. Subtle design means that we can manually define efficient architectures requiring fewer computations. Model surgery means that we can generate compact architectures from a larger model via parameter reduction. In this chapter, we focus on architectures with subtle design and leave the details of network surgery to Chapter 4. An overview of this section is shown in Figure 2.1.

In this section, we describe efficient variants of popular components, including convolution, attention, softmax, and embedding.

Starting from AlexNet (Krizhevsky et al., 2012), it has been a hot direction to build deeper and larger CNNs to achieve better performance (Iandola et al., 2014; Simonyan & Zisserman, 2015; Szegedy et al., 2015; He et al., 2016a; Szegedy et al., 2017). Currently, even a simple CNN baseline contains hundreds of layers and thousands of channels. To reduce deployment costs, previous studies proposed efficient variants. Here we list several widely-used variants.

This architecture has been adopted in Xception (Chollet, 2017) and MobileNet (Howard et al., 2017). Depthwise separable convolution contains two components: depthwise convolution and pointwise convolution. The depthwise convolution applies a single filter for each input channel. The pointwise convolution is a kind of $1\times 1$ convolution. Following this research line, many advanced variants have been proposed (Hoang & Jo, 2018; Sandler et al., 2018). For example, Wang et al. (2017) also proposed a factorized convolution by unravelling the standard convolution and arranging the spatial convolution sequentially.

Iandola et al. (2016) proposed a vision model SqueezeNet. The fire module is the key building block. It is similar with depthwise separable convolutions. A fire module contains two components: a squeeze convolution layer with $1\times 1$ filters and an expand layer with a mixture of $1\times 1$ and $3\times 3$ convolution filters.

It is proposed by Jin et al. (2015) to decrease the redundancy of the filters. It separates the 3D convolution filters into three consecutive 1D filters: convolution across channels (lateral), vertical, and horizontal direction.

Traditional convolutions usually have fixed hyper-parameter settings, like the number of filters. Different from these models, MobileNet (Howard et al., 2017) adopts a dynamic setting, also called shrinked convolution. It introduces a width multiplier to thin a network uniformly at each layer. The standard convolutions with $M$ input channels and $N$ output channels become a shrinked convolution with $\alpha M$ input channels and $\alpha N$ output channels where $\alpha\leq 1$.

1.2 Efficient Attention

Attention (Bahdanau et al., 2015) is first proposed for handling long-distance dependency in machine translation. Currently, it has been widely used in tasks like summarization, natural language understanding, and so on. The key idea is to dynamically attend all tokens at each step. All tokens can be directly aligned together, which can address long-distance dependencies to some extent. Since any two tokens have an attention score, the required computations grow quadratically with the input length. To address this problem, previous studies proposed several efficient attention variants. Currently, dop-product-based attention (Vaswani et al., 2017) becomes the dominant choice for NLP and CV applications. For simplification, we refer to attention as dot-product attention.

We roughly classify these variants into two categories: sparse attention that reduces the span of attention, and attention approximation with different attention estimation formats. Let us review the original self-attention definition. Formally, given a sequence of hidden vectors $x$, we can map it into different representation space $Q$, $K$, and $V$. Then, attention takes $Q$, $K$, and $V$ as inputs and is responsible for generating vector via the following equation:

where $Q$, $K$, $V$ are 3-dimension tensors, with dimensions of sequence length, head number, hidden dimension. The computations mainly come from $(\frac{QK^{T}}{\sqrt{d_{k}}})$ and softmax operations. This section describes several approaches to reduce dot-product computations. We leave the details of efficient softmax variants in Section 2.1.3.

As we can see from Eq. 2.1, all tokens are required to be attended at each step. Several approaches are proposed to reduce attention length by only attending local tokens at each step. A natural solution is to reduce the number of attended tokens by assigning some tokens with zero weights. It is the idea of sparse attention (Martins & Astudillo, 2016; Child et al., 2019; Correia et al., 2019; Dai et al., 2019; Zaheer et al., 2020). Martins & Astudillo (2016) proposed Sparsemax, which added a $L_{2}$ regularization to encourage the attention matrix to be sparse. Sparsemax has been applied to various architectures (Niculae & Blondel, 2017; Maruf et al., 2019; Peters et al., 2019). Child et al. (2019) introduced heuristic rules and defined two sparse attention variants. One attends previous $l$ tokens. The other splits a sequence into different spans where each span has $l$ tokens. Each head attends to every $l$ tokens where $l$ is much smaller than the length of inputs. Sukhbaatar et al. (2019) believed that the naive sparse attention was somehow arbitrary. They found that some attention heads focused on the recent tokens, while other heads took information from the whole context. Motivated by this, they proposed adaptive attention by learning the attention span of each head. An similar idea is proposed by Correia et al. (2019). They introduced an adaptive sparse attention approach. They replaced full softmax operations with $\alpha$ softmax that allowed low-scoring words to receive precisely zero weight. In addition to attention length, the attention head is also an important sparse factor. Voita et al. (2019) found that only a subset of heads matter and the rest can be pruned.

Kitaev et al. (2020) proposed an efficient attention model Reformer. It approximates the dot-product attention computation by one that uses locality-sensitive hashing, reducing the complexity from $O(L^{2})$ to $O(L\log L)$, where $L$ is the length of the sequence. Choromanski et al. (2020) further proposed a more efficient model Performer with an unbiased positive random feature map estimator. Compared to the original attention, Performer is a linear architecture with compatible performance.

1.3 Lightweight Softmax

Softmax layer is a necessary component for deep learning. The key idea is to normalize a vector to a probability distribution of possible labels. Traditional softmax is computed as:

where $y_{i}$ is the $i$-th label and $w$ are learnable parameters. $h_{i}$ is the $i$-th dimension of hidden vector. $x$ is the input sequence. The denominator requires the dot-product over label candidates. If the task has a large label set, the denominator will require large computations. Since the complexity of softmax is proportional to the number of labels and sequence generation (Mikolov et al., 2013a, b) tasks usually have a large vocabulary size, we take sequence generation as an example to show several lightweight variants with fewer computations. In sequence generation tasks, token vocabulary is equal to the label set. Formally, given a hidden vector $h$ and all token embeddings, the softmax for sequence generation is computed as:

where $x$ is the input and $v_{i}$ is the embedding of the $i$-th token. Eq. 2.3 shows that the softmax layer introduces embeddings for all tokens and requires the inner-product between hidden vector and all embeddings. A large vocabulary will require many computations. Therefore, several efficient softmax variants have been proposed.

To reduce memory usage, Press & Wolf (2017) proposed to tie input embeddings and embeddings in Eq. 2.3. They conducted experiments on machine translation. Results show that weight sharing can reduce the size of neural translation models without harming translation results. In addition, reducing the number of labels is another important research direction. Recently, several studies have been (Kim et al., 2016b; Costa-jussà & Fonollosa, 2016) proposed to generate a sequence in character level, rather than in word level. The number of characters is largely less than that of words and the computations for softmax can be largely reduced. Similarly, Józefowicz et al. (2016) implemented character-based softmax on language modeling, which achieved promising results. It is important to note that these character-based methods also bring longer sequences. Current sequence generation models usually adopt auto-regressive generation frameworks. The longer sequence brings higher decoding costs. In all, it should be considered case-by-case whether character-based methods reduce the whole decoding cost. Recently, a trade-off is achieved by sub-word level vocabularies (Sennrich et al., 2016). Sub-word level vocabularies have a tradeoff granularity between character vocabularies and word vocabularies. Sub-word level vocabularies have more tokens than character-level vocabulary but also have much shorter segmented sequences. Therefore, sub-word level vocabularies become the popular choice for almost all sequence generation tasks.

We classify softmax variants with fewer computations into five categories: hierarchical softmax, softmax with dynamic embeddings, sampling-based softmax, hashing-based softmax, and normalization-based softmax. Hierarchical softmax (H-Softmax) (Morin & Bengio, 2005; Mnih & Hinton, 2008) is a kind of softmax variant. To be specific, it formulates a label set as a tree and all labels in the set is the leaf node. The complexity can be dropped from $O(N)$ to $O(log(N))$ where $N$ is the size of the label set. In this way, the traditional one single probability over labels is decomposed into a product of a sequence of probability over each tree layer. The regular softmax can be regarded as a tree of depth $1$, with all labels as leaf nodes. The second research direction focuses on dynamic label embeddings (Chen et al., 2016b). The intuition is that not all labels require the same parameter size. It assigns variable parameter sizes for different labels. In particular, the approach assigns more parameters to frequent labels. The embedding size affects the computation costs. Therefore, this kind of method can reduce the computations required by softmax operations. In addition, sampling-based softmax aims to estimate the full softmax computations with sampled label candidates. The key idea is to sample several label embeddings to estimate all embeddings (Bengio & Senecal, 2003, 2008; Jean et al., 2015). Hoever, it only reduces training costs while the full softmax is still be computed to obtain a variance-free result during inference. Hashing-based softmax is another kind of estimation variant. Vijayanarasimhan et al. (2015) proposed a fast locality-sensitive hashing technique to approximate the actual dot-product. Normalization-based softmax (Devlin et al., 2014; Andreas & Klein, 2015) aims to avoid explicit denominator. The target is to output a vector as close as the probability distribution with the sum being 1.

1.4 Compact Embeddings

Building token embeddings is the first step for NLP tasks. The parameters of embeddings are decided by vocabulary size and embedding length. How to reduce embedding parameters is an important and interesting topic. Learning compact token vectors is related to learning compressed neural networks. There have been several techniques for learning compact neural networks, like pruning, knowledge distillation, low-rank approximation, and quantization. In this part, we only focus on related approaches for compact embeddings. We classify these approaches into four categories: reuse-based approaches, knowledge-distillation-based approaches, low-rank-based approaches, fine-grained vocabularies.

Reuse-based approaches focus on compositional embeddings (Faruqui et al., 2015; Chen et al., 2016c; Shu & Nakayama, 2018; Joshi et al., 2019; Shi et al., 2020). For example, Faruqui et al. (2015) aimed to represent each token embedding as a sparse linear combination of basis vectors. The size of basis vectors is much less than token embeddings. Similar idea has been proposed by Chen et al. (2016c). They split the vocabulary into two parts. One part is a base set containing frequent tokens with fixed size (e.g., 8K), the other part is a set of rare tokens whose embeddings are encoded by the base set’s embeddings. Following these studies, Shu & Nakayama (2018) adopted the quantization approach to construct embeddings with few basis vectors. Recently, this idea has been adapted to other fields beyond NLP, like recommendation systems (Shi et al., 2020).

In addition, traditional compression approaches have been applied to compress embeddings. Mou et al. (2016) used knowledge distillation to transfer knowledge from a big token embedding layer into a smaller embedding layer. Chen et al. (2018a) used vocabulary-partition (block) based low-rank matrix approximation to reduce parameter size. Lam (2018) used 1-2 bits per parameter, rather than traditional 32-bits, for token embedding. Vocabulary size is also an important factor in deciding embedding size. Therefore, fine-grained vocabularies have been proposed to reduce the vocabulary length, like character-level vocabulary (Kim et al., 2016b), subword-level vocabulary (Sennrich et al., 2016), and byte-level vocabulary (Wang et al., 2020a).

2 Component Assembling

This part presents several component assembling solutions for efficient architecture design. Many widely-used architectures are efficient component assembling solutions. CNNs and LSTMs are representative models. A single filter in CNNs can handle all input spans. LSTMs adopt the same parameters for all steps. The key idea of efficient component assembling lies in sharing. We classify these assembling solutions into four categories: memory sharing, static weight sharing, dynamic weight sharing, and deployment weight sharing.

Memory sharing is a common technique to store a large model on devices with limited memories. A natural idea is to share the same storage among intermediate forward vectors (Pleiss et al., 2017) or backward vectors (Chen et al., 2016a; Gruslys et al., 2016). There are also some reversible models (Gomez et al., 2017; MacKay et al., 2018) where the activation of each layer can be reconstructed from the next layer to reduce memory requirements during the backward process. The models do not need to save intermediate activation vectors. Since several vectors share the same storage space, recomputation is necessary in some cases. To achieve a trade-off between efficient memory usage and fewer computations, some studies (Wang et al., 2018a) proposed to combine memory sharing with liveness analysis (Wang et al., 2016). During graph computation, GPUs adopt liveness analysis to create tensors and free tensors. For large intermediate tensors, frequent allocation/deallocation operations are time-consuming. Therefore, the runtime can be reduced by directly reusing memory segments from a huge pre-allocated memory pool. In addition to memory optimization on a single node, Rajbhandari et al. (2020) further explored memory sharing on distributed settings across multiple computation nodes.

2.2 Static Weight Sharing

Unlike memory sharing, static weight sharing aims at exploring how to reuse weights for a neural network. The difference between weights and intermediate vectors is that weights are fixed during inference and shared by all examples. To save memory, many models choose to reuse parameters across different layers or different tasks.

Cross-layer parameter sharing is a common technique for parameter efficiency. The idea of sharing parameters across layers has been well explored (Dehghani et al., 2019; Bai et al., 2019; Lan et al., 2020). Savarese & Maire (2019) proposed a parameter sharing scheme that defined a global bank of templates. The parameters of each layer of a CNN come from the linear combination of these templates. Dehghani et al. (2019) proposed a model, called Universal Transformer, where all layers shared the same parameters. Following these study, Lan et al. (2020) applied cross-layer sharing mechanism on pre-train/fine-tune settings and Takase & Kiyono (2021) proposed diverse sharing strategies. Recently, Plummer et al. (2020) adopted the idea of network architecture search to automatically learn how to share parameters between all layers in a network.

Cross-task parameter sharing is also a popular solution to handle multi-task, multi-domain, or multi-lingual problems (Ramsundar et al., 2015; Duong et al., 2015; Søgaard & Goldberg, 2016; Hashimoto et al., 2017; Yang et al., 2017; Raffel et al., 2020). The key idea of cross-task is to enable all tasks (or languages/domains) to share parameters. Multi-task learning (Ruder, 2019) has two popular implementations, including hard and soft parameter sharing. Compared to soft parameter sharing where different tasks do not have shared networks, hard parameter sharing uses fewer parameters. Therefore, we only focus on hard parameter sharing in this work.

To be specific, cross-task sharing is initially implemented by sharing the hidden layers between all tasks, while keeping several task-specific output layers (Yang et al., 2017; Houlsby et al., 2019; Raffel et al., 2020). For the CV field, multi-task solutions often share CNN layers. For the NLP filed, in addition to naive sharing, researchers also focus on finding better parameter reusing solutions for different tasks. For example, Søgaard & Goldberg (2016) found that low-level tasks, e.g., part-of-speech tagging, should share parameters at lower layers. Motivated by these findings, Hashimoto et al. (2017) proposed a parameter-sharing network across multiple NLP tasks. Currently, the trend of developing large-scale models encourages researchers to directly use one single model to support multiple tasks. T5 (Raffel et al., 2020) is one representative model.

Multilingual is also a special cross-task variant. At the early stage of multilingual models, researchers usually choose to share a part of parameters across different languages (Firat et al., 2016; Upadhyay et al., 2016; Blackwood et al., 2018). Recently, multilingual approaches usually treated all languages equally and mixed them together to train a single model (Ha et al., 2016; Firat et al., 2017; Johnson et al., 2017; Fan et al., 2020a). More recently, adapter-based solutions have been widely used for modeling task-specific features beyond shared parameters (Houlsby et al., 2019; Bapna & Firat, 2019; Pfeiffer et al., 2020).

2.3 Dynamic Weight Sharing

Static parameter sharing usually relies on pre-specified networks. Researchers define heuristic rules based on shared features to decide which layers/components should be shared by different inputs/tasks. Although this solution is natural, hard sharing usually fails in handling tasks that are not closely related. Dynamic solutions are proposed to decide which layers/components should be shared among different input samples. Specifically, dynamic networks are neural networks with dynamic computational graphs where the computational topology or parameters are decided on the fly. Therefore, this kind of network can reduce computation costs and improve the adaptiveness of networks. In this survey, we describe the overview of general dynamic architectures. If you are interested in other dynamic features, you can find surveys focusing on dynamic networks (Han et al., 2021b). Networks with dynamic architecture can be classified into the following classes:

Cascading-style Networks. Multiple basic networks are cascaded in a directed acyclic graph (DAG) in a from-small-to-big manner, where the model first executes smaller networks, then larger networks. If a smaller network can handle the input sample, the model will stop the execution process and does not run execute models.

Early-exit-style Networks. A single network contains multiple internal classifiers, allowing “easy” samples to exit at shallow layers. The difference with cascading-style networks lies in early-exiting networks feed the output of previous layer to the next layer while cascading-style networks cut off this information flow and every network only takes raw samples as inputs.

Skipping-style Networks. It accelerates inference by either skipping certain layers, or skipping unimportant input spans in the whole input sequence.

Mixture-of-experts-style Networks. Multiple experts are provided as candidates in the same block. Only a small part of experts are used in each block for inference.

Cascading-style dynamic networks have a historical development (Viola & Jones, 2001; Lienhart & Maydt, 2002; Viola & Jones, 2004). The authors cascade architectures are originally proposed for unbalanced binary classification tasks. They cascaded multiple basic models and fed the input to the next model only if the current model was not confident of its prediction. For example, Park et al. (2015) cascaded two VGG networks in a small-first manner to obtain a better trade-off between classification accuracy and energy consumption. The smaller model can handle most samples, which largely reduce inference costs. Bolukbasi et al. (2017) cascaded AlexNet, GoogLeNet, and ResNet together. Wang et al. (2018c) introduced a cost-aware objective for jointly training criterion functions among basic models. More recently, Li et al. (2020b) proposed a dynamic framework for accelerating the inference of pre-trained language models, CascadeBERT, which dynamically selected proper-sized and complete models in a cascading manner.

Early-exiting-style dynamic networks might be the most popular dynamic architecture nowadays (Teerapittayanon et al., 2016; Bolukbasi et al., 2017; Gormez & Koyuncu, 2021; Huang et al., 2018; Yang et al., 2020b; Wang et al., 2021c). With multiple internal classifiers on intermediate layers, a network is capable to give intermediate predictions and make decisions about whether to execute the forward process or not. If the answer is yes, current state would be fed to the next layer. Otherwise, the network outputs the intermediate prediction as the final prediction. Different from cascading architectures that cuts off the information flow between networks, early-exiting networks reuse the feature computed by previous layer. For example, BranchyNet (Teerapittayanon et al., 2016) inserted several branch classifiers into a CNN to speedup inference. MSDNet (Huang et al., 2018) designed an exquisite two-dimensional multi-scale architecture to enable early exiting along two dimensions and RANet (Yang et al., 2020b) further utilized the spatial redundancy for image classification.

In addition to the CV field, existing studies also apply early-existing-style dynamic networks to the NLP field (Xin et al., 2020; Liu et al., 2020a; Zhu, 2021). The Two Stage fine-tuning is the most representative approach to train early-existing-style dynamic networks in NLP where the backbone is fine-tuned with the final classifier in the first stage, and the intermediate classifiers are fine-tuned in the second stage. In addition, joint training is also a trend to tune all parameters together including basic backbones and intermediate classifiers (Schwartz et al., 2020b; Liao et al., 2021; Geng et al., 2021). In addition to training algorithms, recent researchers also focus on criterion design. For example, Zhou et al. (2020) and Sun et al. (2021) utilized counting-based criteria to support early exiting. Without relying on heuristic criteria, several approaches (Xin et al., 2021; Schuster et al., 2021) directly learned the criteria by introducing a small module to decide whether to execute the forward process. Due to the simplicity of single-step prediction, dynamic networks are widely applied to classification models. Recently, Elbayad et al. (2020) and Li et al. (2021) extended multi-exit design to translation tasks and sequence labeling tasks.

Dynamic halting is a special case of early exiting, where the parameters across layers are shared and, therefore, the final classifier can also be shared. Specifically, these networks infer samples through a shared layer iteratively, rather than infer through multiple stacked individual layers. One representative network is proposed by Graves (2016). They proposed the adaptive computation time (ACT) mechanism for recurrent models to automatically decide how many times (iterations) each input symbol or token should be computed. Following this work, the ACT mechanism has been applied to various architectures, like ResNets and Transformers. For example, SACT (Figurnov et al., 2017) performed dynamic halting in two dimensions, including the coarse ACT among multiple layers within the same block and the fine-grained ACT on all spatial positions. Universal Transformer (Dehghani et al., 2019) shared all layers within the encoder (or decoder) in Transformer.

In cascading-style or early-exiting style dynamic networks, the key question is to figure out how confident the intermediate classifier is. Previous studies proposed various criteria for judging the reliability of an intermediate prediction. We categorize them into types as shown in Table 2.1. Score margin is the gap between the largest and the second largest scores in the predicted probability distribution (Park et al., 2015). Entropy-based criterion is based on the entropy of the predicted probability distribution (Teerapittayanon et al., 2016; Li et al., 2021; Liu et al., 2020a). The model executes the forward process only if the entropy is larger than the pre-defined threshold. Max-probability based criterion is the gap between the max value of the predicted probability distribution and the pre-defined threshold (Kaya et al., 2019; Wang et al., 2020d). Patience-based criterion terminates the forward process only if the model generates continuously identical predictions (Zhou et al., 2020). Voting-based criterion is inspired by the ensemble technique, which terminates the forward process if the most of historic predictions reach an agreement (Sun et al., 2021). After-prediction based criterion and before-prediction based criterion introduce additional learning functions to learn whether to execute the forward process. The only difference lies in that after-prediction based criterion uses the prediction distribution as inputs and before-prediction based criterion uses the naive hidden vector as inputs.

Skipping-style dynamic networks skip some computations during forward process. These dynamic networks are capable to obtain higher efficiency. This dynamic solution has been widely-use in various models, like SkipNet (Wang et al., 2018d), ConvNet-AIG (Veit & Belongie, 2020), and BlockDrop (Wu et al., 2018b). They introduced additional policy networks responsible for deciding to skip certain layers or not. The main formula for those dynamic networks could be summarized as:

where $\bm{x}_{l}$ is the input of the $l$-th residual unit, $F_{l}(\cdot)$ is the network layers within the $l$-th residual unit except skip connection, and $z_{l}\in\{0,1\}$ is a binary value predicted by the policy network or the $l$-th policy module. By utilizing reinforcement learning or the Gumbel re-parameterization trick, the network can be trainable in an end-to-end way.

Another research skipping-style line chooses to skip inputs given a long input sequence (Yu et al., 2017; Campos et al., 2018; Yu et al., 2018) or assign fewer computations to unimportant steps (Jernite et al., 2017; Seo et al., 2018) or exit reading (Yu et al., 2017; Liu et al., 2018c; Yu et al., 2018). 1) Skipping unimportant inputs is the natural way. Campos et al. (2018) introduced Skip-RNN where a binary gate unit was used to learn to skip current input token or not. If the answer is yes, Skip-RNN copies current hidden state to the next time step, saving computations on those unimportant inputs. LSTM-Jump (Yu et al., 2017) achieved the same goal by directly predicting how many steps to jump through, or whether to exit reading inputs. Although skipping partial inputs saves computations largely, these models, like Skip-RNN and LSTM-Jump, suffer from missing or repeating outputs at skipped positions thus are not suitable for token-level tasks. 2) To address this problem, assigning fewer computations to unimportant steps is a flexible solution. To this end, Seo et al. (2018) proposed Skim-RNN that dynamically decided to update the full-sized hidden state or partial-sized hidden state at each time step. 3) Exiting-style reading is a special kind of skipping reading (Shen et al., 2017; Yu et al., 2018; Liu et al., 2018c). It decides to truncate the next inputs. For example, Liu et al. (2018c) applied the exit mechanism to multi-task scenario. ReasoNet (Shen et al., 2017) adopted the exit mechanism for machine comprehension tasks. Despite good trade-off between accuracy and inference speed, skipping-style dynamic networks are harder to train, introducing more tuning overhead.

Mixture-of-experts-style dynamic networks are representative dynamic models (Lepikhin et al., 2021; Lin et al., 2021; Fedus et al., 2021). In those models, a layer contains multiple experts and only part of these experts will be activated for each instance. For example, Switch Transformer (Fedus et al., 2021) is the representative model that has trillion-level parameters. It replaces the normal feed-forward layer in the Transformer with a switch feed-forward layer, consisting of a routing module and multiple structure-identical experts. In each switch layer, only a single expert will be executed for each token. Compared to general dense computation architectures, mixture-of-expert-style networks provide an affordable and practical way to modify and train large models with sparse activation.

2.4 Deployment Sharing

When deploying deep learning models on edge devices, we have to consider realistic constraints, such as storage, memory, computation, latency, and power consumption. Previous researchers have designed lightweight and compact models for mobile devices or other edge devices, such as MobileNets (Howard et al., 2017; Sandler et al., 2018; Howard et al., 2019). However, with different hardware resources, the optimal neural network architecture varies significantly (Cai et al., 2020). Thus, developing elastic or dynamic models to satisfy different constraints is critical for practical applications.

In the recent two years, some studies have paid attention to efficient deployment. In these studies, a super-network is trained together with its massive sub-networks by task-specific losses. During inference, the appropriate sub-network is selected to satisfy the resource constraints. By amortizing the only-once training cost, the total cost of specialized designing is reduced from O(N) to O(1). During inference, the model can dynamically choose an appropriate network for different devices. To be specific, Yu et al. (2019) proposed slimmable neural networks where several widths are predefined, supporting instant and adaptive accuracy-efficiency trade-offs by selecting corresponding width. Following this work, Yu & Huang (2019) further proposed US-Nets to support arbitrary width selection. Fan et al. (2020b) proposed an elastic network that can select sub-networks of any depth from one large network without having to finetune them. Beyond aforementioned studies, the temporal or input length is also an elastic selection. For example, Kim & Cho (2021) proposed length-adaptive Transformer to support arbitrary progressively length deduction. The Length-adaptive Transformer can be directly adopted into the downstream task and satisfy any efficiency constraints by searching the corresponding length deduction configurations.

3 Compact-architecture Search

In addition to model design, there are studies working on search efficient networks towards resource-constraint devices, like mobile. They borrow the idea of neural architecture search and apply it to design tiny networks. For example, Tan et al. (2019a) proposed a neural architecture search approach, which explicitly incorporated model latency into the main objective so that the search can identify a model with a good trade-off between accuracy and latency. On the ImageNet classification task, this approach achieved 75.2% top-1 accuracy with 1.8x faster than MobileNet V2. Howard et al. (2019) combined neural architecture search and network design together to develop a stronger mobile net MobileNet V3. Cai et al. (2019) directly learned the architectures on the target task and hardware. Wu et al. (2019) proposed a differentiable neural architecture search framework that used gradient-based methods to optimize ConvNet architectures towards mobile devices.

Chapter 3 Energy-Efficient Training

Many advanced approaches have been proposed to reduce training costs for deep learning. In Chapter 2, we describe efficient networks that can reduce computations in a single execution. In this chapter, we focus on the computations required during the whole training, including weight tuning and hyper-parameter tuning. To be specific, we survey approaches that aim to accelerate weight tuning/hyper-parameter tuning by using fewer iterations, including initialization, normalization, progressive training, and efficient NAS. An overview is shown in Figure 3.1.

The training of deep learning starts from architecture design and parameter initialization. We have explored efficient architecture design in Section 2. In this part, we focus on how weight initialization affects model training.

It is widely accepted that good initialization of weights in a neural network is critical to convergence (Glorot & Bengio, 2010; Krizhevsky et al., 2012; He et al., 2015; Mishkin & Matas, 2016; Kumar, 2017). At the beginning, deep networks are usually initialized via random weights drawn from uniform distributions or Gaussian distributions. Many previous studies found that these kinds of initialization failed in handling very deep models (Glorot & Bengio, 2010; Saxe et al., 2014; Romero et al., 2015; Hanin & Rolnick, 2018). The problem is caused that the mean/variance of activations and gradients exponentially with the depth. To enable training with a very deep model, some advanced initialization solutions, like Kaiming initialization (He et al., 2015), Xaiver initialization (Glorot & Bengio, 2010), LSUV initialization (Mishkin & Matas, 2016) and Fixup initialization (Zhang et al., 2019), have been proposed. The key idea is to normalize the variance of weights to make the variance of activation in each layer to be around 1. We list the details of two widely-used initialization approaches in Table 3.1. Compared to the naive baseline, these initialization approaches can achieve better performance and faster convergence (Mishkin & Matas, 2016).

1.2 Pre-trained Models for Initialization

In addition to random initialization, many approaches borrow models pre-trained from other domains (or other tasks) as initialization. It is widely believed that initialization from existing models is an effective technique to improve the generalization ability with fewer training iterations. We split these pre-training initialization into different categories according to different dimensions. First, according to whether the borrowed parameters keep unchanged, these methods can be classified into feature-based initialization, and fine-tuning-based initialization. Second, according to the knowledge source of pre-trained parameters, these methods can be classified into supervised initialization and self-supervised initialization.

Feature-based initialization borrows the parameters (usually from low-layers or mid-layers) as initialization from other domains/tasks while these parameters keep fixed during training. Generally speaking, feature based initialization can keep the generalization ability of the borrowed parameters better and thus is more suitable for extremely few-shot settings.

Fine-tuning-based initialization uses the target data to train all parameters, including new parameters and borrowed parameters. Fine-tuning based initialization can further optimize the target objectives via fine-tuning all parameters and thus can better fit training data. It is the most popular solution nowadays for the NLP field.

Supervised initialization is widely investigated in the earlier stage of deep learning. A common solution is to pre-train the target model on similar tasks/datasets, and then reuse the pre-trained parameters as initialization for the target task (Huang et al., 2013; Oquab et al., 2014; Yosinski et al., 2014; Duong et al., 2015; Long et al., 2016). This solution is especially popular for low-resource settings and is extensively studied on domain adaptation/transfer learning.

The most representative example for supervised initialization is the pre-training of deep CNN backbones (Simonyan & Zisserman, 2015; Ren et al., 2015; He et al., 2016a; Simon et al., 2016; He et al., 2017a; Iglovikov & Shvets, 2018). Fine-tuning pre-trained CNNs on different downstream datasets usually leads to improved performance compared to training from scratch and also reduces the number of training steps. Previous researches have explored advanced initialization methods. It is widely accepted that layers near to inputs usually are responsible to capture local features (Zeiler & Fergus, 2014). Therefore, many studies focus on transferring knowledge via initialization from low-layer features and mid-layer features. Interestingly, with the increase of large-scale training data, current trends directly adopt the simplest solution that uses all parameters for initialization (Li et al., 2020a).

Supervised initialization is also successfully applied to NLP (Socher et al., 2013). It is widely-accepted that features computed in higher layers of the network usually depend on the specific dataset and task. Following this belief, many studies borrow the parameters of low-level layers and mid-level layers from other domains as initialization (Dong et al., 2015; Luong et al., 2016; Yang et al., 2017; Lin et al., 2018; Liu et al., 2018b). Recently, the trend of large-scale networks enables researchers to reuse all parameters from pre-trained networks. Similar with CV, the widely-adopt setting in NLP is directly reusing all parameters (Johnson et al., 2017; Aharoni et al., 2019; Tan et al., 2019b; Bapna & Firat, 2019; Lin et al., 2020).

Self-supervised initialization is also a popular direction. With the increasing parameters in state-of-the-art DNNs, more and more training data are required to achieve better generalization results. To reduce the requirements of supervised data, previous studies investigated self-supervised pre-training that exploited unlabeled data to construct supervision signals to learn representations. Since self-supervised pre-training does not require any human-annotated labels, it is easy to get sufficient training data. To this end, researchers designed various methods to construct self-supervised training signals with unlabeled data. Here we take CV and NLP as examples to review recent self-supervised pre-training models.

For the NLP fields, using a model pre-trained on self-supervised data as initialization is the most popular solution. At the start, researchers use language modeling to pre-train word embeddings which are then used to initialize downstream word embeddings (Joshi et al., 2016; Qi et al., 2018; Ruder et al., 2019). Glove is one widely-used word embedding toolkit (Pennington et al., 2014) which trains word embeddings based on global word-word co-occurrence counts. With the development of representation learning, researchers begin to explore and reuse contextualized models. Contextualized models define that the representation of a word depends on its contexts and each word has two representations, fixed word embeddings and contextualized representations. Peters et al. (2018) proposed the first widely-used contextualized representations, ELMo. Following this work, many advanced contextualized representation models begin to spring up, like BERT (Devlin et al., 2019), GPT (Schick & Schütze, 2021), T5 (Raffel et al., 2020). The development of pre-trained networks also affect the application of CV. In recent years, CV began to explore large-scale self-supervised models for initialization (Lu et al., 2019; Li et al., 2020a; Chen et al., 2020b). The learning objective is similar with NLP’s pre-trained networks, either recovering masked/noised regions, or generating the original image from scratch.

Empirical results demonstrate that these pre-trained networks for initialization can achieve better performance and faster convergence. However, since current pre-trained networks generally require downstream tasks to use the exactly same networks, the training time still depends on architecture execution in addition to convergence speed. Therefore, it should be considered case by case to conclude whether pre-trained models for initialization reduce downstream training costs in implementation.

2 Normalization

In addition to initialization approaches, normalization is another solution to accelerate training. Strictly speaking, normalization is a special component. Considering that it can accelerate convergence (Bjorck et al., 2018; Santurkar et al., 2018; Zhang et al., 2019), we describe normalization in this chapter.

Normalization is a technique to normalize hidden outputs in deep neural networks. Batch normalization (Ioffe & Szegedy, 2015) is the first widely-used normalization for deep models. The key idea is to normalize the hidden vectors of neural networks to the distribution with mean $\mu=0$ and standard deviation $\sigma=1$. The hidden vectors usually are tensors and batch normalization is applied on the batch dimension. To be specific, it generates the output given the hidden output $h$:

where $h$ is a intermediate tensor where the first is batch dimension. $|B|$ is the batch size. $y$ and $h$ are the output and input of the normalization component. Ioffe & Szegedy (2015) fond that applied to a state-of-the-art image classification model, batch normalization achieved the same accuracy with 14 times fewer training steps, and beat the original model by a significant margin.

Following batch normalization, many normalization variants have been proposed, like layer normalization (Ba et al., 2016), group normalization (Wu & He, 2020), weight normalization (Salimans & Kingma, 2016). These variants have almost the same calculation process except they are applied to different dimensions or different objectives.

Despite good performance, it is still controversial where the benefits of normalization come. At the start, normalization is proposed to address internal covariate shift by normalizing layer inputs. Internal covariate shift is a phenomenon where the distribution of each layer’s inputs changes during training. The parameters of the higher layer are required to continuously fit for the new distribution of lower layers, which slows down the training. To keep distribution steady, normalization is proposed to fix the distribution of input to a standard distribution. However, Santurkar et al. (2018) overturned this belief and they found that the distributional stability of layer inputs had little to do with the success of batch normalization. Instead, normalization makes the optimization landscape significantly smoother. This smoothness induces more predictive and stable gradients, allowing for faster training. Motivated by this paper, Xu et al. (2019) proved that normalization indeed normalized backward gradients, which plays an important role in deciding the success of normalization.

3 Progressive Training

Progressive training is another strategy to effectively train DNNs. The key idea is constructively adding layers. Compared to full training, progressive training does not require full gradients to all parameters, thus can largely reduce computations required for training. In addition, the well-trained lower layers also accelerate the training of higher layers. Hinton et al. (2006) applied progressive training to deep belief networks. They trained layers sequentially starting from bottom layers in a greedy, layer-wise fashion. It is based on an assumption that upper layers represent more “abstract” concepts whereas lower layers extract “low-level features”. This method is unsupervised because each layer learns a higher-level representation of the layer below and the training criterion does not depend on the labels. Following this work, Bengio et al. (2006) extended this method to handle continuous inputs.

With the development of deep learning, layer-wise progressive training methods are exploited to train CNNs (Rueda-Plata et al., 2015; Kulkarni & Karande, 2017; Belilovsky et al., 2019) and RNNs (Xu et al., 2018). Recently, Gong et al. (2019) and Yang et al. (2020a) extended the idea of layer-wise training to large-scale NLP models by progressively stacking new layers on top of previously trained layers. Their experimental results show that layer-wise training can successfully improve the efficiency of training large transformer language models with huge amounts of data. To be specific, experimental results show that such progressive training policy can achieve more than 110% training speedup without significant performance degradation.

4 Efficient Hyper-parameter Optimization

During training, hyper-parameter optimization (HPO) is a common and fundamental step for AI participants to find better model settings. Hyper-parameters keep fixed during training, including but not limited to optimization settings (e.g., learning rate, batch size) and model settings (e.g., the number of layers). Since deep learning performs like a black-box model and the learning landscape is non-convex, current optimization approaches usually find a random local minimum. Due to the uncertainty, AI engineers tend to taking a lot of computations to find better hyper-parameter settings on real-world applications (Yu & Zhu, 2020). HPO or autoML is a field to automatically find the optimal settings. Considering that previous approaches mainly study architecture settings, we take efficient neural architecture search (NAS) as an example in this survey to review recent progress. Following previous studies, we split NAS methods into three components: search space, search strategy, and architecture evaluation. In this survey, we give an overview of efficient NAS. There are also surveys describing more details of NAS (Elsken et al., 2018).

The search space defines all architecture candidates. At the first, the search space is defined as a discrete space, including structured space or unstructured space. Considering that network candidates in unstructured space is too massive, researchers usually incorporate inductive bias to build a structured search space (Liu et al., 2018a; Dong & Yang, 2020; Shu et al., 2020). One of the representative methods is cell-based search space. Cell-based search space assumes that each architecture contains repetitions of fixed structures. In this way, the search space can be limited to the cell space where the number of candidates is largely reduced. In addition, to enable faster search, differentiable approaches (Jiang et al., 2019) adopt a continuous search space where edge weights are considered.

The search strategy defines a policy to explore search space. Random search is one of traditional search approaches. The key idea is to randomly evaluate architectures and to select the best one based on their validation results. To reduce wasted evaluation costs, researchers proposed reinforcement learning based search policies (Ying et al., 2019). It is a direction that introduces an architecture generator to generate well-performing architectures. Since random search and reinforcement learning require validation accuracy as search criterion, these methods usually need expensive computations. To reduce search costs, evolution based search has been proposed (Real et al., 2019). It is a two-stage search approach. The first stage selects several well-performing parent architectures. The second stage applies mutation on these parent architectures to select the best one. The second stage starts from pre-trained parent networks and does not require too much computations to train child networks. Recently, So et al. (2019) applied the evolution search on Transformer networks and achieved new state-of-the-art results on machine translation and language modeling tasks. Although these approaches can reduce exploration costs, the dependence on validation accuracy still leads to considerable computation costs. To fully get rid of the dependence on validation accuracy, several studies (Jiang et al., 2019; Liu et al., 2019a; Dong & Yang, 2019; Zela et al., 2020; Chu et al., 2020) proposed differentiable search that re-formulated the task in a differentiable manner and allowed efficient search using gradient descent. In addition, another research line aims to represent a model into a continuous space where there is a mapping between structures and results. In this way, the model only learns how to predict the performance of architectures based on their continuous representations where the downstream training is not required (Luo et al., 2018). To further reduce training costs, researchers proposed training-free NAS approaches that directly extracted features from randomly-initialized models and used these features as evaluation criterion to select networks (Mellor et al., 2020; Chen et al., 2021a; Abdelfattah et al., 2021; Xu et al., 2021b).

Architecture evaluation takes almost all computations in NAS approaches. At the first, full training is required to evaluate the performance of a network, which is very heavy. Early stop is a widely used trick to estimate the results of a network. Besides, parameter-sharing is also a popular solution that network candidates can share parameters with each other (Pham et al., 2018). In this way, the model can reuse pre-trained blocks during downstream training.

Current NAS solutions are well-explored in the CV field, and widely-used benchmarks are also based on CV datasets. In the future, it is a promising direction to apply NAS in other fields, like NLP, to address more real-world problems. In addition, existing models focus more on models towards a single task for simplification. The multi-task/domain/lingual model is attracting more attention. Therefore, how to use NAS to search a shared multi-task/domain/lingual model is also a promising direction. Furthermore, the essential question of NAS is how model architecture affects downstream results. More understanding studies are expected to reveal the fundamental connection between architecture and performance.

Chapter 4 Energy-Efficient Inference

In this chapter, we describe common network surgery methods for reducing inference costs, including pruning, low-rank factorization, quantization, and knowledge distillation. A brief review of these methods is presented in Figure 4.1 and Table 4.1.

Model pruning is a popular solution to reduce redundant parameters in DNNs. Back to 1980s, Hanson & Pratt (1988) and LeCun et al. (1989) already verified that parameters are not equally important to the final performance. By removing unimportant weights from a network, we can simultaneously reduce the number of parameters, accelerate training/inference, save training examples, and improve generalization. This motivates a great amount of studies on pruning neural networks in the past 30 years. Specifically, given an initial network that is large and accurate, the key idea of pruning is to remove parameters from the original network to produce a smaller network that can retain the accuracy of the original model.

We first provide a formal definition of pruning. Let us define a neural network model as $f(X;\theta)$, which is a function over an input set $X$ and the set of parameters $\theta$. Pruning approaches usually take a model $f(X;\theta)$ as input and then produce a new model $f(X;\theta^{\prime})$. $\theta^{\prime}$ is a set of parameters with the size of $\theta^{\prime}$ being less than that of $\theta$. Usually, $\theta^{\prime}$ is a subset of $\theta$.

Previous pruning approaches mainly follow the work of Han et al. (2015) to produce a pruned model $f(X;\theta^{\prime})$ from an original model $f(X;\theta)$. We show the generic framework in Algorithm 1. First, the network is trained to convergence to get pre-trained parameters. Second, each parameter or structural element in the network is assigned a score. This score indicates the relative importance to the final performance. The network is then pruned based on these scores. Third, since pruning generally reduces the accuracy of the network, it is a general practice to fine-tune (train further after pruning) the pruned network. The process of pruning and fine-tuning is usually iterated several times.

We describe some key components of network pruning algorithms:

Pruning Unit refers to the basic unit that the algorithm aims to prune. According to pruning units, we can classify modern pruning approaches into two categories, unstructured pruning, and structured pruning. Some pruning algorithms prune individual parameters (i.e., unstructured pruning), which produce a sparse neural network. While the resulting sparse network is smaller in terms of the number of parameters, it is hard to yield speedups since the pruned weights are not well arranged. In contrast, structured pruning considers parameters in groups. They keep the dense features of the network by removing entire weight matrices, filters, channels, or layers.

Scoring Function defines the metric used to prune parameters. Common practices for parameter scoring usually are based on importance coefficients, network activations, or gradients. After assigning scores to each part of the parameters, we have two choices to prune networks. First, we can choose to prune a fraction of the parameters with the locally lowest scores within each structural sub-component of the network (e.g., layers). Second, we also can choose parameters with the globally lowest scores within the entire network.

Scheduling decides the total step that pruning algorithms use to prune parameters. Some methods prune weights in a single step while another methods use multiple steps to prune parameters where each step only prunes a part of parameters.

Fine-tuning is usually required for the pruned network to recover the original accuracy. Many methods choose to fine-tune the pruned network, or re-train the pruned network.

Pruning is first applied to fully-connected networks. For example, LeCun et al. (1989) analyzed the importance of parameters and showed that small magnitude weights had less impact on training losses. Specifically, they computed the saliency of parameters based on the second derivative. Then, they pruned parameters with small saliency scores. To recover the original performance of the network, the network was fine-tuned after pruning. Hassibi et al. (1993) extended this idea by using the inverse of the Hessian as saliency score. In addition to weight pruning, Suzuki et al. (2001) proposed to prune network connections based on their influence on training losses, and then re-train the network to compensate the performance drop. Unlike these approaches, Srinivas & Babu (2015) argued that similar neurons were redundant. They proposed to remove redundant neurons, instead of removing individual weight connections one by one.

Currently, many pruning algorithms are applied to CNNs. Han et al. (2015) proposed a simple magnitude-based method to remove unimportant connections in fully-connected layers and convolutional layers. However, the resulting model, despite being sparse, does not bring significant inference speedups due to the feature of sparsity. To address this problem, several structured pruning algorithms have been proposed to prune dense blocks, like filters, channels, or layers (Li et al., 2017; Molchanov et al., 2017; He et al., 2017b; Lin et al., 2017; He et al., 2018; Luo et al., 2019).

In addition to CV models, pruning has been successfully applied to NLP tasks. At the early stage, several studies successfully pruned recurrent neural networks (RNNs). See et al. (2016) used iterative pruning and retraining to prune a recurrent model for neural translation. Narang et al. (2017a) pruned RNNs via magnitude based pruning. Narang et al. (2017b) used iterative ground lasso regularization to induce block sparsity in RNNs. Lee et al. (2019) and Zhang & Stadie (2020) proposed one-shot RNN pruning methods based on connection sensitivity and Jacobian spectrum. More recently, with the success of Transformer, several studies investigated pruning transformer models (Michel et al., 2019; Voita et al., 2019; McCarley et al., 2019; Fan et al., 2020b; Wang et al., 2020c; Sanh et al., 2020). One trend is to use structured pruning since the transformer architecture is highly parallelized. For instance, Fan et al. (2020b) introduced LayerDrop to prune transformer layers for efficient inference. Michel et al. (2019) and Voita et al. (2019) revealed that the multi-head attention mechanism in the transformer architecture led to redundant attention heads. Motivated by this finding, they proposed to directly prune attention heads. McCarley et al. (2019) used structured pruning to compress a BERT-based question answering model. Xu et al. (2020) recently proposed a progressive module replacing approach by replacing a whole module from the original model with a compact module to reduce model size. Unlike these studies, Guo et al. (2020) proposed an unstructured pruning approach, diff pruning, to compress a multi-task model.

While most aforementioned pruning algorithms require re-using or re-training the originally trained network, a recent research direction (Frankle & Carbin, 2019) suggested that dense, randomly-initialized, feed-forward networks contained sub-networks (winning tickets), which can reached the test accuracy of the original network. They also found that a standard pruning technique naturally uncovered some of the winning tickets and then proposed an algorithm to identify winning tickets at the early stage of training. Also, Frankle & Carbin (2019) and Liu et al. (2019c) found that once the “winning ticket” is found, it can be trained from scratch to get an equivalent or better performance compared to pruned and fine-tuned one. Recently, there are also several studies investigating the lottery ticket hypothesis for BERT-like models (Prasanna et al., 2020; Chen et al., 2020c). For example, Prasanna et al. (2020) found that with structured pruning, the “random” sub-networks are still almost as good as the “good” ones, and even the “worst” ones perform on par with a strong baseline.

Pruning is very effective for reducing the number of parameters in DNNs. With structured pruning, it can accelerate inference and reduce computations. However, there are also several limitations of pruning methods. First, it requires iteratively scoring weights and re-training the network for many iterations. Also, pruning often leads to a non-negligible performance drop when applied to powerful DNNs. The lottery ticket hypothesis provides an interesting direction for more efficient “pruning” algorithms. In this survey, we give an overview of pruning methods. If you are interested in this direction, there are several surveys describing more details of pruning (Liang et al., 2021).

2 Low-rank Factorization

Similar to pruning algorithms, low-rank factorization is successfully applied to CV models. Sainath et al. (2013) applied matrix factorization to the final weight layer. If the original weight matrix has the dimension $m\times n$ and rank $r$, the full rank matrix can be factorized in two weight matrices as $m\times r$ and $r\times n$. Their approach reduced the number of parameters and achieved up to 30–50% speedup. Similarly, Xue et al. (2014) proposed to use SVD decomposition to compress fully-connected neural networks. Rigamonti et al. (2013) proposed to approximate trained CNNs with low-rank filters. Denton et al. (2014) further exploited the linear structure present within the convolutional filters. Their approach was able to reduce the memory requirement of the weights in the first two convolutional layers by 2–3 times. While low-rank matrix factorization can optimize both the spatial and computational complexity of neural networks, the plane view of a matrix limits the potential for extreme compression.

Tensor factorization algorithms, in contrast, are more flexible and can be employed to achieve an extremely high compression ratio. Among popular tensor factorization methods, classical prolongation (CP) (Kolda & Bader, 2009), where each factor matrix has the same rank and the kernel tensor is a superdialognal tensor, generally achieves better compression performance. Lebedev et al. (2015) leveraged CP decomposition to compress CNN weight kernels into several sub-kernels to reduce the number of parameters. Specifically, non-linear least squares were used to compute low-rank CP-decomposition of the 4-D tensor into a sum of rank-one tensors. Using this decomposition, the original convolutional layer was replaced by a sequence of 4 convolutional layers with smaller filters. Following this idea, Kim et al. (2016a) introduced a one-shot compression method to compress the whole network. In addition, Chen et al. (2018b) introduced a collective residual unit based on block term decomposition (BTD), which is a combination of Tucker and CP, to enhance the utilization of parameters in residual CNNs. Zhou et al. (2019) conversely used neural networks to learn an appropriate CP rank for tensor decomposition.

Apart from the applications on CV models, low-rank factorization has also been applied to NLP models. For example, Grachev et al. (2017) used low-rank factorization to train RNN language models. Winata et al. (2019) investigated the use of low-rank factorization as an effective post-processing compression method for LSTMs. They applied low-rank factorization on ELMo, one of widely-used pre-trained models. Recently, low-rank factorization has also been applied on Transformer models (Ma et al., 2019). Noach & Goldberg (2020) further proposed a two-stage model-compression method to reduce the inference time cost of BERT, a kind of Transformer-based model. Their approach decomposed the matrices into smaller matrices and then performed feature distillation on the internal representation. Also, Lan et al. (2020) applied embedding matrix factorization along with layer sharing to reduce the amount of parameters.

Compared with other popular compression methods, low-rank factorization can effectively reduce the size of models with a large compression ratio while preserving the performance well. Low-rank factorization is also relatively flexible. However, low-rank factorization also suffers from the issue of computational efficiency because SVD over large weight matrices can be computationally heavy. Also, compared with the compression ratio in terms of model size, low-rank factorization is less effective for reducing the computational cost and inference time.

3 Quantization

The goal of quantization is to compress the original network by reducing the number of bits. The idea of network quantization can be back to early 1990s (Fiesler et al., 1990; Balzer et al., 1991; Tang & Kwan, 1993). Recently, due to the success of DNNs and their growing sizes, the research of quantization has received increasing attention. In the beginning of 2010s, Vanhoucke et al. (2011) discovered that CNNs encoded with 32-bit can be converted to CNNs encoded with 8-bit, which significantly reduced both storage and computation costs.

Generally speaking, quantization techniques can be classified into two types: deterministic quantization and stochastic quantization. In deterministic quantization, there is an deterministic mapping between the quantized value and the real value. In stochastic quantization, the quantized value is sampled from discrete distributions (Guo, 2018). Usually, post-training quantization is the most simplest solution by applying quantization on a trained model to reduce inference costs. Despite simple, post-training quantization may brings dropped performance. quantization-aware training is proposed to address this problem by fine-tuning the quantized model before inference.

defines a deterministic mapping between real weights and quantized weights. Rounding quantization is the simplest mapping function. The key idea of rounding quantization is to map a high-bit floating-point number to its nearest fixed-point low-bit number (Gupta et al., 2015). For example, suppose a number $x$ and the target fixed-point representation [IL, FL]. The number of integer bits IL plus the number of fractional bits FL yields the total number of bits used to represent the number. This approach considers the following rounding scheme:

where $\epsilon(=2^{-\textnormal{FL}})$ is the smallest positive number that can be represented in this fixed-point format, $\lfloor x\rfloor$ is defined as the largest integer multiple of $\epsilon$ smaller than $x$. Following this approach, more advanced approaches have been proposed (Rastegari et al., 2016; Polino et al., 2018; Wu et al., 2018a).

In addition to scalar quantization for individual numbers in weight vectors, there is also a research line focusing on clustering-based quantization. Gong et al. (2014) proposed to classify weights into groups and to use the centroid of each group to replace the actual weights during inference. Han et al. (2016) employed a similar approach but fine-tuned the quantized centroids for better performance. Following this idea, Choi et al. (2017) further proposed a Hessian weighted k-means clustering approach.

does not define one-to-one mapping from real weights to quantized weights. In random rounding, the quantized value is sampled from a discrete distribution parameterized by real values. For example, Courbariaux et al. (2015) proposed the following random rounding function:

where $\sigma$ is the “hard sigmoid” function:

If $x$ is a positive value, we will have a high probability to quantize it to $+1$, otherwise to $-1$. This gives us a more flexible quantization scheme. In probabilistic quantization, the weights are assumed to be discretely distributed and a learning algorithm is used to infer the parameters of the distributions. Soudry et al. (2014) proposed an expectation back-propagation algorithm to train neural networks with binary or ternary weights. They first assumed some discrete prior distribution on the weights and then updated the weights in an online setting based on the Bayesian formula.

Recently, quantization-aware training (Jacob et al., 2018; Dong et al., 2017; Fan et al., 2020c) has become the de facto approach towards designing robust quantized models. It simulated quantization effects in the forward pass of training and the backward pass was accomplished via straight through estimator (Bengio et al., 2013b). It generally relied on techniques like gradient clipping to make the training stable. Recently, several studies analyzed and introduced new quantization-aware training approaches. For example, Fan et al. (2020c) and Dong et al. (2017) stochastically applied quantization to a portion of the weights at each training step, while Sheng et al. (2018) and Alizadeh et al. (2019) re-ordered the blocks or layers.

Network quantization is first widely applied to CNNs. In addition to CNNs, quantization has been also applied to other models, like RNN, Transformer. Ott et al. (2016) first investigated RNN quantization and found that weight binarization did not work well on RNNs. For simplification, they proposed to apply weight quantization for RNN weights and to leave activations as floating-point numbers. Hubara et al. (2017) explored different combinations of bit-widths for weights and activations. He et al. (2016b) proposed to quantize the structure of gates and interlinks in LSTM and GRU cells. Recently, Wang et al. (2018b) proposed to use a threshold ternary quantization method for weight quantization and a Bernoulli ternary quantization method for activation quantization.

With the recent success of Transformer, a number of studies investigated the application of quantization on Transformers. For example, Bhandare et al. (2019) and Prato et al. (2020) showed that 8-bit quantization can successfully reduce the size of a Transformer-based model and accelerate inference without compromising translation quality. Recently, quantization has been applied on Transformer-based language models (Zafrir et al., 2019; Zhang et al., 2020; Bai et al., 2020; Kim et al., 2021). Zafrir et al. (2019) first applied 8-bit quantization on BERT. Following this work, Kim et al. (2021) proposed I-BERT, which employed lightweight integer-only approximation methods for nonlinear operations to quantize BERT. Zhang et al. (2020) proposed TernaryBERT, which ternarized the weights in a fine-tuned BERT model with both approximation-based and loss-aware ternarization methods.

Very recently, several studies have investigated the application of quantization on GNNs. Feng et al. (2020) proposed a GNN-tailored quantization algorithm, and used an automatic bit-selecting approach to pinpoint the most appropriate quantization bits. Wang et al. (2021a) and Bahri et al. (2020) further proposed binarized GNNs. Tailor et al. (2021) proposed Degree-Quant, an architecture-agnostic method for quantization-aware training on graphs. Moreover, Zhao et al. (2020) proposed to use neural architecture search to find the optimal architecture coupled with the best quantisation strategy for different components in GNNs.

Quantization is very effective for reducing the size of neural networks. However, post-training quantization often leads to non-neglieable performance drop. In contrast, quantization-aware training can effectively reduces the performance drop. Incorporating knowledge distillation for quantization can also improve the performance of quantized models. In this section, we give an overview of quantization. If you are interested in more details, please refer to surveys (Guo, 2018; Gholami et al., 2021).

4 Knowledge Distillation

The idea of knowledge distillation (KD) is exploiting the knowledge inside a large trained “teacher” model to help the training of a “student” model (Bucila et al., 2006; Ba & Caruana, 2014; Hinton et al., 2015). In this way, we can use a smaller student model to distill a trained model as a replacement for inference.

The traditional KD solution is to minimize the difference between the output produced by the teacher model and that produced by the student model. Formally, given a labeled dataset $\mathcal{D}$ of $N$ samples $\mathcal{D}=\{\left(x_{1},y_{1}\right),\dots,\left(x_{N},y_{N}\right)\}$, we can write the loss function of the student network during the process of knowledge distillation as follows:

where $\alpha$ is a hyper-parameter to control the relative importance of the two terms; $\theta_{T}$ and $\theta_{S}$ are the parameters of teacher $T$ and student $S$, respectively. $\mathcal{L}_{\mathcal{T}}$ refers to the task-specific loss and $\mathcal{L}_{\mathit{KD}}$ refers to the knowledge distillation loss which measures the similarity of the student and the teacher.

In general, KD exploits the knowledge from the teacher model to help train the student model by minimizing the discrepancy between the knowledge in the teacher model and that in the student model. According to the source of teacher knowledge, we can classify KD approaches into three categories: logits-based KD, feature-based KD, and relation-based KD. According to teacher types, we also can classify KD approaches into three categories: KD with static teacher, KD with multiple teachers, and KD with dynamic teacher.

Logits-based KD focuses on the output class distribution of the teacher model, also referred as “soft labels”. This is the vanilla form of knowledge distillation (Hinton et al., 2015; Ba & Caruana, 2014). Soft targets generated by the teacher model provide much more information than hard targets. Therefore, training the student model to fit soft targets can help the student model generalize better like the teacher model.

Feature-based KD exploits intermediate features to teach the student model, which is believed to be important for representation learning (Bengio et al., 2013a). The simplest solution is to minimize the distance between intermediate representation of each student layer and its corresponding teacher layer (Romero et al., 2015; Sun et al., 2019a; Aguilar et al., 2020). It enables the student model to exploit richer information from the teacher model. Recently, a number of feature-based KD studies have been proposed. In summary, these studies mainly focused on two key factors: selection of intermediate representations, and distance metric. The intuition of investigating better intermediate representations is that the knowledge of teacher should be easy to learn for the student model (Zhang et al., 2017; Zagoruyko & Komodakis, 2017; Huang & Wang, 2017; Ahn et al., 2019; Heo et al., 2019; Sun et al., 2019a; Aguilar et al., 2020). For example, Huang & Wang (2017) proposed to match the distributions of neuron selectivity patterns between the teacher and the student models. Kim et al. (2018) trained a paraphrase model as $\textnormal{TF}_{t}$ to extract transferable features from the teacher’s intermediate representations, and a translator model as $\textnormal{TF}_{s}$ to map the student intermediate representation to teacher’s representations. As for distance metrics, $L_{2}$ distance is the most widely used distance metric. Besides, $L_{1}$ distance (Wang et al., 2019b) and KL-divergence (Liu et al., 2019b; Aguilar et al., 2020) are also used in previous KD approaches.

Relation-based KD aims to minimize the correlation between feature pairs from the teacher model and the student model (Yim et al., 2017; Srinivas & Fleuret, 2018; Lee et al., 2018; Tung & Mori, 2019; Lee & Song, 2019; Peng et al., 2019). Distance can be seen a special kind of relation measure. Recently, many approaches have been proposed to explore better relation measures. For example, MHGD (Lee & Song, 2019) employed a multi-head attention network to encode relations between any two feature maps in a certain batch of training instances. CCKD (Peng et al., 2019) transferred correlation between instances with a generalized kernel method based on Taylor series expansion.

Traditional KD methods focus on transferring the knowledge from one teacher model to the student model. A number of recent studies investigated knowledge transfer from multiple teachers or an ensemble of teachers. The most popular and straightforward way is to learn the ensemble of teacher logits (Tarvainen & Valpola, 2017; You et al., 2017). Following this idea, multiple studies have been proposed to model the diversity of teachers using a weighted average of teacher logits (Ruder et al., 2017; Lan et al., 2018; Xiang et al., 2020). Apart from averaged logits, using the ensemble of features from multiple teachers is another line of research. (Park & Kwak, 2019) proposed to train the student’s feature map to minimize the gap from the feature maps of multiple teachers with different feature transformation functions. While achieving promising performance, traditional KD methods using multiple teachers suffer from expensive computations because they require multiple pre-trained teachers. To alleviate this problem, Zhang et al. (2018) proposed a deep mutual learning approach, which was an initial form of online KD that has been developed by various studies (Lan et al., 2018; Anil et al., 2018; Chen et al., 2020a; Chung et al., 2020; Kim et al., 2020). In online KD, a set of student models, or peers, was trained simultaneously by learning from each other in a peer-teaching fashion.

In traditional KD, the teacher model is fixed during KD. However, this can be sub-optimal because the generalization ability of the student model is dynamic during training. A number of studies explored KD with an evolving teacher model to keep a reasonable capacity difference between student and teacher (Mirzadeh et al., 2020; Shi et al., 2021; Zhou et al., 2021b; Park et al., 2021). For example, Jin et al. (2019) designed a sequence of intermediate targets to impose curriculum-based constraint on the optimization path of the student model for improved KD. Shi et al. (2021) and Park et al. (2021) proposed to jointly update the teacher model and the student model with task specific objectives during KD.

Knowledge distillation is a widely-used approach to get a smaller but more competitive model. However, although the idea of KD is easy to implement, it also has several limitations. First, the performance of KD is very sensitive to the size gap between the teacher model and the student model. The discrepancy between the expressive power of the models would make it hard to teach the student model. Second, it relies on training data and may not be suitable for few-shot or zero-shot settings. In addition, recent studies (Xu et al., 2021a; Stanton et al., 2021) have revealed that while knowledge distillation can effectively improve student generalization, there was still a large discrepancy between the predictive distributions of student and teacher models. It means that there is still a long way to distill full knowledge in a teacher model to a student model.

Chapter 5 Efficient Data Usage

Following the definition of Green deep learning, this chapter mainly explores how to achieve competitive results with fewer data resources, including active learning and pre-training. It is worth noticing that although pre-trained models take massive computations during pre-training, they are widely believed to be a practical solution to release the burden of data in downstream tasks. Therefore, we also include pre-trained models in this chapter.

Active learning is a research direction aiming at using as few samples as possibles to achieve good results. It is initially proposed to reduce annotation costs. Nowadays, pool-based active learning is widely-used to reduce training costs by selecting the most useful examples to train networks. The intuitive behind active learning is quite simple. The annotated training data do not equally contribute to the final performance. If we can always select the most useful example to train models, the wasted training on unimportant examples can be largely reduced.

Active learning usually starts from a randomly-initialized model or a pre-trained model. It defines several query strategies to select new unlabeled data to query annotation. The new data associated with labels are then used to train the model. This process keeps running until it reaches the maximum number of labelled data or other termination conditions are satisfied. Previous active learning approaches mainly focused on query strategies to improve performance. Following the work of Yoo & Kweon (2019), given a pool of unlabeled data, we classify active learning approaches into three categories according to the selection criteria: uncertainty-based approaches, diversity-based approaches, and expected-model-change based approaches.

Uncertainty-based active learning is used to be the most common choice in active learning, using uncertainty scores to select data (Ranganathan et al., 2017; Schröder et al., 2021; He et al., 2019; Shen et al., 2018). The simplest solution is to utilize class posterior probabilities to define uncertainty. To be specific, Lewis & Gale (1994) used the probability of a predicted class as uncertainty score. Joshi et al. (2009) defined an entropy of class posterior probabilities as uncertainty score. Another research line is to train multiple models to construct a committee, and used the committee to evaluate uncertainty (Beluch et al., 2018). Recently, uncertainty-based active learning has been widely applied to various fields. Ranganathan et al. (2017) applied active learning on image classification which selected the most informative unlabeled samples to train a deep belief network model. Shen et al. (2018) applied uncertainty-based active learning to sequence tagging. They selected sentences for which the length-normalized log probability of the current prediction was the lowest. Ducoffe & Precioso (2018) focused on margin-based active learning for deep networks. Despite promising effectiveness, uncertainty-based sampling can easily lead to insufficient diversity of batch query samples.

Diversity-based active learning has been proposed to address the challenges of uncertainty based approaches. For example, Sener & Savarese (2018) defined a core-set to estimate the whole training data. Nguyen & Smeulders (2004) proposed to incorporate clustering into active learning. It first constructed a classifier on the set of the cluster representatives, and then propagated the classification decision to the other samples via a local noise model.

Expected-model-change based active learning selected data points that would cause the greatest change to current model. For example, Roy & McCallum (2001) selected examples according to the reduced error rate on future test examples. Freytag et al. (2014) measured the expected change of model outputs and incorporated the underlying data distribution. For each example of an unlabeled set, the expected change of model predictions was calculated and marginalized over the unknown label.

2 Pre-training as Few-shot Learners

It is widely-believed that pre-trained models as initialization is an effective approach to reduce data requirements in downstream tasks. In this survey, we describe widely-used pre-trained models.

self-supervised learning (SSL) is the most popular solution to get pre-trained models. We take NLP as an example to review recent progress of self-supervised learning. Self-supervised objectives can be classified into types including masked language modeling (MLM), language modeling (LM), de-nosing auto-encoding (DAE). In this survey, we give an overview of these models. We refer the reader to Qiu et al. (2020) and Han et al. (2021a) for more details of pre-trained networks.

Devlin et al. (2019) proposed to pre-train an Transformer encoder, i.e., BERT, via MLM objective on unlabeled text. MLM builds a corrupted token sequence where 15% tokens are randomly sampled and replaced with a special token [MASK], then requires the Transformer to predict the original tokens. Formally, given a sequence $\bm{x}_{1:L}=[x_{1},x_{2},\cdots,x_{L}]$ and the masked token set $\mathcal{M}$, the MLM objective can be formulated as

where $\bm{x}_{\setminus\mathcal{M}}$ indicates the unmasked part of the input sequence. With the inherited knowledge, fine-tuned BERT performs well compared to baselines without pre-training on many classification tasks, including single sentence classification (Warstadt et al., 2019; Socher et al., 2013), sentence pair classification or matching (Dolan & Brockett, 2005; Cer et al., 2017), natural language inference (Williams et al., 2018; Wang et al., 2019a; Bentivogli et al., 2009; Levesque et al., 2012), and question answering (Rajpurkar et al., 2016, 2018), etc.

Several studies have made effort to improve MLM through developing more effective MLM-like objectives or exploring more efficient training tricks. SpanBERT (Joshi et al., 2020), ERNIE (Sun et al., 2019b), and BERT-WWM (Cui et al., 2019) proposed to mask adjacent tokens. ELECTRA (Clark et al., 2020) proposed a GAN-like replaced token prediction objective which required discriminator to discriminate whether a token is replaced or not. Instead of masking, StructBERT (Wang et al., 2020b) learned to predict the shuffled span in its original order, which incorporated language structures into pre-training.

predicts next tokens one by one in an auto-regressive way. Specifically, given a sequence $\bm{x}_{1:L}=[x_{1},x_{2},\cdots,x_{L}]$, the joint probability of $\bm{x}$ can be written as

while $x_{0}$ is the special [BOS] token which indicates the beginning of a sentence. LM is usually implemented with a Transformer decoder. GPT-2 (Radford et al., 2019), and GPT-3 (Brown et al., 2020) are two representative LM-based pre-trained models. The LM objective directly models the probability of next token given its left context. LM-based pre-trained models as initialization can largely improve conditional natural language generation tasks like summarization (Rush et al., 2015; See et al., 2017) and question answering (Rajpurkar et al., 2018; Reddy et al., 2019). Therefore, LM models usually are used to initialize generative models.

There are lots of follow-up studies improving the original LM objective from various aspects. For example, inspired by the fact that neural networks could have different reading orders with human, XLNet (Yang et al., 2019) proposed to use permuted language modeling to predict a sentence in permuted orders in an auto-regressive way. UniLM (Dong et al., 2019) proposed a prefix LM where next tokens are predicted in an auto-regressive way with all context tokens visible to each other. ProphetNet (Qi et al., 2020) introduced a future $n$-gram prediction objective to predict the next $n$ tokens simultaneously based on previous context at each time step, through a multi-stream attention similar to XLNet.

requires a model to recover the original sentence based on the corrupted version. Formally, DAE-like objectives can be formulated as

where $\bm{\hat{x}}$ is the corrupted input. By learning to distinguish which part of content is corrupted in a text and recovering it in a natural order, DAE-based pre-trained models as initialization can improve various models, including language understanding and language generation.

Actually, there are also flexible variants to corrupt a sequence, such as shuffling, masking, dropping, rotating, etc. BART (Lewis et al., 2020) corrupted its input sequence with arbitrary noise transformations. The transformations include token masking, token deletion, sentence permutation, document rotation, and text infilling. More recently, Zhou et al. (2021a) proposed to use text rewriting instead of text infilling (e.g., BART, T5) for improving seq2seq pre-trained transformers.

Another line of SSL objectives is multilingual SSL objectives. For example, mBERT (Devlin et al., 2019) https://github.com/google-research/bert/blob/master/multilingual.md, mBART (Liu et al., 2020b), mT5 (Xue et al., 2021) pre-trained the multilingual version BERT, BART, T5 via the multilingual masked language modeling objective. These models highly improve the results of few-shot or low-shot multilingual learning or multilingual generation tasks. In addition to these simple variants, recent researchers also designed more sophisticated cross-lingual objectives, such as cross-lingual word recovery, cross-lingual paraphrase classification, and cross-lingual masked language modeling.

2.2 Contrastive Learning

Contrastive learning focuses on pair-wise relationships, aiming at learning closer or similar representation for positive samples while pushing negative samples away. The recent years have witnessed the rapid progress of contrastive-based pre-trained models, especially in CV (van den Oord et al., 2018; He et al., 2020; Chen et al., 2020d, 2021b).

The standard contrastive learning utilizes positive and negative pairs at the same time to construct its objective. This kind of loss can back to noise contrastive estimation (NCE) loss (Gutmann & Hyvärinen, 2010) that is defined as

where $q$ is the anchor sample; $k^{+}$ and $k^{-}$ are its positive sample and negative sample; $q$, $k^{+}$ and $k^{-}$ are vectors generated by a neural network; $f(\cdot,\cdot)$ is a similarity function; $\tau$ is the temperature controlling the concentration of the induced distribution. When more than one negative samples exist, the NCE loss becomes the InfoNCE loss:

where $k_{i}$ represents the $i$-th negative sample of anchor/query sample $q$; $K$ is the size of negative samples. van den Oord et al. (2018) proved that minimizing the InfoNCE loss was equivalent to maximizing the lower bound of mutual information between $q$ and $k^{+}$:

where the more negative samples are, the tighter the lower bound is.

Overall, contrastive losses are easy to implement and contrastive-based pre-trained models generally have strong transfer ability. In the CV field, it largely improve downstream tasks, such as ImageNet classification (Deng et al., 2009), object detection (Everingham et al., 2010), and egmentation (Lin et al., 2014), etc. MoCo (He et al., 2020; Chen et al., 2020e, 2021b) is one of representative models which applied advanced contrastive learning methods to train pre-trained models on CV fields.

Existing state-of-the-art contrastive-based pre-trained models are simple variants of Eq. 5.6 where positive pairs and negative paris are required. For example, Deep InfoMax applied contrastive losses to learn image representations via maximizing the mutual information between local patches and the whole image; CMC (Tian et al., 2020) applied contrastive losses to learn representations where different views of the same scene or instance were sampled as positive pairs. According to Wang & Isola (2020), the InfoNCE loss optimized two properties of a representation space, including the alignment of representations between positive samples and the uniformity of the induced distribution of normalized representations on a hypersphere in the representation space. Nevertheless, the recent studies such as BYOL (Grill et al., 2020), SwAV (Caron et al., 2020), Simsiam (Chen & He, 2021) found that contrastive learning also works without negative samples. In all, the research of contrastive-based pre-training is still in rapid development. More understanding studies are expected to better explain how contrastive losses work.

2.3 Prompt Learning

As the size of pre-trained models grows rapidly, fine-tuning the super large pre-trained model usually requires massive data to get better generalization ability, thus failing on few-shot/zero-shot applications. The objective gap between pre-training and fine-tuning stages are one of important reasons behind failure. Therefore, to improve data efficiency, prompt learning is proposed by extracting the similar/same template for pre-training and fine-tuning stages. Scao & Rush (2021) demonstrated that a prompt may be equal to 100 conventional data points, indicating that prompts can greatly improve the sample efficiency.

In the recent survey about prompt learning (Liu et al., 2021a), prompt is defined as $\bm{x}^{\prime}$ = $f_{\text{prompt}}(\bm{x})$, where $f_{\text{prompt}}(\cdot)$ is the prompting function to rewrite the input $\bm{x}$ with an task-specific template. Suppose that we have a review “I dozed off when watching this movie.” as input $\bm{x}$ associated with sentiment label $\bm{y}$. A prompt example $\bm{x}^{\prime}$ is “I dozed off when watching this movie. It is so [MASK]”. In this way, we transfer sentiment prediction to a masked language modeling task. As pointed by Wei et al. (2021), language models at scale like GPT-3 (Brown et al., 2020) contain substantial world knowledge and can perform a range of NLP tasks, which makes large pre-trained models a kind of “neural knowledge bases” (Petroni et al., 2019). In fact, prompts can be seen as queries for those neural knowledge bases. The difficulty is (1) how to generate appropriate queries to achieve your goals, and (2) how to understand or interpret the response in the format of predicted textual content.

Generally, prompts are defined as discrete templates. In addition, there are also studies focusing on continuous prompts, rather than textual prompts. For example, Prefix Tuning (Li & Liang, 2021) and P-tuning (Liu et al., 2021c) defined virtual tokens where only parameters of virtual tokens were fine-tuned. The more recent P-tuning V2 (Liu et al., 2021b) used multi-task learning objectives and obtained competitive even better results than vanilla fine-tuning. SPoT (Vu et al., 2021) further utilized transfer learning to support unseen tasks, where the prefix was pre-tuning on related tasks then used for initialization in unseen task. Overall, these methods are ideologically similar to adaptation approaches (Houlsby et al., 2019; Bapna & Firat, 2019; Pfeiffer et al., 2020) because both of them do not change the most of parameters of the pre-trained model. However, the adaptation methods usually insert adaptors between layers, undermining the original model architecture.

Chapter 6 Conclusions and Future Directions

We believe that Green deep learning will be an important research direction in the future. With recent advances in deep learning, the community have made significant progress in developing super-large models for downstream tasks, making it possible to apply AI models on complicated applications. Considering that it is the ultimate goal to deploy AI models on real-world devices with high requirements on resource usage, how to transfer strong models to resource-constraint devices (e.g., mobile) becomes a priority goal. In this section, we list several challenges for Green deep learning.

Green deep learning theory. We have harnessed some advanced Green deep learning techniques, but many questions still remain to be explored. For example, 1) If we already have a well-performing model, why was additional training required to transfer knowledge to a small model? 2) How many parameters do we need at least for feasible training and inference? 3) How to design Green learning algorithms to enable efficient zero-shot learning or few-shot learning like human? 4) How our model store knowledge and how to make models to achieve lifelong learning without forgetting learned knowledge during training? 5) Is linear algebra is the only basic theory for deep learning and whether can we develop a new operation system beyond linear algebra? In this survey, we just show limited questions due to page limitation. The community still have a long way to go in the theory of Green deep learning

Green deep learning under extreme computation constraints. Deploying models on edge devices enjoys multiple advantages. First, it can avoid uploading user information to the cloud amid the tide of protecting user privacy. Second, it can empower more applications with high requirements on latency if deployed a light-weight model on edge devices. While recent advances in machine learning greatly facilitate efficient training and inference in the cloud, edge devices bring new challenges caused by extremely strict memory and computation constraints. Therefore, how to design advanced training and inference algorithms towards tiny devices is also an important and challenging direction. First, algorithm-hardware cooperation is a promising direction to satisfy speed requirements when deploying large models. For example, Lightseq (Wang et al., 2021b) and Faster Transformer are two representative models that use CUDA implementation to accelerate Transformer execution. Second, edge-cloud cooperation is also a practical solution by combing powerful and elastic cloud computing and immediate edge computing. An intuitive solution is to design a dynamic network where edge devices can handle simple samples with smaller models and the cloud is responsible for handing complicated samples with larger models.

References