Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition

Introduction

Recently, with the advancement of deep learning, great progress has been made in end-to-end (E2E) automatic speech recognition (ASR). With the goal of directly mapping a sequence of speech frames to a sequence of output tokens, an E2E ASR system incorporates the acoustic model, language model and pronunciation model of a conventional ASR system into a single deep neural network (DNN). The most dominant approaches for E2E ASR include connectionist temporal classification (CTC) , recurrent neural network transducer (RNNT) and attention-based encoder-decoder (AED) models .

However, the performance of E2E ASR degrades significantly when an acoustic mismatch exists between training and test conditions. An intuitive solution is domain adaptation where a well-trained source-domain E2E model is adapted to the data in the target domain. Different from speaker adaption, domain adaptation allows for the usage of a large amount of adaptation data in both source and target domains.

There has been plenty of domain adaptation methods for hybrid systems that we can leverage for adapting E2E systems. One popular approach is the adversarial learning in which an intermediate deep feature or a front-end speech feature is learned to be invariant to the shifts between source and target domains. Adversarial domain adaptation is suitable for the situation where no transcription or parallel adaptation data in both domains are available. It can also effectively suppress the environment and speaker variability during domain adaptation. However, in speech area, a parallel sequence of target-domain data can be easily simulated from the source-domain data such that the speech from both domains are frame-by-frame synchronized. To take advantage of this, teacher-student (T/S) learning was proposed for the unsupervised domain adaptation of acoustic models in DNN-hidden Markov model (HMM) hybrid systems . In T/S learning, the Kullback-Leibler (KL) divergence between the output senone distributions of teacher and student acoustic models given parallel source and target domain data at the input is minimized by updating only the student model parameters. T/S training was shown to outperform the cross entropy training directly using the hard label in the target domain .

One drawback of unsupervised T/S learning is that, the teacher model is not perfect and will sometimes make inaccurate predictions that mislead the student model toward suboptimal directions. To overcome this, one-hot ground-truth labels are used to compensate for teacher’s imperfections. Hinton et al. proposed interpolated T/S (IT/S) learning to interpolate the teacher’s soft class posteriors with one-hot ground truth using a pair of globally fixed weights. However, the optimal weights are data-dependent and can only be determined through careful tuning on a dev set. More recently, conditional T/S (CT/S) learning was proposed in where the student model selectively chooses to learn from either the teacher or the ground truth depending on whether the teacher’s prediction is correct or not. CT/S does not disturb the statistical relationships among classes naturally embedded in the class posteriors and achieves significant word error rate (WER) improvement over T/S for domain adaptation on CHiME-3 dataset .

In this work, we focus on the domain adaptation of AED models for E2E ASR by using T/S learning which was previously applied to learn small-footprint AED models in by distilling knowledge from a large powerful teacher AED. For unsupervised domain adaptation, we extend T/S learning to AED models by introducing a two-level knowledge transfer: in addition to learning from the teacher’s soft token posteriors, the student AED also conditions its decoder on the one-best token sequence decoded by the teacher AED.

We further propose an adaptive T/S (AT/S) learning method to improve T/S learning using ground-truth labels. By taking advantage of both IT/S and CT/S, AT/S adaptively assigns a pair of weights to the teacher’s soft token posteriors and the one-hot ground-truth label at each decoder step depending on the confidence scores on each of the labels. The confidence scores are dynamically estimated as a function of soft and one-hot labels. The student AED learns from an adaptive linear combination of both labels. AT/S inherits the linear interpolation of soft and one-hot labels from IT/S and borrows from CT/S the judgement on the credibility of both knowledge sources before merging them. It is expected to achieve improved performance over the other T/S methods for domain adaptation. As a general deep learning method, AT/S can be widely applied to the domain adaptation or model compression of any DNN.

With 3400 hours close-talk and far-field Microsoft Cortana data for domain adaptation, T/S learning achieves up to 24.9% and 6.3% relative WER gains over close-talk and far-field baseline AEDs, respectively. AT/S improves the close-talk and far-field AEDs by 28.2% and 10.3%, respectively, consistently outperforming IT/S and CT/S.

Attention-Based Encoder-Decoder (AED) Model

In this work, we perform domain adaptation on AED models . AED model was first introduced in for neural machine translation. Without any conditional independence assumption as in CTC , AED was successfully applied to to E2E ASR in and has recently achieved superior performance to conventional hybrid systems in .

AED directly models the conditional probability distribution $P(\mathbf{Y}|\mathbf{X})$ over sequences of output tokens $\mathbf{Y}=\{y_{1},\ldots,y_{L}\}$ given a sequence of input speech frames $\mathbf{X}=\{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\}$ as below:

where $\mathbf{q}_{l}$ is the hidden state of the decoder RNN. bias $\mathbf{b}_{y}$ and the matrix $K_{y}$ are learnable parameters.

where $\mathbf{Y}^{G}=\{y^{G}_{1},\ldots,y^{G}_{L^{G}}\}$ is the sequence of grouth-truth tokens, $L^{G}$ represents the number of elements in $\mathbf{Y}^{G}$ and $\theta$ denotes all the model parameters in AED.

T/S Learning for Unsupervised Domain Adaptation of AED

For unsupervised domain adaptation, we want to make use of a large amount of unlabeled data that is widely available. As shown in Fig. 1, with T/S learning, only two sequences of parallel data are required: an input sequence of source-domain speech frames to the teacher AED $\mathbf{X}^{T}=\{\mathbf{x}^{T}_{1},\ldots,\mathbf{x}^{T}_{N}\}$ and an input sequence of target-domain speech frames to the student model $\mathbf{X}^{S}=\{\mathbf{x}^{S}_{1},\ldots,\mathbf{x}^{S}_{N}\}$. $\mathbf{X}^{T}$ and $\mathbf{X}^{S}$ are parallel to each other, i.e, each pair of $\mathbf{x}^{S}_{n}$ and $\mathbf{x}^{T}_{n},\forall n\in\{1,\ldots,N\}$ are frame-by-frame synchronized. For most domain adaptation tasks in ASR, such as adapting from clean to noisy speech, close-talk to far-field speech, wide-band to narrow-band speech, the parallel data in the target domain can be easily simulated from the data in the source domain .

Our goal is to train a student AED that can accurately predict the tokens of the target-domain data by forcing the student to emulate the behaviors of the teacher. To achieve this, we minimize the Kullback-Leibler (KL) divergence between the token-level output distributions of the teacher and the student AEDs given the parrallel data $\mathbf{X}^{T}$ and $\mathbf{X}^{S}$ are fed as the input to the AEDs. The KL divergence between the token-level output distributions of the teacher and student AEDs are formulated below

where $\mathbf{Y}^{T}=\{y^{T}_{1},\ldots,y^{T}_{L^{T}}\}$ is the sequence of one-best token sequence decoded by the teacher AED as follows

The steps of token-level T/S learning for unsupervised domain adaptation of AED model are summarized as follows:

Clone the student AED from a teacher AED well-trained with transcribed source-domain data by minimizing Eq. (4).

Compute error signal of the T/S loss function in Eq. (7) , back-propagate the error through student AED and update the parameters of the student AED.

After T/S learning, only the adapted student AED is used for testing and the teacher AED is discarded.

From Eqs. (6) and (7), to extend T/S learning to AED-based E2E models, two levels of knowledge transfer are involved: 1) the student learns from the teacher’s soft token posteriors $P(u|\mathbf{Y}^{T}_{0:l-1},\mathbf{X}^{T};\mathbf{\theta}^{T})$ at each decoder step; 2) the student AED conditions its decoder on the previous token $y^{T}_{l-1}$ predicted by the teacher to make the current prediction.

Sequence-level T/S learning is another method for unsupervised domain adaptation in which a KL divergence between the sequence-level output distributions of the teacher and student AEDs are minimized. Equivalently, we minimize the sequence-level T/S loss function below with respect to $\theta^{S}$

From Eq. (8), we see that only one level of knowledge transfer exists in sequence-level T/S, i.e., the one-best token sequence $\mathbf{Y}^{T}$ decoded by the teacher AED. The student AED learns from $\mathbf{Y}^{T}$ and conditions its decoder on it at each step. Different from token-level T/S, in sequence-level T/S, one-hot labels in $\mathbf{Y}^{T}$ are used as training targets of the student AED instead of the soft token posteriors.

Adaptive T/S (AT/S) Learning for Supervised Domain Adaptation of AED

In this section, we want to make good use of the ground-truth labels of the adaptation data to further improve the T/S domain adaptation. Note that different from unsupervised T/S in Section 3, in supervised domain adaptation, the teacher AED conditions its decoder on the ground-truth token instead of its previous decoding result because the token transcription $\mathbf{Y}^{G}$ is available in addition to $\mathbf{X}^{S}$ and $\mathbf{X}^{T}$.

One shortcoming of unsupervised T/S learning is that the teacher model can sporadically predict inaccurate token posteriors which misleads the student AED towards suboptimal performance. One-hot ground-truth labels can be utilized to alleviate this issue. One possible solution is the interpolated T/S (IT/S) learning in which a weighted sum of teacher’s soft posteriors and the one-hot ground truth is used as the target to train the student AED. A pair of global weights summed to be one is applied to each pair of soft and one-hot labels. However, the optimal global weights are hard to determine because they are data-dependent and need to be carefully tuned on a dev set.

To address this issue, conditional T/S learning (CT/S) was proposed recently in which the student selectively chooses to learn from either the teacher AED or the ground truth conditioned on whether the teacher AED can correctly predict the ground-truth labels. CT/S have shown significant WER improvements over T/S and IT/S for both domain and speaker adaptation on CHiME-3 dataset. However, in CT/S, the student is still not “smart” enough because, for each token, the student AED solely relies on either the teacher’s posteriors or the ground truth instead of dynamically extracting useful knowledge from both.

In order to quantify the value of the knowledge to be transferred, $w_{l}$ should be positively correlated with a confidence score $c_{l}$ on the teacher’s prediction on token posteriors, while $(1-w_{l})$ should be positively correlated with a confidence score on the ground truth $d_{l}$. To achieve this, we compute $w_{l}$ by normalizing $c_{l}$ against its summation with $d_{l}$.

where both $f_{1}$ and $f_{2}$ are any monotonically increasing functions on the interval $$. In this work, we simply assume that$f_{1}$and$f_{2}$are both power functions of the same form, i.e.,$f_{1}(x)=f_{2}(x)=x^{\lambda},\;\lambda>0$. Note that$w_{l}$equals to$P(y^{G}_{l}|\mathbf{Y}^{G}_{0:l-1},\mathbf{X}^{T};\theta^{T})$when$\lambda=1$.

In AT/S, a linear combination of the teacher’s soft posteriors and the one-hot ground truth weighted by $w_{l}$ and $(1-w_{l})$, respectively, is used as the training target for the student AED at each decoder step. The AT/S loss function is formulated as

The steps of AT/S learning for supervised domain adaptation of AED model are summarized as follows:

Perform token-level unsupervised T/S adaptation by following the steps in Section 3 as the initialization.

Compute the confidence scores $c_{l}$ and $d_{l}$ for teacher’s soft posteriors and one-hot vector of ground truth $y^{G}_{l}$ by Eqs. (10) and (11), compute the adaptive weight $w_{l}$ by Eq. (9).

Compute error signal of the AT/S loss function in Eq. (12) , back-propagate the error through student AED and update the parameters of the student AED.

AT/S is superior to IT/S in that the combination weights for soft and one-hot labels at each decoder step are adaptively assigned according to the confidence score on both labels. AT/S will degenerate to IT/S if the combination weights $w_{l}$ are fixed globally. Compared to CT/S, in AT/S, the student always adaptively learns from both the teacher’s soft posteriors and the one-hot ground truth rather than choosing either of them depending on the correctness of teacher’s prediction.

Experiments

We adapt a close-talk AED model to the far-field data through various T/S learning methods with parallel close-talk and far-field Microsoft Cortana data for E2E ASR.

For both training and adaptation, close-talk data consisting of 3400 hours of Microsoft live US English Cortana utterances are collected through a number of deployed speech services including voice search and SMD. We simulate 3400 hours of far-field Microsoft Cortana data by convolving the close-talk signal with different room impulse responses and adding various environmental noise for both training and adaptation. The 3400 hours far-field data is parallel with the 3400 hours close-talk data. We collect 17.5k far-field utterances (about 19 hours) from Harman Kardon (HK) speaker as the test set.

80-dimensional log Mel filter bank features are extracted from the training, adaptation and test speech every 10 ms over a 25 ms window. We stack 3 consecutive frames and stride the stacked frame by 30 ms, to form a sequence of 240-dimensional input speech frames. We first generate 34k mixed-units consisting of words and multi-letter units as in based on the training transcription and then tokenize the training, adaptation transcriptions correspondingly. We insert a special token between every two adjacent words to indicate the word boundary and add , to the beginning and end of each utterance, respectively.

2 AED Baseline System

We first train an AED model predicting 34k mixed units with 3400 hours close-talk training data and it ground-truth labels for E2E ASR as in . The encoder is a bi-directional gated recurrent units (GRU)-recurrent neural network (RNN) with 6 hidden layers, each with 512 hidden units. We use GRU instead of long short-term memory (LSTM) for RNN because it has less parameters and is trained faster than LSTM with no loss of performance. Layer normalization is applied for each encoder hidden layer. Each mixed unit is represented as a 512-dimensional embedding vector. The decoder is a uni-directional GRU-RNN with 2 hidden layers, each with 512 hidden units. The 34k-dimensional output layer of the decoder predicts the posteriors of all the mixed units in the vocabulary. During training, scheduled sampling is applied to the decoder with a sampling probability starting at 0.0 and gradually increasing to 0.4 . Dropout with a probability of 0.1 is used in both encoder and decoder. A label-smoothed cross-entropy loss is minimized during training. Greedy decoding is performed to generate the ASR transcription. We use PyTorch toolkit for the experiments. Table 1 shows that the close-talk AED model achieves 7.58% and 17.39% WERs on a close-talk Cortana test set used in and the far-field HK speaker test set, respectively.

Using the well-trained close-talk AED as the initialization, we then train a far-field AED with 3400 hours far-field data and its ground-truth labels by following the same procedure. When evaluated on the HK speaker test set, the baseline far-field AED achieves 13.93% WER for ASR as in Table 1.

3 Unsupervised Domain Adaptation with T/S Learning

We adapt the close-talk baseline AED to the 3400 hours far-field data using token and sequence level T/S learning as discussed in Section 3. To achieve this, we feed the 3400 hours close-talk adaptation data as the input to the teacher AED and the 3400 hours parallel far-field adaptation data as the input to the student AED. The student AED conditions its decoder on one-best token sequences generated by the teacher AED through greedy decoding. In token-level T/S, the soft posteriors generated by the teacher serve as the training targets of the student while in sequence-level T/S, the one-best sequences decoded by the teacher are used the targets.

As shown in Table 1, the token-level T/S achieves 13.06% WER on HK speaker test set, which is 24.9% and 6.25% relative improvements over the close-talk and far-field AED models, respectively. The sequence-level T/S achieves 14.00% WER, which is 19.5% relative improvement over the close-talk AED model. The sequence-level T/S performs slightly worse than the far-field AED trained with ground-truth labels because the one-best decoding from the teacher AED is not always reliable to serve as the training targets for the student model. The sequence-level T/S can be improved by using multiple decoded hypotheses generated by the teacher AED as the training targets as in . We did not perform N-best decoding because it will drastically increase the computational cost and will consumes much more adaptation time than the other T/S methods. The 6.7% relative WER gain obtained by token-level T/S over sequence-level T/S shows the benefit of using soft posteriors generated by the teacher AED as the training target at each decoder step when a reliable ground-truth transcription is not available.

The 6.3% relative WER gain of token T/S over far-field AED baseline shows that the unsupervised T/S learning with no ground-truth labels can significantly outperform the supervised domain adaptation with such information available. Compared to the one-hot labels, the soft posteriors accurately models the inherent statistical relationships among different token classes in addition to the token identity encoded by a one-hot vector. It proves to be a more powerful target for the student to learn from which is consistent with what was observed in .

4 Supervised Domain Adaptation with AT/S Learning

As discussed in Section 4, we want to further improve the T/S learning by using one-hot ground-truth labels when they are available. As in , we perform IT/S learning for supervised domain adaptation by using the linear interpolation of soft posterior and one-hot ground truth as the training target of the student. The interpolation weights are globally fixed at 0.5 and 0.5 for all decoder steps. By following , we also conduct CT/S for supervised domain adaptation where soft posteriors are used as the training target of the student if the teacher’s prediction is correct at the current decoder step, otherwise the one-hot ground truth is used as the target. Finally, AT/S domain adaptation is performed by adaptively adjusting the weights assigned to the soft and one-hot labels at each decoder step as in Eqs. (9) to (11). We explore using different power functions as $f_{1}(x)$ and $f_{2}(x)$ to compute the confidence scores by adjusting $\lambda$. For all the above supervised T/S learning methods, the 3400 hours close-talk and 3400 hours far-field parallel adaptation data is fed as the input to the teacher and student AEDs, respectively.

As shown in Table 1, IT/S with $w=0.2$ achieves 13.95% WER on HK speaker test set which is 25.5%, 7.0% and 0.8% relative improvements over the close-talk, far-field and token-level T/S adapted AED models, respectively. With a 12.82% WER, CT/S relatively improves the close-talk, far-field and token-level T/S adapted AED models by 26.3%, 8.0% and 1.8% respectively. Among different $\lambda$s for AT/S, the best WER is 12.49%, which is 28.2%, 10.3% and 4.4% relative gains over close-talk, far-field and token-level T/S adapted AEDs. The minimum WER is reached when $\lambda=0.25$ and $c_{l}=P(y_{l}|\mathbf{Y}^{G}_{0:l-1},\mathbf{X}^{T};\theta^{T})^{0.25}$. Compared to $\lambda>1$, AT/S works better for $\lambda\in$ when confidence scores $c_{l}$, $d_{l}$ are both concave functions of the correct token posterior and the sum of incorrect token posteriors, respectively. All the IT/S, CT/S and AT/S outperform the unsupervised T/S learning indicating that the one-hot ground truth can further improve T/S domain adaptation when it is properly used. AT/S achieves the largest gain in supervised domain adaptation methods showing the superiority of adaptively extracting useful knowledge from both the soft and one-hot labels depending on their confidence scores.

Conclusion

In this paper, we extend T/S learning to unsupervised domain adaptation of AED models for E2E ASR. T/S learning requires only unlabeled parallel source and target domain data as the input to the teacher and student AEDs, respectively. In T/S, the student AED conditions its decoder on the one-best token sequences generated by the teacher. The teacher’s soft posteriors and decoded one-hot tokens are used as the training target of the student AED for token-level and sequence-level T/S learning, respectively.

For supervised domain adaption, we propose adaptive T/S learning in which the student always learns from a linear combination of the teacher’s soft posteriors and the one-hot ground truth. The combination weights are adaptively computed at each decoder step based on the confidence scores on both knowledge sources.

Domain adaptation is conducted on 3400 hours close-talk and 3400 hours far-field Microsoft Cortana data. Token-level T/S achieves 6.3% relative WER improvement over the baseline far-field AED model trained with CE criterion. By making use of the ground-truth labels, AT/S further improves the token-level T/S by 4.4% relative and achieves a total 10.3% relative gain over the far-field AED. AT/S also consistently outperforms IT/S and CT/S showing the advantage of learning from both the teacher and the ground truth as well as the adaptive adjustment of the combination weights.

References