The RWTH ASR System for TED-LIUM Release 2: Improving Hybrid HMM with SpecAugment

Introduction & Related Work

One of the most common neural network (NN) based acoustic modeling methods is the hybrid hidden Markov model (HMM) approach , which still gives state-of-the-art performance, as recently shown for benchmarks like Librispeech and Switchboard . Bi-directional long short-term memory (BLSTM)-HMM are widely used for acoustic modeling in hybrid HMM systems. Based on the alignment generated from a Gaussian mixture model (GMM)-HMM baseline, cross-entropy (CE) training is usually applied to train the baseline NN models. Additionally, speaker adaptive training (SAT) using i-vectors and sequence discriminative training, such as the state-level minimum Bayes risk (sMBR) criterion, are often applied for further improvements.

Language models (LM) based on LSTM have been widely applied to automatic speech recognition (ASR). Large improvements are observed for both hybrid HMM systems and end-to-end systems . Transformer based LMs are reported to further improve over LSTM LMs . For hybrid HMM systems, they are usually applied in lattice rescoring , but also may be used in single-pass search .

SpecAugment , as a simple feature augmentation method, has been successfully applied to end-to-end speech recognition systems. With increased model size and training time, end-to-end systems benefit strongly from SpecAugment , and large improvements are also reported for end-to-end speech translation . However, its effect on hybrid HMM systems has not been thoroughly studied yet.

In this work, we describe a complete training pipeline to build a state-of-the-art hybrid HMM-based ASR system on the 2nd release of the TED-LIUM corpus (TED-LIUM-v2). We apply SpecAugment in our training pipeline and obtain further improvement over our best SAT model using i-vectors. By investigating the effect of different maskings on hybrid models, we achieve improvements from SpecAugment without increasing model size and training time. And no additional effort on learning rate scheduling is needed. Subsequent sequence discriminative training using the sMBR criterion is used to fine-tune the final acoustic model. For language modeling, both LSTM and Transformer based LMs are trained and evaluated. Our best system outperforms the previous state-of-the-art by a large margin.

Baseline Acoustic Model

All acoustic models are trained on the 207 hours of training data of TED-LIUM-v2. The official dictionary with roughly 152k words and 160k pronunciations is used. To evaluate all intermediate acoustic models with recognition experiments, we use a fixed heavily pruned 4-gram LM, whose details are described in Sec. 5 denoted as ‘4-gram-small’. This allows us to simplify the tuning, and report the relative improvement of each step. All trainings are done using our NN modeling toolkit RETURNN , our ASR toolkit RASR and our work-flow manager Sisyphus . All recognition results are obtained with maximum a posteriori (MAP) Viterbi decoding.

2 Baseline

We follow the standard steps described in to train the GMM-HMM baseline. Starting with linear alignment, monophone GMMs are trained with 16-dimensional MFCCs and their first oder derivatives. With each triphone modeled by 3 HMM states, generalized triphone states are obtained by state tying using classification and regression tree (CART). We use 9k CART labels. Generalized triphone state GMMs are then trained on windowed MFCC with linear discriminant analysis (LDA) transformation. This step is repeated once to refine the CART labels with better alignment. Subsequently, vocal tract length normalization (VTLN) and SAT using constrained maximum likelihood linear regression are applied to further improve the GMMs. The final alignment from the VTLN-SAT GMM is used in the next step to train NN baseline with the CE criterion.

We use 80-dimensional logmel features for the NN training. The NN model contains six BLSTM layers with 512 units for each direction. This topology is used in all further steps. The Nesterov-accelerated adaptive moment estimation (Nadam) optimizer with an initial learning rate of 0.0009 is used. Greedy layer-wise pre-training and Newbob learning rate scheduling with a decay factor of 0.9 are applied. CE and focal loss with factor of 2 are used. The training set is split into 5 subepochs and models converge well with roughly 32 full epochs. Sequences are decomposed into chunks of 64 frames with 50% overlap and a mini-batch of 128 chunks is used. Additionally, a 10% dropout and $L_{2}$ regularization with a factor of $0.01$ are applied to all hidden layers.

Table 1 shows the word error rate (WER) results of each of the aforementioned training steps. We also try to use the BLSTM baseline to generate a new alignment and repeat the NN training, but no further improvement is obtained. This new alignment is used in further steps of training.

3 I-Vector Adaptation

We follow to apply SAT using i-vectors as speaker embeddings. The embeddings are concatenated to the logmel features at each frame. The universal background model (UBM) is trained on the whole training set. To train the UBM, logmel features with a context of 9 frames are concatenated and then reduced to a dimension of 60 with LDA. I-vectors are then estimated for each recording separately using all feature frames including non-speech. We follow to use a size of 100 for the i-vectors. As shown in the last two rows of Table 1, 6% relative improvement is achieved by applying SAT with i-vectors. We expect to achieve larger improvements with further tuning of the embedding parameters.

SpecAugment

The original SpecAugment applies time warping, time masking and frequency masking on logmel features. Since time warping is reported to have minor effect, we skip it in our training. This also avoids the additional effort to handle the alignment accordingly. We apply the two maskings on logmel features concatenated with i-vectors. Since i-vectors are included, the frequency masking is renamed as feature masking. Both maskings are bounded to the fixed chunk size and feature dimension, and are realized in a similar way as described in .

With a chunk of $T$ frames $(x_{1},...,x_{T})$, a position $t$ is randomly selected from $[1,T]$. Then a time mask of length $\Delta t$ is randomly selected from $[0,\Delta t_{\text{max}}]$, where $\Delta t_{\text{max}}$ is a predefined maximum time mask length. TM is then applied by setting the consecutive frames $(x_{t},...,x_{t+\Delta t})$ to . This procedure is repeated $m$ times, where $m$ is randomly selected from $[1,M]$. $M$ is a predefined maximum iteration number for TM. Thus, TM can be controlled by setting $M$ and $\Delta t_{\text{max}}$ accordingly, which we denote as $M\times\Delta t_{\text{max}}$.

2 Feature Masking (FM)

With features of dimension $D$, an index $d$ is randomly selected from $[1,D]$. Then a feature mask of length $\Delta d$ is randomly selected from $[0,\Delta d_{\text{max}}]$, where $\Delta d_{\text{max}}$ is a predefined maximum feature mask length. FM is then applied by setting the features within dimension $[d,d+\Delta d]$ to . This procedure is again repeated $n$ times, where $n$ is randomly selected from $[1,N]$. $N$ is a predefined maximum iteration number for FM. Similar to TM, FM can be controlled by setting $N$ and $\Delta d_{\text{max}}$ accordingly, which we denote as $N\times\Delta d_{\text{max}}$.

3 SpecAugment on Logmel with I-Vectors

To further improve the previous best baseline model, we directly apply the TM and FM on the 80-dimensional logmel features concatenated with 100-dimensional i-vectors. The random selections in both TM and FM are independently applied for each chunk in a batch. BLSTM models are trained from scratch. The predefined $M$ and $N$ are halved in the first 2000 steps for a more stable pre-training. We set a default $\Delta d_{\text{max}}$ to 10% of the feature dimension. With $D=180$, this means $\Delta d_{\text{max}}=18$. Then a default $N=5$ is used for a maximum of 50% FM (denoted as $5\times 18$). For hybrid HMM systems, CART labels consume much less frames than label units used in end-to-end systems. By setting a very large $\Delta t_{\text{max}}$, evidence of several continuous CART labels are masked out, which might be less beneficial. Therefore, we start with a default $\Delta t_{\text{max}}=5$ to match the maximum duration of a speech CART label based on our experience. Due to the fixed chunk size $T=64$, $M$ also has to be limited to keep a reasonable ratio of TM. With $\Delta t_{\text{max}}=5$, we set a default $M=6$ for roughly a maximum of 50% TM (denoted as $6\times 5$).

We first investigate the effect of different TM with the default FM ($5\times 18$). Under the same maximum ratio of TM, we compare a set of different $M\times\Delta t_{\text{max}}:\{15\times 2,6\times 5,3\times 10,2\times 15,1\times 30\}$ to find the optimal $\Delta t_{\text{max}}$. As shown in Table 2, too long TM gives less improvement, which matches our expectation. Surprisingly too short TM is also less beneficial, which should be resulted from the decreased effect of TM. With the optimal $\Delta t_{\text{max}}=10$, we further optimize $M$ by training models with $M:\{2,4\}$ to apply less and more TM (denoted as $2\times 10$ and $4\times 10$). From Table 2 we see neither of them brings further improvement. The best result achieves 7% relative improvement over the SAT baseline using i-vectors.

We then investigate the effect of different FM with the best TM ($3\times 10$). Similarly under the same 50% maximum ratio of FM, we compare a set of different $N\times\Delta d_{\text{max}}:\{10\times 9,5\times 18,3\times 30\}$ to find the optimal $\Delta d_{\text{max}}$. As shown in Table 2, our default FM setting still gives the best result. With optimal $\Delta d_{\text{max}}=18$, we train models with $N:\{3,7\}$ to vary the maximum ratio of FM (denoted as $3\times 18$ and $7\times 18$). Both give worse results. Additionally, we also investigate the importance of i-vectors in terms of FM. Since they are fixed for each frame, we train a model with default FM applied only within the logmel features and i-vectors are left untouched. This is reflected by the column ’FM on Ivec’ in Table 2. For 80-dimensional logmel features, 10% of the feature dimension results in $\Delta d_{\text{max}}=8$. The result of applying FM only within logmel features is much worse. This shows that including i-vectors for FM is essential, which brings certain variation also into the speaker features.

Finally, we also investigate the effect to continue training the i-vectors-based SAT baseline with SpecAugment. We use the best masking settings obtained so far, i.e. TM=$3\times 10$ and FM=$5\times 18$. In this case, we turn off the pre-training and its corresponding 2000 steps of halved masking. The learning rate is reset to allow an escape from local optimum. The model converges slightly faster than training from scratch directly with SpecAugment, but it only achieves the same performance of 9.1% WER in the end. Considering the much longer training time in total, there is not too much benefit to follow this track.

4 Discussion

Overall, the improvements from SpecAugment are not as large as reported for end-to-end systems . However, improvements are obtained without increasing model size and training time. Models converge well with roughly the same number of epochs as needed for the baseline training. Additionally, no careful design of learning rate scheduling is needed (only Newbob is applied here), although more improvements can be explored by doing this.

In general, end-to-end systems need larger amounts of training data to be competitive with state-of-the-art hybrid HMM systems. This situation is eased by training end-to-end systems with SpecAugment for many more epochs. Together with the results in this work, we tend to infer that in terms of SpecAugment, end-to-end systems benefit most from having more data, whereas hybrid HMM benefit from more variation introduced into the data. However, more investigation is needed for a thorough understanding.

Sequence Discriminative Training

We follow to further apply sequence discriminative training on the best model from the previous step. In this case, we take the SAT model using i-vectors trained with the best SpecAugment setting. We use a lattice-based version of sMBR training criterion to fine-tune the model weights. No SpecAugment is applied in this step. This converged CE model and a bi-gram LM trained on the TED-LIUM-v2 LM training data are used for lattice generation and initialization of model training. We then continue training with a small constant learning rate of $1\times 10^{-5}$ and use early stopping to prevent overfitting on the training data. CE smoothing with a scale of 0.1 is applied. As shown in Table 2, the sequence discriminative training achieves an additional 6% relative improvement.

Language Modeling

The LM training data consists of 7 subsets including the TED-LIUM-v2 training audio transcriptions, with a total of 270 M running words. The small 4-gram LM is trained in a similar way as the Kaldi example recipe . All the rest of our LMs have been described in . We refer readers interested in more details to this paper.

We first train modified Kneser-Ney 4-gram language models on each subset of the training data with the word level vocabulary of size 152K. We linearly interpolate these sub-LMs including a background 4-gram model trained on all training text, using the interpolation weights optimized for the development perplexity.

We train both LSTM and Transformer language models. The LSTM LM has 4 layers with 2048 nodes in each layer. The Transformer model has 32 layers with a feed-forward inner dimension of 4096, a self-attention embedding dimension of 768, and 12 attention heads per layer. No positional encoding is used. The input word embedding dimension is 128 for both models. Table 3 shows the corresponding perplexities.

Experimental Results

The final acoustic model trained with the sMBR criterion is evaluated with better language models. LM scales are optimized on the development set. A one-pass recognition setup with MAP Viterbi decoding is applied for both the 4-gram LM and the LSTM LM, where the generated lattices from the LSTM LM-based recognition are used for lattice rescoring with the Transformer LM.

Tabel 4 shows the WER results of these experiments together with a brief summary of best results from the literature. These include hybrid HMM systems as well as end-to-end (E2E) systems using different model types, topologies and label units, such as byte pair encoding (BPE) and SentencePiece . We refer readers to the original papers for more details. As shown in the table, the previous best system has a 7.7% WER on the test set. Our best result is 5.6% on the test set, which achieves 27% relative improvement.

Conclusion

In this work, we presented the integration of data augmentation using SpecAugment into the training pipeline of a state-of-the-art ASR system based on hybrid HMM approach for the TED-LIUM-v2 corpus. SpecAugment provides 7% relative improvement on top of our best SAT model using i-vectors, more precisely from 9.8% to 9.1% WER on the dev set with a small 4-gram LM. We analyzed the effect of different maskings and found out that SpecAugment is beneficial in all cases. The major impact comes from the maximum time and feature mask lengths, which have to be optimized. Then with a good control of maximum ratio of TM and FM, decent improvements are achieved without increasing model size and training time. For feature masking, it is essential to include all features even if i-vectors are fixed for each frame of the segment. Additionally, we found that training from scratch with SpecAugment directly is more efficient than continuing training with SpecAugment to achieve similar performance. Together with subsequent sMBR training and Transformer LM, our best hybrid HMM system achieves the state-of-the-art performance with 5.6% WER on the test set, which improves over the previous best WER of 7.7% by 27% relative.

Acknowledgements

This work has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement No 694537, project “SEQCLAS”) and from a Google Focused Award. The work reflects only the authors’ views and none of the funding parties is responsible for any use that may be made of the information it contains.

We thank Albert Zeyer, Christoph Lüscher, Pavel Golik, Peter Vieting and Tobias Menne for useful discussions.

References