Unsupervised Speech Recognition via Segmental Empirical Output Distribution Matching

Introduction

Over the past years, the performance of automatic speech recognition (ASR) has been improved greatly and the recognition accuracy in certain scenarios could be on par with human performance (Xiong et al., 2016). Most of the state-of-the-art ASR systems are constructed by training deep neural networks on large-scale labeled data using supervised learning (Hinton et al., 2012; Dahl et al., 2012; Xiong et al., 2016; Graves et al., 2013; 2006; Graves, 2012); they rely on a large number of human labeled data to train the recognition model. In this paper, we are working towards the grand mission of training speech recognition models without any human annotated data. Such an approach could potentially save a huge amount of human labeling costs for developing ASR systems by leveraging massive unlabeled speech data. It is especially valuable when developing ASR systems for low-resource languages, where labeled data are more expensive to obtain.

Specifically, we consider the phoneme recognition problem, for which we learn a sequential classifier that maps speech waveform into a sequence of phonemes. In our unsupervised learning setting, the learning algorithm can only access (i) the input speech acoustic features, and (ii) a pretrained phoneme language model (LM). There is no human supervision presented to the algorithm at any level; that is, we do not provide any (frame-level) label for input samples, nor do we provide any (sentence-level) transcription for input utterances. The language model could be trained from a separate (text) corpus in an unsupervised manner with the help of a pre-defined lexiconA lexicon in ASR is a pre-defined dictionary that maps word sequences into phoneme sequences..

There have been some recent successes in developing fully unsupervised method for neural machine translation (Artetxe et al., 2018; Lample et al., 2018) and sequence classifications (Liu et al., 2017). However, different from these problems, speech recognition problem has segmental structures that impose unique challenges for developing unsupervised learning algorithms. First, each phoneme generally consists of a segment of consecutive input samples (frames) that are associated to the same phoneme label. Second, the lengths and the boundaries of these segments are usually unknown a priori. For this reason, we could not directly apply the previous techniques to develop unsupervised ASR algorithms. To address the first challenge, we develop a novel unsupervised learning cost function for ASR systems by extending the Empirical Output Distribution Matching (Empirical-ODM) cost in (Liu et al., 2017) to segmental structures. The key ideas of our Segmental Empirical-ODM are: (i) the distribution of the predicted outputs across consecutive segments shall match the phoneme language model and (ii) the predicted outputs within each segment should be equal to each other as they belong to the same phoneme. This cost function allows us to learn the classifier without labeled data for a given set of phoneme segmentation boundaries. To address the second challenge, we develop a novel unsupervised approach to estimate (and refine) the segmentation boundaries using the current classification model. Our algorithm alternates between these two steps of learning classifier and estimating the boundaries to successively improve the performance of each other. Therefore, unlike previous works in (Liu et al., 2018), which relies on an oracle or forced alignment methods to obtain the phoneme segmentation boundaries, our method is fully unsupervised in both segmentation and classification. Furthermore, we also adapt the semi-supervised HMM learning technique (Zavaliagkos et al., 1998; Kemp and Waibel, 1999; Nallasamy et al., 2012) to our unsupervised setting to further improve the performance. In our experiments on TIMIT phoneme recognition task, our unsupervised learning method achieves a promising phone error rate (PER) of $41.6\%$. To our best knowledge, this is the first empirical success of a fully unsupervised speech recognition that does not use any oracle segmentation or labels. Furthermore, when the oracle phoneme segmentation boundaries are given (similar to the setting in Liu et al. (2018)), our method achieves a PER of $32.5\%$ with matching language model, which approaches supervised learning with the same model architecture, demonstrating a great potential of our method.

Fully Unsupervised Speech Recognition

2 Unsupervised frame classification with given segmentation boundaries

Next, we define the cost function that characterizes the intra-segment distribution matching as:

Note that our method directly optimizes the classifier $p_{\theta}(y_{t}|x_{t})$ that takes the raw acoustic feature vector $x_{t}$ (e.g., MFCC features) and maps it into output space. This is different from the previous work (Liu et al., 2018), which first performs clustering in the speech space and then maps the clusters into output space using adversarial training. This makes its performance upper-bounded by the purity of the initial clusters since input frames of different phonemes may be mapped into the same cluster. In contrast, our algorithm is end-to-end trained without using a separate clustering algorithm. This enables us to outperform the cluster purity upper bound, as shown in our experiment section.

3 Segmentation boundary refinement using the classification model

Note that $p(y_{t}|y_{1},\ldots,y_{t-1})$ is the transition probability of the frame labels. Assuming that $y_{t}$ belongs to the $i$-th segment, we can express $p(y_{t}|y_{1},\ldots,y_{t-1})$ as:

4 Alternating Training Algorithm

Our unsupervised learning algorithm for $p_{\theta}(y_{t}|x_{t})$ alternates between the above two steps of estimating $\theta$ for a given $b$ and refining $b$ for a given $\theta$. The overall algorithm is summarized in Algorithm 1. We initialize the algorithm by thresholding the temporal update gate activation in Wang et al. (2017) to obtain an initial rough estimate for $b$. After the training converges,We observe in our experiments that two iterations are sufficient to converge. we could apply the unsupervised HMM training technique discussed in Section 3 to further boost the performance. Note that although the training process requires boundary estimation, at testing stage, it is not necessary because the learned $p_{\theta}(y_{t}|x_{t})$ could be used in standard speech decoders just as supervised models.

5 Unsupervised Model Selection

Since there are no labeled data during the training process, we need to develop an unsupervised self-validation metric to perform model selection. We propose to use the value of the loss function (1) on a heldout validation set (including only input features) to perform model selection. This self-validation loss gives us an estimate of which model configuration is better, and is used to a) determine when to stop training, and b) to select the best hyper-parameters. To validate the effectiveness for our self-validation loss, we show the learning curves of this self-validation loss and the validation frame error rate in Figure 2. We observe that the self-validation loss aligns well with the true validation error. Furthermore, in Figure 2 we plot the self-validation loss and the validation FER for different values of $\lambda$, which shows that the two metrics are highly correlated. The results demonstrate that the self-validation loss can be effectively used to select a good model.

Unsupervised HMM Training

To further improve the performance of proposed unsupervised speech recognition system, we explore the semi-supervised hidden Markov model (HMM) training strategy (Zavaliagkos et al., 1998; Kemp and Waibel, 1999) that has commonly been used in speech recognition. The semi-supervised HMM training is an effective technique where a seed model trained on a relatively small amount of labeled speech data is used for providing labels for larger amount of non-transcribed speech data for iterative model refinement. A major difference of the HMM training strategy used in this work compared to the ones used in semi-supervised learning is that we use the transcripts generated from proposed unsupervised speech recognition system (i.e., predicted labels for 3969 TIMIT training utterances) for bootstrapping the training of HMM-based models. Therefore, the training of HMM models in this work does not require any manually provided supervised information. The training of HMM based speech recognition models follows the standard recipes in Kaldi speech recognition toolkit (Povey et al., 2011). We experimented with monophone and triphone models with MFCC feature as input as well as more advanced speaker adaptive training (SAT) (Matsoukas et al., 1997) approach with feature space maximum likelihood linear regression (fMLLR) (Gales et al., ) as input.

Experiments

We perform experiments on the TIMIT dataset where 6300 prompted English speech sentences are recorded. The preparation of training and test sets follow the standard protocol of the TIMIT dataset. The phoneme transcription of these utterances are manually segmented and labelled with a lexicon of 61 distinct phoneme classes. These phoneme labels are mapped to 39 phoneme classes for scoring phone error rate (Lee and Hon, 1989). We use 39 dimensional feature vectors including 13 mel-frequency cepstral coefficients (MFCC) plus its acceleration features that are extracted with 25 ms Hamming window at 10 ms interval. The classifier $p_{\theta}(y_{t}|x_{t})$ is modeled by a fully connected neural network with one hidden layer of $512$ ReLU units. The input to the neural network is a concatenation of frames within a context window of size $11$. We follow the default hyper-parameters in Wang et al. (2017) to estimate the phoneme boundaries, which are used to initialize our algorithm. The optimization of (3) is performed with momentum SGD with a fixed schedule of increasing batch size from 5000 to 20000. $\lambda$ in (3) is chosen to be $10^{-5}$. We use both frame error rate (FER) and phone error rate (PER) as our evaluation metrics. Details of the experiment setting and other hyper-parameters can be found in Appendix B.

2 Baseline Methods

The first baseline we consider is the work by Liu et al. (2018), which learns an unsupervised embedding by a sequence-to-sequence autoencoder followed by k-means clustering. Each cluster is then mapped to a phoneme by adversarial training between the cluster sequences and the phoneme sequence. The phoneme boundaries are given by a supervised oracle.

Cluster Purity

The accuracy of Adversarial Mapping (Liu et al., 2018) is upper-bounded by the cluster purity, which is the frame accuracy when assigning all the frames in each cluster to its most frequent phonemes. It is a supervised baseline since it relies on the phoneme labels. We show the cluster purity for 1000 clusters, which is the largest number of clusters used by Liu et al. (2018).

Supervised Neural Network

We train a supervised neural network with the same architecture as our unsupervised model with standard cross-entropy loss.

Supervised RNN Transducer

It is one of the state-of-the-art methods, which learns a BiLSTM-RNN Transducer with supervised learning (Graves et al., 2013).

3 Experiment Results

Fully unsupervised speech recognition

We now consider the fully unsupervised setting where only input speech features and a language model is given. The phoneme boundaries are not given and has to be estimated in an unsupervised manner using our Algorithm 1. We show the quality of the learned model after each iteration of the learning process in Table 2. And we observe that our iteration process improves the results by a great margin especially in the non-matching LM case, significantly lowering the FER and PER by over $10\%$. This demonstrates that our boundary refining process has resulted in a better set of boundaries, which greatly improves the output distribution matching. Moreover, we also report the results of using unsupervised HMM training where the PER can be further improved. In the matching LM setting, HMM training with monophone, triphone, and speaker adaptation training (SAT) improves the PER by a similar amount. In the non-matching LM setting, HMM training significantly improves the PER, and SAT additionally improves $3\%$ in PER. Overall, our hybrid system with matching and non-matching LM achieved $36.5\%$ and $41.6\%$ PER, respectively, which is only $10\%$ below the supervised system of the same architecture.

Unsupervised Phoneme Segmentation

To understand how much our proposed boundary refinement method in Section 2.3 improves the segmentation quality, we follow the setting in previous works and report in Table 3 the recall, precision, F-score, and R-value with a 20-ms tolerance window on TIMIT’s training set (Scharenborg et al., 2010; Versteegh et al., 2016; Rasanen, 2014). We compare our results (obtained with matching LM) with several unsupervised phoneme segmentation methods (Dusan and Rabiner, 2006; Qiao et al., 2008; Lee and Glass, 2012; Rasanen, 2014; Hoang and Wang, 2015; Michel et al., 2016; Wang et al., 2017). Note that our refined segmentation significantly improves over the initial boundaries generated by Wang et al. (2017) and also outperforms other baselines. This result also confirms its contribution to the much improved phoneme recognition performance in the 2nd iteration (see “Our Model: 2nd iteration” in Table 2) to the 1st iteration. However, we emphasize that our method is designed towards unsupervised speech recognition rather than unsupervised phoneme segmentation. Estimating the segmentation boundary only serves as an auxiliary task to enable the unsupervised learning of the recognition model. And in the testing stage, there is no need to estimate the segmentation boundaries. Instead, our trained model could be directly used with a speech decoder just as any supervised recognition model would do.

Further analysis

We include some further experiments and analysis in Appendix C, where we show the importance of the frame smoothness term in (3). We also compare the performance of our unsupervised algorithm to supervised learning with different amounts of labeled training data.

Related Work

Recently, unsupervised sequence-to-sequence learning has achieved great success in several problems. Liu et al. (2017) showed that it is possible to learn a sequence classifier without any labeled data by exploiting the output sequential structure using an unsupervised cost function named Empirical-ODM. Artetxe et al. (2018) and Lample et al. (2018) showed that unsupervised neural machine translation (uNMT) systems can be achieved by utilizing cross-lingual alignments and an adversarial structure without any form of parallel information. The success in the unsupervised sequence-to-sequence learning in various applications shed light on building our fully unsupervised speech recognition system. In particular, our work extends the Empirical-ODM in Liu et al. (2017) to problem with segmental structures.

Unsupervised speech segmentation

One line of unsupervised segmentation methods designs robust acoustic features that are likely to remain stable within a phoneme, and capture the change of features for phoneme boundaries (Esposito and Aversano, 2005; Hoang and Wang, 2015; Khanagha et al., 2014; Rasanen et al., 2011; Michel et al., 2016; Wang et al., 2017). Another line of research uses a simpler segmentation method as an initialization, and jointly trains the segmenting and acoustic models for phonemes or words (Kamper et al., 2015; Glass, 2003; Siu et al., 2014; Lee and Glass, 2012). Qiao et al. (2008) use dynamic programming methods in order to the derive optimal segmentation, but requires the number of segments and is not fully unsupervised. In Wang et al. (2017), the authors use the update gate of a GRNN autoencoder to discover the phoneme boundaries.

Unsupervised spoken term discovery

Recently, the discovery of acoustic tokens including subword and word units has become a popular research topic (Dunbar et al. (2017); Versteegh et al. (2016); Burget et al. ). The term “Spoken term discovery” includes lexicon discovery, word segmentation, and subword matching (Dunbar et al. (2017)). The standard approaches segment audio signals that are acoustically similar, and cluster the obtained segmented signals (Lee and Glass, 2012; Glass, 2012; Park and Glass, 2008; Driesen et al., 2012). Walter et al. (2013) uses the discovered unit index sequence as the transcription for the acoustic model training, similar to the HMM training in section 3. Kamper et al. (2017) iterates between the clustering and segmentation steps. Ondel et al. (2016) improves upon previous methods by replacing Gibbs sampling by variational inference, and Ondel et al. (2017) further improves the result by including a bigram language model. The effectiveness of these approaches has been demonstrated on query-by-example spoken term detection or by calculating the normalized mutual information between the self-discovered units and the actual labels. Overall, these methods differ from our method in that they segment and cluster the raw speech signals to self-discovered units, but does not recognize them into phoneme or word labels directly. More recently, Chung et al. (2018) show that unsupervised spoken word classification is possible by using adversarial cross-modal alignments similar to that in uNMT systems.

Unsupervised speech recognition with oracle segmentation

There have been several attempts (Liu et al., 2018; Chen et al., 2018) on building an unsupervised speech recognition model inspired by the success of the uNMT. These methods first learn an embedding from the acoustic data, and then map the clustered embeddings to the output space by either adversarial training or iterative mapping. In contrast, our approach learns a neural network model that directly maps the raw acoustic features into the output space by optimizing the Segmental Empirical-ODM cost, and outperforms the upper bound of the above cluster-based approaches. Furthermore, all methods in Liu et al. (2018); Chen et al. (2018) assume that the phoneme boundaries are given by a supervised oracle. In contrast, our method iteratively estimates the boundaries without any labeled data, making it fully unsupervised.

Conclusion

We have developed a fully unsupervised learning algorithm for phoneme recognition. The algorithm alternates between two steps: (i) learn a phoneme classifier for a given set of phoneme segmentation boundaries, and (ii) refining the phoneme boundaries based on a given classifier. For the first step, we developed a novel unsupervised cost function named Segmental Empirical-ODM by generalizing the work (Liu et al., 2017) to segmental structures. For the second step, we developed an approximate MAP approach to refining the boundaries obtained from Wang et al. (2017). Our experimental results on TIMIT phoneme recognition task demonstrate the success of a fully unsupervised phoneme recognition system. Although the fully unsupervised system is still far away from the state-of-the-art supervised methods (e.g., supervised RNN transducer), we show that with oracle boundaries the performance of our algorithm could approach that of the supervised system with the same model architecture. This demonstrates the potential of our method if, in future work, we can further improve the accuracy of boundary estimation. We want to further point out that the techniques we proposed in this paper, although was evaluated in speech recognition, can be exploited to attack other similar sequence recognition problems where the source and destination sequences have different lengths and labels are not available or hard to get.

References

Supplementary Material

Appendix A Derivation of the MAP estimate for the segmentation boundaries

In this appendix, we derive the MAP estimate for $y=(y_{1},\ldots,y_{T})$ given an input utterance sequence $x=(y_{1},\ldots,y_{T})$. Specifically, we have

where in step (a) we approximate the $p(x|y)$ by its factored form and in step (b) we dropped the constant term that is independent of $y$.

Appendix B Detailed Experiment setting

We perform experiments on the TIMIT dataset where 6300 prompted English speech utterances are recorded. The phoneme transcription of these utterances are manually segmented and labelled with a lexicon of 61 distinct phoneme classes, where we compact the 61 phoneme classes into 48 phone classes and train the language model with the validation dataset, which is later used to train our main algorithm. These 48 phoneme classes are mapped to 39 phoneme classes for scoring phone error rate [Lee and Hon, 1989].

The 39 dimensional feature vectors including 13 mel-frequency cepstral coefficients (MFCC) plus its acceleration features that are extracted with 25 ms Hamming window at 10 ms interval. The classifier $P_{\theta}$ is modeled by a fully connected neural network with one hidden layer of $512$ ReLU units. The input to the neural network is a concatenation of frames within a context window of size $11$, and we repeat the starting or ending frames if the window has reached the start or end of the sentence.

The optimization of (3) is performed with momentum SGD with momentum 0.9 and learning rate of $10^{-3}$ with learning rate decay with a fixed schedule of increasing batchsize from 5000 to 20000 and decreasing temperature for softmax. The scheduler parameter is listed below: first 200 epochs with batchsize 500 and temperature 1.0, next 300 epochs with batchsize 5000 and temperature 0.9, followed by 300 epochs with batchsize 10000 and temperature 0.8, finally 300 epochs with batchsize 20000 and temperature 0.7. Whenever the batchsize is increased, we set the learning rate to the inital learning rate value. The scheduling procedure is determined by self-validation, and is not extensively tuned during the experiments.

In our experiments, we chose $N=5$ for the N-gram in (1), and for computational issues we only consider the most frequent 10000 5-grams. We do not observe any noticeable performance drop by considering only the 10000 5-grams. Among the 5-gram language model $P_{\text{LM}}$, 69553 5-grams (out of $48^{5}$ possible 5-grams) are non-zero, and the top 10000 5-grams account for almost half of the probability. To sample $\tau$, we use a standard truncated normal distribution for sampling the frame in every segment, with some necessary scaling and rounding. The truncated distribution is to ensure that our sampling will give us bounded frames that lie in the correct segment. This distribution can also be replaced by the uniform distribution.

We randomly sample 10000 continuous frames to optimize (2), which is sampled every batch. $\lambda$ in (3) is chosen to be $10^{-5}$. We use both frame error rate (FER) and phone error rate (PER) as our evaluation metrics. All phone error rate (PER) results reported has been obtained by a Kaldi decoder by considering the per-frame softmax value and the language model, and the weight between the two set to 1, which is fixed in all unsupervised setting.

Appendix C Additional experiments and analysis

First, we examine the importance of the frame smoothness term in (3) in the fully unsupervised setting. In Figure 3, we show the FER of our model after the first iteration of Algorithm 1 for different values of $\lambda$. Note that when $\lambda$ is close to the order of $10^{-5}$, the result does not differ a lot from the best result. However, when $\lambda$ is set to zero, the performance degrades significantly. This confirms the importance of incorporating the temporal structure of speech data into the cost function, as discussed in Section 2.2. Second, we would like to study another important question regarding our unsupervised learning method: how much labeled data is it equivalent to? In Figure 3, we show the supervised neural network with different sizes of training data, where x-axis is the percentage of the original labeled set being used to train the model. We observe that with oracle boundary and matching LM, our algorithm is equivalent to supervised learning with $30\%$ labeled data. With unsupervised boundary estimation, we still see a big performance loss. Therefore, it is critical to improve the boundary estimation performance in our future work.