Low-Resource Speech-to-Text Translation

Introduction

Conventional systems for speech-to-text translation typically pipeline automatic speech recognition and machine translation, and since both of these applications require large training sets, these systems are available for only a tiny fraction of the world’s highest-resource languages. But speech-to-text translation could be especially valuable in low-resource scenarios, for example in documentation of unwritten or endangered languages ; or in crisis relief, where emergency workers might need to respond to calls or requests in a foreign language . These applications have motivated recent research on low-resource speech translation trained on a (potentially) cheaper resource: speech paired with its translation, with no intermediate transcriptions.There is also recent work using multitask learning to learn both translation and transcription models, showing improvements on the individual tasks. We focus here on the scenario without transcriptions. Initial work studied speech-to-text alignment without translation , or focused on translating a few keywords using heuristic methods with just a few hours of training data .

Recently, recurrent encoder-decoder models have been used to develop end-to-end speech-to-text translation models, which have been tested in high-resource settings on synthesized speech , audiobooks , and a large dataset of conversational telephone speech . So far, these neural models have been shown to produce high-quality translations with substantial resources—typically more than 100 hours of translated audio, from which models are trained for many days on multiple GPUs. But in our scenarios of interest, we expect to have less data, less time, and fewer computational resources. How will neural models perform in such low-resource settings?

In this paper we perform an extensive investigation of the effects of training end-to-end speech-to-text translation models with limited resources. We implement a model inspired by the state-of-the-art architecture of Weiss et al. , but modify it to permit training in reasonable time on a single GPU (§2). The biggest change, compared to (and also ), is to use word-level decoding instead of character-level. We show that word-level models can be trained much faster than character-level models and still obtain comparable precision and recall; the tradeoff for this speed improvement is that they struggle to translate infrequent word types, leading to a drop in overall accuracy as measured by BLEU and METEOR.

We investigate the model’s performance as we gradually reduce training data from the full 160 hours to as little as 20 hours. With only 50 hours of training data, our model still produces accurate translations for short utterances, with precision and recall around 50%. Although translation quality is much worse with 20 hours of data, precision and recall are still around 30%, which may be useful in low-resource applications. The 50-hour model trains in less than three days on a single GPU.

Speech-to-Text Model

Following Weiss et al. , we combine convolutional neural network (CNN) and recurrent neural network (RNN) components to build an encoder-decoder model with attention, but we modify the system (Table 1) so that we can train even our larger models in 3-5 days on a single GPU.

Raw speech input is converted to Mel filterbank features computed every 10ms. Weiss et al.’s model uses 240-dimensional input speech features, consisting of 80 filterbanks stacked with delta and delta-deltas. We use only 80-dimensional filterbank features. In preliminary experiments, we did not find much difference between 40, 40+deltas and 80 dimensions.

The filterbank features are fed into a stack of two CNN layers with rectified linear unit (ReLU) activations , each with 64 filters (compared to 32 used in Weiss et al.) of shape 3$\times$3 along time and frequency, and a stride of 2$\times$2. Striding reduces the sequence length by a factor of 4, which is important for reducing computation in the subsequent RNN layers.

At training time we use bucketing—80 buckets, with width increments of 25 frames—and padding of speech data. The utterances in the training set vary in length from 2 to 30 seconds; those longer than 20 seconds (80$\times$25 frames) are truncated. We select up to 64 utterances from a bucket to create a mini-batch.

The output of the CNN layers is fed into three stacked bi-directional long short-term memory (LSTM) layers, with 256-dimensional hidden states in each direction. Since the RNN operations are the main bottleneck, for initial hyper-parameter tuning we used uni-directional LSTMs with 300-dimensional hidden states. We then switch to bi-directional LSTM encoders to generate the final results.

2 English decoder

We use a word-level decoder, whereas Weiss et al. used a character-level decoder. Since there are roughly five times as many characters as words, this greatly reduces sequence length, which speeds training for each individual utterance and allows us to use larger mini-batch sizes.

The English words are fed into an embedding layer followed by a stack of three uni-directional LSTM layers. We implement attention using the global attentional model with general score function and input-feeding, as described in . We use beam decoding with a beam size of 8.

3 Optimizer

We use cross-entropy as the loss function. We regularize with dropout , with a ratio of $0.5$ over the embedding and the LSTM layers , and an L2 penalty of $0.0001$. We use a teacher-forcing ratio of 0.8. Our code is implemented in Chainer .Code available at: https://github.com/0xSameer/speech2text/tree/seq2seq Weiss et al.’s model is trained using asynchronous stochastic gradient descent (ASGD) across 10 replicas; we train using a single model copy. Although Weiss et al.’s model benefited greatly from adding Gaussian noise to the weights during training (personal communication), we were unable to replicate this benefit and did not use noise injection.

Experiments

We use the Fisher Spanish speech dataset : a multispeaker corpus of telephone calls in a variety of Spanish dialects recorded in realistic noise conditions. The English translations were collected through crowdsourcing, as described in , and are used to train all models. There are four English references per utterance for the development and test sets, and one per utterance for the training set. We only use one of the two development sets (dev, not dev2). The training set comprises 160 hours of speech, split into 140K utterances; the development and test sets have about 4.5 hours of speech split into 4K utterances each. We lower-case and remove punctuation from the English translations and tokenize the text using NLTK .http://www.nltk.org/api/nltk.tokenize.html#nltk.tokenize.word_tokenize This gives about 17K training word types and 1.5M tokens. There are about 300 out-of-vocabulary (OOV) word types (400 tokens) in the dev set, out of 40K tokens.

We first train a model using the entire 160 hours of labeled training data. To understand the impact of training data size on translation quality, we further train models using smaller subsets: 80, 50, 25, 20 hours of data, selected at random, from the entire training data. We use the same set of hyper-parameters—tuned for 160hrs—for all models. With these model parameters and training setup, we are able to train an epoch—a complete pass through the entire 160 hours of training data—in about $2$ hours on a single Titan X (or equivalent) GPU.

2 Evaluation

In order to understand different aspects of model behavior, we evaluate with several metrics: BLEU , METEOR , and unigram precision/recall on the Fisher dev set, using all 4 human references.BLEU and precision are computed using multi-bleu.pl from the MOSES toolkit , which computes 4-gram BLEU. METEOR score and recall are computed using the script from http://www.cs.cmu.edu/~alavie/METEOR/. BLEU measures how well a predicted translation matches a set of references based on a modified $n$-gram precision; it does not compute recall, and instead uses a brevity penalty to account for mismatch in reference and predicted lengths. METEOR computes both precision and recall and combines them via a harmonic mean, with greater weight given to recall. The final score is computed using a set of parameters, tuned for individual languages to correlate well with human judgments.

Whereas BLEU looks only for exact token matches between a prediction and set of references, METEOR also takes into account stem, synonym, and paraphrase matches. These four categories are weighted (by default) 1.0, 0.6, 0.8 and 0.6, respectively. For example, with these weights, a prediction of “eat” will score a recall of 1 against reference “eat” and 0.8 against reference “feed”, a synonym match. METEOR can therefore be considered a more semantic measure. In low-resource settings, inexact translations that capture the semantics of an utterance are still useful. We use default settings and configuration files provided by the METEOR script for English. For comparison, we also provide human-level BLEU and METEOR scores by comparing one reference against the remaining 3.

In low-resource settings, BLEU scores may be very low and therefore difficult to interpret, but a model might still be able to predict (potentially important) keywords, which could be useful for cross-lingual applications. So, we also report word-level unigram precision using the BLEU script: if a predicted token is present in any of the 4 reference translations, it is considered a True Positive; otherwise it is a False Positive. Unigram recall is computed using the METEOR script, which includes stem, synonym, and paraphrase level matches. We also compute recall for exact matches only, by setting the METEOR weights to 1.0, 0.0, 0.0 and 0.0.

3 Results and discussion

Figures 1 and 2 show the BLEU, METEOR, and precision/recall scores on the dev set for each model as we change the amount of training data. Table 2 shows the BLEU scores on the test set.

Word vs. character models. Weiss et al.’s character-level model achieves close to human performance.The human scores are computed using 3 references; BLEU scores for all models are 2-4 points lower when using 3 instead of 4 references. Our word-level model, trained on 160 hours of data, converges to a BLEU score of 29.5 in about five days, a much lower score than Weiss et al.’s. One reason for this discrepancy is our different architecture and training setup (§2 and Table 1), which allowed us to train our models on our available computing resources.Since our paper was submitted, reported results on 20 hours of Spanish-English data for several multitask (translation/transcription) models and a baseline speech-to-text model. They used a character-level decoder and a different corpus (CALLHOME). They did not report detailed word-level BLEU scores, but said they were “between 7 and 10” for all models.

Training the character-level model takes nearly twice as long per epoch (4 hours for 160hrs of data) as the word-level model (2 hours). Our character-level model also has much smaller performance gains per epoch. To speed up character-level model training, we truncate utterances longer than 15 seconds (20 seconds for word models), reducing training time to 3 hours per epoch. Figure 3 compares character-level models trained in this way to word-level models for two training set sizes.

One of the main benefits of character models is their ability to gracefully handle OOV or infrequent words. On the dev set, the Weiss et al. model predicts about 130 word types that were not seen in training, which helps the model recall 7 OOV tokens out of 400. This is too small an effect to account for the performance differences, so we also analyze performance for a range of word frequencies. Table 3 shows the precision/recall for the 160hrs model and Weiss et al.’s model, for words of different frequencies. We only consider content word types—words that are more than five characters long and are not in the NLTK stopword lists. The word-level model recall drops rapidly for medium frequency words, and for rare word types it has almost 0% recall. From this, we see that the main benefit of the character-level model is in handling of rare words, rather than previously unseen words.

Training considerations. Regularization parameters are critical to model performance. Figure 4 shows that increasing the L2 weight decay to a rate of $10^{-4}$ from $10^{-6}$ improved BLEU by about 2 points. Even though we use a high L2 penalty and dropout ratio, the models can overfit; the training loss continues to decrease, and we use early stopping based on dev set BLEU.

We also tried using batch normalization at each CNN layer, and layer normalization at each LSTM layer; but these did not have any noticeable impact on training performance.

Other design considerations. Using a bi-directional LSTM encoder does not seem to have an effect for the largest training sets, but improves BLEU by about one point in the $\leq$50 hour cases (Figure 4). This comes at a training time cost: bi-directional encoder models are almost $50\%$ slower to train per epoch.

Figure 5 shows the improvement in BLEU scores by using beam decoding, over greedy decoding. Beam decoding always helps, but has a larger effect with more training data.

Exact vs. semantic matches. Figure 1 shows that the gap in METEOR scores between our models and Weiss et al.’s is much smaller than the gap in BLEU, and METEOR degrades more slowly as training data is reduced. This suggests that although our models are much worse at predicting the exact words in the reference translations, they often predict near-exact matches.

Table 4 shows some example predictions. As expected from the BLEU/METEOR results, the translations trained on more than 50 hours are fairly good, though they may contain different forms of the content words than are in the reference (e.g. bothering/bothers vs. bothered). The models trained on less data are clearly worse: they usually get some words right, which could be useful for keyword spotting or topic modeling in low-resource settings, but in some cases (as in the first example) the correctly predicted words do not carry much of the meaning.

Neural models in the extreme low-data setting. One may wonder whether, for very low-data settings, neural models still outperform older non-neural models at all. While we did not directly compare to a non-neural model, one indication is that, in the lowest-data settings, BLEU and METEOR are much worse but unigram precision and recall are still in the 25-35% range. These results compare very favorably to the 2-10% precision/recall reported by , who used a heuristic speech-to-text translation system trained on the CALLHOME corpus (also about 20 hours of Spanish conversational telephone speech with English text translations). So, it appears that even in a very low-resource setting with a model that is not state-of-the-art, the neural approach significantly outperforms previous non-neural models.

Conclusion

We performed a thorough analysis of a neural end-to-end speech-to-text translation model, with a specific focus on how performance is affected when using limited computational resources and limited amounts of data. We also showed the effects of a number of architectural design decisions using several performance metrics. While word-level models fall behind previously proposed character-level models when trained on around 160 hours of translated speech, our word-level models can be trained much faster and give reasonable performance on smaller training sets. Although translation quality drops, models trained on only 20 hours of translated speech achieve precision and recall of around 30% for content words. This could still be useful in search applications in severely low-resource scenarios. We believe that our extensive analyses in this work will contribute to better decision-making for architectural choices in computation- and data-limited settings.

In future work we aim to consider sub-word modelling , which could balance the trade-off between training costs and translation performance. In addition, we plan to try speech features that are targeted to low-resource multi-speaker settings and speaker normalization .

Acknowledgements

We thank Ron Weiss and Jan Chorowski for sharing their translation output, and Kenneth Heafield for giving access to GPUs. This work was supported in part by a James S. McDonnell Foundation Scholar Award and a Google faculty research award.

References