Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation

Ye Jia, Yifan Ding, Ankur Bapna, Colin Cherry, Yu Zhang, Alexis Conneau, Nobuyuki Morioka

Introduction

Speech-to-speech translation (S2ST) aims to break the communication barriers between people speaking different languages. Conventionally, S2ST systems are built with a cascade of automatic speech recognition (ASR), text-to-text machine translation (MT), and text-to-speech synthesis (TTS) sub-systems, all of which rely on intermediate text representations. Recently, there has been progress towards developing S2ST without relying on intermediate text representations, such as end-to-end direct S2ST and cascaded S2ST based on discrete speech representations . These works have demonstrated several advantages over conventional cascade S2ST systems, such as preserving paralinguistic and non-linguistic speech information during translation (e.g. speakers’ voices) , supporting languages without written form , avoiding error compounding across sub-systems and better handling of content that does not need to be translated (e.g. names and proper nouns) .

However, training direct S2ST models without relying on intermediate text typically requires a large amount of parallel S2ST training data, which is scarce and significantly more expensive to obtain than acquiring data for training each individual component in a conventional cascade S2ST system. While some of these challenges are shared with research on direct speech-to-text translation (ST), there are several challenges unique to S2ST. To train a S2ST model that generates translated speech with high naturalness and cleanness, it is usually desirable to collect clean target speech with high acoustic quality, which can significantly drive up the cost of data collection. Although it has been demonstrated in that the performance of direct S2ST is approaching cascade S2ST with a very small gap, when the two are trained on comparable datasets, the lack of large scale high-quality training data is still a primary limiting factor for direct S2ST models in practice.

In this paper, we first propose a small change to the Translatotron 2 S2ST model , which significantly improves its translation quality on the CVSS-C corpus by +1.4 BLEU (or +16% relatively) average on 21 language pairs. Secondly, we conduct empirical studies exploring multiple approaches to leverage more widely available unsupervised data (e.g. unlabeled speech and text) and weakly-supervised data (e.g. labeled ASR and MT data) to improve the translation quality of Translatotron 2. With the most effective approaches applied, we further improve the translation quality on the CVSS-C corpus by +15.5 BLEU (or +153% relatively). Our best result outperforms the prior state-of-the-art, which was trained only on CVSS-C, by +13.6 BLEU (or +113% relatively) on average on 21 language pairs, and +398% relatively on average on 12 low-resource language pairs. Our comparative studies suggest future research directions on S2ST and speech representation learning.

Related works

In the past few years, self-supervised speech representation learning has been shown very successful on ASR and ST tasks. Recently, it has been expanded to also learn from text, with the emergence of semi-supervised speech-text joint representation learning . The success of these self-supervised methods has been magnified by scaling up the size of the training data and the models . In fact, the progress on such representation learning on massive amounts of unlabeled data has been the biggest driver for performance improvements in direct ST on a few benchmarks, e.g. CoVoST 2 .

Before the emergence of self-supervised speech representation learning, direct ST often leveraged pre-training on other supervised tasks, e.g. ASR and MT, in order to benefit from additional weakly-supervised data . Other approaches for utilizing weakly-supervised data in direct ST include multi-task learning , knowledge distillation , and TTS-based data augmentation using a back-translation style approach .

Although most of the above approaches can be expanded to apply to direct S2ST, there has been little work on examining their effectiveness. explored pre-training part of the direct S2ST model on weakly-supervised tasks, e.g. ASR and ST, which brought the performance of direct S2ST close to cascade S2ST trained on comparable datasets. built cascade S2ST based on discrete speech representations learned from a large unlabeled speech dataset, but its performance was not better than recent direct S2ST models trained from scratch, e.g. . demonstrated that it is possible to mine S2ST data from multilingual untranscribed speech corpora and improve the performance of the trained S2ST models.

Methods

Translatotron 2 is a direct S2ST model that can be trained end-to-end. It consists of a speech encoder, a linguistic decoder, an acoustic synthesizer, and a single attention module that connects them together. The use of a single attention module provides temporally synchronized linguistic information (in the translated language) and acoustic information (from the source speech) to the synthesizer. As a result, the model is capable of preserving fine granularity non-linguistic and paralinguistic information during translation.

The original Translatotron 2 uses a Conformer encoder, an LSTM decoder, and a duration-based LSTM synthesizer. In this work, we modify it to use an autoregressive Transformer decoder , which obtained better translation quality in our experiments (Sec. 4.1).

2 Pre-training

We explore multiple pre-training strategies in order to utilize large amounts of unsupervised or semi-supervised speech and/or text data, as well as weakly-supervised MT data for improving the performance of direct S2ST.

To utilize large amounts of unsupervised speech data, we use speech representations learned from a w2v-BERT model.

w2v-BERT is a self-supervised learning model that combines contrastive learning and masked language model (MLM). It consists of three components which are trained end-to-end: a feature encoder that encodes input speech signals into a continuous latent representation; a contrastive network that discretizes continuous latent representations into discrete speech tokens, optimized by a contrastive objective; and a context network that learns contextualized speech representations on the discretized speech tokens by optimizing an MLM objective.

We pre-train a w2v-BERT model on approximately 429k hours of unlabeled speech data in 51 languages. This dataset is originated from a combination of VoxPopuli , Common Voice , MLS and Babel , as described in . The pre-trained w2v-BERT model is used as the encoder of Translatotron 2 and is fine-tuned on its S2ST task.

2.2 Semi-supervised speech-text joint representation learning

In addition to unsupervised speech data, we also explore using a large amount of unsupervised text data and a small amount of paired speech-transcript data in a semi-supervised manner. To this end, we use speech-text joint representations learned from an mSLAM model. One advantage of this speech-text joint representation is the ability to fine-tune these models in a multi-task setup (Sec. 3.3), to make use of text-based weakly-supervised data, such as MT data.

mSLAM combines w2v-BERT and SpanBERT into a single model to support speech-text joint representation learning. It is trained in a multi-task learning setup that can learn from unlabeled speech, unlabeled text, and paired speech-transcript data. The training on unlabeled speech is identical to w2v-BERT. For unlabeled text, it prepends a text embedding layer to w2v-BERT’s context network to form a text encoder that is trained with a SpanBERT objective. To encourage alignment between the learned speech and text representations, it adds a Connectionist Temporal Classification (CTC) objective and a Translation Language Model (TLM) objective , which are trained on a small amount of speech-transcript paired data.

We pre-train a mSLAM model on the same unlabeled speech data as described in Sec. 3.2.1, plus the mC4 unlabeled text corpus containing 6T tokens in 101 languages, and 2.4k hours of paired speech-transcript data spanning 32 languages, as described in . The pre-trained mSLAM model is used as the speech encoder in Translatotron 2 and the text encoder in MT models, and is fine-tuned on the corresponding tasks.

2.3 Supervised sequence-to-sequence pre-training

S2ST and MT share a lot of similarity in terms of translating linguistic concepts from one language to another. To utilize the more widely available MT data in the S2ST task, we explore pre-training the linguistic decoder of Translatotron 2 with MT tasks on a large MT dataset. Because the linguistic decoder of Translatotron 2 predicts phonemes instead of text tokens, and its input is encoded speech representations instead of text, we explore multiple variants of MT tasks that predict phonemes as output and/or take phonemes as inputs.

We pre-trained Transformer-based MT models on a large MT dataset including 21 X $\to$ English language pairs. This dataset is originated from WMT , TED and CoVoST 2 , as described in . The decoder of the pre-trained MT model is used for initializing the linguistic decoder of Translatotron 2.

3 Multimodal multi-task fine-tuning

While pre-training the decoder in an MT task can potentially transfer the knowledge learned from the MT data to the S2ST model, such transferred knowledge may be susceptible to catastrophic forgetting during the course of the fine-tuning . To mitigate this potential degradation, we explore fine-tuning the Translatotron 2 model in a S2ST and MT multi-modal multi-task learning setup, with the decoder shared by the two tasks. For consistency between the two tasks, we modify the MT task to predict phonemes in the target language instead of text tokens. When the pre-trained speech-text joint encoder (Sec. 3.2.2) is used, the context network in the encoder can be shared and fine-tuned on both tasks as well. Figure 1 illustrates the S2ST and MT multimodal multi-task fine-tuning with a shared pre-trained encoder and a shared decoder, in which unsupervised speech and text data, speech-transcript pair data, and weakly-supervised MT data are all utilized.

4 TTS-based data augmentation

As an alternative to utilizing weakly-supervised MT data, we use a multilingual multi-speaker TTS model to augment the MT data into synthetic S2ST data, and train the Translatotron 2 model in a mixture of the original S2ST dataset and the augmented dataset. Compared to the multimodal multi-task fine-tuning approach, this method has the advantage of training the model entirely in the matched modality and thus potentially better utilizing the weakly-supervised data. On the other hand, one possible shortcoming is that it may bias the trained model towards synthetic speech input but not generalize well to human speech. As suggests, such concerns can be addressed by jointly fine-tuning on a mixture of real and synthetic data.

Experiments

To compare the effectiveness of the various approaches described in Sec. 3, we conducted comparative experiments on the CVSS-C corpus . CVSS-C is derived from the CoVoST 2 and Common Voice corpora, containing sentence-level paired S2ST data in 21 X $\to$ English language pairs. The source speech in the corpus is 1,153 hours of human read speech collected via crowdsourcing; the target speech in the corpus is 719 hours of high-quality TTS synthetic speech, with naturalness on-par with human recordings. The target speech is shorter than the source speech because of better fluency in the TTS synthetic speech.

We used the modified Translatotron 2 (Sec. 3.1) as the direct S2ST model, implemented in the Lingvo framework . Unless specified otherwise, all the models followed the hyper-parameters from except for the decoder, which used a 512 $\times$ 6 Transformer following hyper-parameters from . The decoder used a higher dropout probability 0.3 when trained or fine-tuned on CVSS-C only; otherwise the default probability 0.1 was used.

Following , the translation quality of S2ST is evaluated by BLEU on ASR transcription from the translation speech (in lowercase, excluding punctuation marks). We used the same ASR model from for evaluation, following , therefore the results are comparable to . The results are grouped into high/mid/low-resource language groups based on the amount of data available in the CVSS-C corpus, following .

As can be seen from the first group in Table 1, replacing the LSTM-based linguistic decoder in the original Translatotron 2 with a Transformer-based decoder improved translation quality by +1.4 BLEU (or +16% relatively) on average on all 21 language pairs, which is consistent with the general observations on the superior performance of Transformer in MT and ST tasks.

2 Encoder pre-training

We compared a w2v-BERT encoder pre-trained on unsupervised speech data and an mSLAM encoder pre-trained on semi-supervised speech and text data, as described in Sec. 3.2. Both encoders used a 1024 $\times$ 24 Conformer stack, with 0.6B parameters. The results are shown in the second group in Table 1. Both pre-trained encoders brought large boosts to the translation quality from the direct S2ST model (+7.8 BLEU, or +77% relatively). Such improvement is shown across the board on low, mid and high-resource languages. These results suggest the effectiveness of pre-training on a massive amount of unlabeled speech data.

Consistent with ST experimental results from , additional pre-training on a large amount of text data and a small amount of paired speech-transcript data did not further improve the performance of direct S2ST when it was directly used. However, as shown in Sec. 4.4, the speech-text joint representations from such pre-training is beneficial in multi-task fine-tuning.

3 Decoder pre-training

Compared to encoder pre-training in self-supervised tasks, decoder pre-training in supervised MT-derived tasks was less effective. Experiments on three variants of MT tasks did not show any obvious gains, as shown in the third group in Table 1.

4 Multimodal multi-task fine-tuning

As an alternative approach for utilizing MT data in S2ST, we trained a Translatotron 2 model and a Transformer MT model in a multi-task learning setup, with a pre-trained mSLAM encoder and a randomly initialized Transformer decoder shared between the two tasks (Figure 1). At each training step, one task was sampled randomly with equal probabilities and was trained with the corresponding data. Because the MT dataset is extremely unbalanced among languages, we experimented with different data sampling temperature on the MT task. As the results shown in the fourth group in Table 1, such multimodal multi-task fine-tuning further improved the translation quality by +1.3 BLEU, suggesting that it is effective on untilizing weakly-supervised MT data for improving direct S2ST. There was no obvious performance difference by increasing the data sampling temperature $\tau_{\textrm{MT}}$ from 1.0 to 5.0.

5 TTS-based data augmentation

We used a multilingual multispeaker PnG NAT model to synthesize the source text from the MT dataset into speech, and used the same English TTS model as in to synthesize the targets. The multilingual PnG NAT model was trained on an proprietary dataset covering 20 source languages from CVSS-C except for Mongolian. For Mongolian text, we used a proprietary Mongolian G2P system to convert the text into phonemes and fed them into the multilingual PnG NAT together with the text written in the Cyrillic system, regardless that the model was not trained on any Mongolian data. The S2ST model is trained on a mixture of the CVSS-C corpus and the augmented dataset with equal data sampling probabilities.

As can been from the fifth group in Table 1, training with the augmented dataset obtained higher performance improvement than the multimodal multi-task fine-tuning approach. In contrast to the latter, applying a high data sampling temperature $\tau_{\textrm{aug}}=5.0$ on the augmented dataset further boosted the performance significantly, especially on the low resource languages, which is consistent with increased data sampling ratio on them. With $\tau_{\textrm{aug}}=5.0$ applied, the translation quality is further improved by +4.2 BLEU (or +24% relatively) on average on top of the encoder pre-training. The improvement on low-resource languages is drastic, e.g. 1.3 $\to$ 18.5 on Indonesian, 7.0 $\to$ 22.4 on Arabic.

6 Scaling up

We further examined scaling up the encoder and the decoder of Translatotron 2, specifically by increasing the encoder to a 1408 $\times$ 32 Conformer with 1.8B parameters and the decoder to a 768 $\times$ 12 Transformer with 113M parameters, following hyper-parameters from , respectively. The decoders were not pre-trained with MT tasks since no benefits were observed.

The results are depicted in Table 2. As can be seen, the impact of scaling up the encoder and the decoder is consistent among all approaches. Scaling up the encoder from 0.6B to 1.8B consistently further improved the performance by $\sim$ 3 BLEU in all groups; while scaling up the decoder from 25M to 113M brought only a minor improvement ( $\leq$ 0.3 BLEU). The best result was obtained by scaling up both the encoder and the decoder, alongside TTS-based augmentation, which outperformed the prior state-of-the-art trained on CVSS corpus only by +13.6 BLEU (or +113% relatively). The improvement on the low resource languages are most significant (+398% on average).

Discussion

As the experimental results show, leveraging unsupervised and weakly-supervised data can drastically improve the performance of direct S2ST. Such improvements can be critical in practice because S2ST is a data scarce task. In our experiments, such improvements primarily came from self-supervised pre-training on a massive amount of unlabeled speech data and utilizing weakly-supervised MT data in the supervised S2ST training, both of which were effective across the board on high/mid/low-resource languages. We did not observe improvements from pre-training on an MT task and its variants.

Between the two approaches for utilizing MT data in supervised S2ST training that we explored, augmenting MT data into S2ST data by a TTS system obtained superior performance than S2ST and MT multimodal multi-task learning, when a pre-trained speech-text joint representation was utilized for the latter.

Despite the smaller improvement obtained from the multimodal multi-task learning approach, it has the advantage of lower computational overhead by avoiding synthesizing a large amount of text data into speech, and without concerns on potential bias towards synthetic speech input in the trained S2ST model. These results confirmed the effectiveness of the speech-text joint representation learned from mSLAM, while also suggest headroom for further improvement.

While using a Transformer decoder obtained significant translation quality improvements, because it includes cross-attention in every decoder layers, it may have a potential disadvantage on maintaining the temporal synchronization between the linguistic information and the acoustic information in the decoder output, compared to the original Translatotron 2. Such a difference may negatively impact the performance on fine granularity paralinguistic and non-linguistic information preservation during speech translation, which we did not explore in this work. Possible mitigations include modifying the Transformer decoder to use cross-attention only in its bottom layer.

Because the baseline Translatotron 2 already obtained human-parity on the naturalness of the predicted translation speech on the CVSS-C corpus , its performance is unlikely bounded by the synthesizer component. Therefore, we did not explore utilizing more data for improving the synthesizer.

Conclusions

We explored multiple approaches for leveraging much more widely available unsupervised and weakly-supervised speech and text data to improve the performance of direct S2ST based on Translatotron 2. With the most effective approaches applied, the translation quality of direct S2ST on the CVSS-C corpus was improved by +13.6 BLEU (or +113% relatively) as the average on all the 21 language pairs, and +398% relatively as the average on 12 low-resource language pairs, compared to the previous state-of-the-art trained without additional data. Our comparative studies suggest future research directions on S2ST and speech representation learning.

Acknowledgments

The authors thank Benjamin Lee for his help on improving the multi-task learning infrastructure in the Lingvo framework and Chung-Cheng Chiu for helpful feedback.