Cross-modal Contrastive Learning for Speech Translation

Introduction

End-to-end speech-to-text translation (E2E ST) becomes important in many internet products and real applications. An E2E ST system accepts audio signals as the input and generates the target translation using a single model. Compared with the conventional cascade ST models, E2E ST models have achieved almost comparable Bentivogli et al. (2021) or even superior Ansari et al. (2020); Potapczyk and Przybysz (2020); Xu et al. (2021) performance.

The performance of an E2E ST model is still restricted for languages with relatively small parallel data, compared to text machine translation (MT). Existing approaches for ST focus on using additional data from MT and automatic speech recognition (ASR). This can be realized through pre-training approaches Zheng et al. (2021); Dong et al. (2021b, a) or multi-task training frameworks Tang et al. (2021b); Ye et al. (2021); Han et al. (2021).

Different from the data perspective, this paper investigates the bottleneck of E2E ST from the neural representation perspective. We believe that when the representation of audio input is similar to its corresponding textual representation, it is easier for information to transfer from MT to ST, thus improving speech translation performance.

We analyze Transformer models for speech translation and observe a noticeable modality gap between encoder representations of speech and text (Sec. 6 has more details) from existing ST models. An ideal representation should satisfy: if the content of the speech and transcription are similar, their encoded representations should likewise be close to each other. Nevertheless, how to learn unified and aligned speech-text representations?

Inspired by the recent progress of contrastive learning approaches in cross-lingual Lample and Conneau (2019); Pan et al. (2021) and cross-modal vision-and-language domains (Li et al., 2021; Zhou et al., 2020; Dong et al., 2019), we designed a simple contrastive learning method for ST (ConST) to learn the representations that meet the aforementioned conditions explicitly. On the one hand, our model inherits the advantages of the previous multi-task learning methods. On the other hand, it reduces the gap between the representations of speech and its corresponding transcription.

We develop ConST for speech translation, a cross-modal contrastive learning method, on top of the multi-task training framework.

Our experiments on the MuST-C benchmark to show ConST achieves an average BLEU score of 29.4, outperforming the best previous baseline.

We show that ConST indeed learns similar representations for two modalities and better retrieves text with speech input.

Related Work

End-to-end ST To alleviate the error propagation in the cascaded ST systems and to make the deployment simpler, Bérard et al. (2016); Weiss et al. (2017) proposed to use an end-to-end architecture to directly translate speech into text in another language, without the intermediate transcription. Kano et al. (2017); Berard et al. (2018); Inaguma et al. (2020); Wang et al. (2020a); Zhao et al. (2021a) implemented several off-the-shelf encoder-decoder E2E-ST models, such as BiLSTM Greff et al. (2016) and Speech-Transformer Dong et al. (2018). However, training an end-to-end speech translation model is difficult because we need to design a cross-modal cross-language model, meanwhile, the speech-transcription-translation supervised data for speech translation is significantly less than that of MT and ASR. Methods, like data augmentation Park et al. (2019); Pino et al. (2020); Chen et al. (2021), pre-training Weiss et al. (2017); Berard et al. (2018); Bansal et al. (2019); Wang et al. (2020b); Alinejad and Sarkar (2020); Dong et al. (2021a); Zheng et al. (2021), self-training Pino et al. (2020); Wang et al. (2021a), utilizing self-supervised pre-trained audio representation Wu et al. (2020); Han et al. (2021); Ye et al. (2021); Wang et al. (2021a), are proved to be effective. Meanwhile, some work has shown that the encoder-decoder model with a single encoder cannot encode speech information well. For example, Dong et al. (2021b) first proposed a second encoder to further extract semantic information of the speech sequence. Xu et al. (2021) proposed a stacked acoustic-and-textual encoder and introduced large-scale out-of-domain data. Also, multi-task frameworks Le et al. (2020); Tang et al. (2021b); Ye et al. (2021) are widely applied to further enhance the robustness for ST. As a cross-modal task, some work has noted the problem of the modality gap. Han et al. (2021) designed a fix-size semantic memory module to bridge such a gap, from the neuroscience perspective. However, we find that this approach actually sacrifices the effect of MT. So in this paper, we propose a simple yet effective contrastive learning method to bridge the gap and to improve ST performance.

Cross-modal grounding learning This paper attempts to address the problem in speech translation from the perspective of cross-speech-text representation learning. We are also inspired by cross-modal representation learning in the acoustic word embedding (AWE) Palaskar et al. (2019); Kamper et al. (2020); Hu et al. (2020) and the visual-language pre-training (VLP) Wu et al. (2019); Lu et al. (2019); Chen et al. (2020b); Li et al. (2021) tasks. These works usually focus on enhancing textual representations with acoustic or visual information, in other words, grounding learning. In this work, we consider the its dual form, i.e., grounding speech representations using text.

Contrastive learning Our method is motivated by the recent success in contrastive representation learning. The contrastive learning method was first proposed to learn representations from unlabeled datasets (hence the term, self-supervised learning) by telling which data points are similar or distinct, especially in the field of computer vision Chopra et al. (2005); Gutmann and Hyvärinen (2010); Schroff et al. (2015); Sohn (2016); Oord et al. (2018); Chen et al. (2020a); Grill et al. (2020). Khosla et al. (2020) extended the self-supervised batch contrastive approach to the fully-supervised setting and proposed a supervised contrastive learning method. In speech processing, representative methods focused on speaker identification Ravanelli and Bengio (2018), speech recognition Schneider et al. (2019), and audio representation learning van den Oord et al. (2018); Baevski et al. (2020). In the NLP area, the contrastive framework is used for sentence representation learning Fang et al. (2020); Shen et al. (2020); Gao et al. (2021); Wu et al. (2021); Yan et al. (2021), machine translation Pan et al. (2021), and summarization Wang et al. (2021b); Cao and Wang (2021). Very recently, contrastive learning is also applied to learning a unified representation of image and text Dong et al. (2019); Zhou et al. (2020); Li et al. (2021). Motivated by the contrastive learning frameworks in cross-lingual and cross-modal topics, we introduce a similar idea in speech translation.

The ConST Approach

An end-to-end speech translation model directly translates audio sequence $\mathbf{s}=(s_{1},...,s_{|\mathbf{s}|})$ to the text $\mathbf{y}=(y_{1},...,y_{|\mathbf{y}|})$ in the target language. Speech translation corpus $\mathcal{D}=\{(\mathbf{s},\mathbf{x},\mathbf{y})\}$ provides transcript $\mathbf{x}=(x_{1},...,x_{|\mathbf{x}|})$ in the source language, as well.

In this section, we present the overall speech translation model and cross-modal contrastive learning. We also provide several feasible strategies to construct more positive and negative pairs to enhance the contrastive learning.

Our model consists four sub-modules: a speech encoder, a word embedding layer, a Transformer Encoder and a Transformer decoder (Figure 2). It is designed to take either speech or a sentence as input, and to output either source transcript or target translation text. Such architecture enables a universal framework for multiple tasks, including ST, MT and ASR.

The speech encoder module (S-Enc) is designed to extract low-level features for speech signals. It contains Wav2vec2.0 Baevski et al. (2020) and two additional convolutional layers. The input is raw waveform signal sampled at 16kHz. Each convolutional layer has a stride of 4 and $d$ channels. In total, it shrinks the time dimension by a factor of $4$. Denote $\mathbf{a}=\text{S-Enc}(\mathbf{s})$ as the audio representation of the speech, $|\mathbf{a}|\ll|\mathbf{s}|$.

Parallel to the speech encoder is the word embeeding layer. It is the same as word embedding for text translation.

Both the speech encoder and word embedding layer are connect to Transformer encoder and then passed to the Transformer decoder. The Transformer encoder and decoder are using the same configuration as the original Vaswani et al. (2017). To explain, the Transformer encoder further extracts the high-level semantic hidden representation of two modalities. The Transformer decoder generates the word sequences (transcription and translation) for ST, MT and ASR tasks. Since our model has a complete Transformer encoder-decoder as a sub-module, this makes it possible to pre-train using large-scale extra MT parallel data.

Previous work has shown that multi-task learning on ST, MT and ASR improves translation performance Indurthi et al. (2020); Tang et al. (2021b); Ye et al. (2021). Our training loss consists of the following elements.

The first three elements are cross-entropy losses on , and pairs. These pairs are built from the triplet ST data. We also introduce a cross-modal contrastive loss term $\mathcal{L}_{\text{CTR}}$ (see Section 3.2 for details). It aims to bring the representation between the speech and textual transcription modalities closer (its effect will be analyzed in detail in Section 6). $\lambda$ is a tuned hyper-parameter of the weighted contrastive loss term.

2 Cross-modal Contrastive Learning

As mentioned in the beginning, since we need to produce similar representations for the speech and transcript sharing the same semantic meanings, we propose cross-modal contrastive learning method to bring their representations closer together. The main idea of cross-modal contrastive learning is to introduce a loss that brings speech and its corresponding transcript (positive example) near together while pushing irrelevant ones (negative examples) far apart.

Given a positive example of such a speech-transcript pair $(\mathbf{s},\mathbf{x})$, we randomly pick a set of $N-1$ transcripts $\{\mathbf{x}_{i}^{-}\}_{i=1}^{N-1}$ from the same batch as negative examples. For speech $\mathbf{s}$ and its transcript $\mathbf{x}$, we first average them in terms of the time dimension,

and apply the multi-class N-pair contrastive loss Sohn (2016):

where $\mathcal{A}=\{\mathbf{x}\}\cup\{\mathbf{x}_{i}^{-}\}_{i=1}^{N-1}$, $\tau$ is the temperature hyper-parameter, and $sim$ is the cosine similarity function $sim(a,b)=a^{\top}b/\|a\|\|b\|$. In the implementation, negative examples $\{\mathbf{x}_{i}^{-}\}_{i=1}^{N-1}$ are from the same training batch of data (Figure 2(b)).

3 Mining Hard Examples for Contrastive Learning

To further enhance the contrastive learning, we introduce three strategies to mine additional hard examples. These strategies are at input and representation (gray shaded modules in Figure 2(a)). Specific schematic illustrations of each operations are shown in Figure 3.

Span-Masked Augmentation We mask consecutive segments of an original audio waveform sequence $\mathbf{s}$ to obtain a new modified speech $\mathbf{s}^{\prime}$. We take $\mathbf{s}^{\prime}$ as an input to the model, and compute the contrastive loss on its original corresponding transcript. We randomly sample without replacement all time steps in the original waveform of the speech to be the starting indices with a probability $p$, and then we set the sub-sequence $M$ successive time steps to be blank. In the experiment, we tried multiple configurations, and found $p=0.25,M=3600$ the best, resulting in a masked span of $0.225$ second. Since the masked speech fragment is very short, we consider the masked speech and the original transcript to be positive pairs, and the remaining transcripts in the same batch to be negative pairs.

Word Repetition The word repetition strategy randomly replicates some words (or sub-words) in the original sentences, with two advantages for improving representation robustness. First, as the length of the sentence is shorter than that of its audio representation, randomly repeating the words in the sentence is a simple yet useful technique to increase the length. Second, repeating words does not change the semantics and is suitable as an extra positive example of the corresponding speech. Specifically, given sentence $\mathbf{x}$, each sub-word token $x_{i}$ can be duplicated $k$ more times, resulting in the duplicated sentence $\mathbf{x}^{\prime}$, where $k=0,1,2,...$ and $k\sim\text{Poisson}(1)$. We regard $\mathbf{x}^{\prime}$ as the additional positive example for the speech $\mathbf{s}$ and the samples with the same operation in the same batch as the negative examples.

Cut-off strategy Recent studies on natural language understanding and generation have proved cut-off strategy to be successful Shen et al. (2020); Yan et al. (2021). We analogize a similar idea to the cut-off approach for speech representation. We entirely erase a slice of the $T\times d$ representation matrix along each dimension and set the erased terms to 0. Here, we present two variants: sequence cut-off, which erases some sequence dimension, and feature cut-off, which erases some feature dimension. Note that there is a difference between cut-off and dropout. Dropout randomly sets some elements to 0, while cut-off is a dimensional “block" dropout. Similarly, we treat the cut-off audio representation and the original transcribed sentence as positive pairs, and the rest sentences in the same batch as negative pairs.

Experiments

ST datasets We conduct experiments on all the translation directions in MuST-C dataset We use v1.0. https://ict.fbk.eu/must-c/ Di Gangi et al. (2019): English (En) to German (De), Spanish (Es), French (Fr), Italian (It), Dutch (Nl), Portuguese (Pt), Romanian (Ro) and Russian (Ru). As one of the largest ST benchmarks, MuST-C contains more than 385 hours of TED talks for each direction.

MT datasets We also introduce external WMT datasets Bojar et al. (2016) for En-De/Es/Fr/Ro/Ru and OPUS100 datasets Zhang et al. (2020) for En-It/Nl/Pt directions, as the expanded setup.

Table 8 (in Appendix. A) lists the statistics of all the datasets included.

Model Configurations The Wav2vec2.0 in the S-Enc is only pre-trained on Librispeech Panayotov et al. (2015) speech without any downstream fine-tuninghttps://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small.pt. Two layers of CNNs after the Wav2vec2.0 are set to kernel size 5, stride size 2 and hidden size 512. The Transformer follows the base configuration, with 6 layers of encoder and decoder, hidden size $d=512$, 8 attention heads, and 2048 FFN hidden states. We use pre-layer normalization for stable training. The model with the above configurations has a total of about 150M parameters.

Experiment Details We evaluate case-sensitive detokenized BLEU using sacreBLEUhttps://github.com/mjpost/sacrebleu, BLEU Signature: nrefs:1 — bs:1000 — seed:12345 — case:mixed — eff:no — tok:13a — smooth:exp — version:2.0.0 Post (2018) on MuST-C tst-COMMON set. In the analysis, we also report the ChrF++ score ChrF2++ Signature: nrefs:1 — bs:1000 — seed:12345 — case:mixed — eff:yes — nc:6 — nw:2 — space:no — version:2.0.0 Popović (2017) and the learning-based BLEURT score https://github.com/google-research/bleurt Sellam et al. (2020). As recommended, the checkpoint we use is BLEURT-20.. We use the raw 16-bit 16kHz mono-channel speech input. We jointly tokenize the bilingual text using SentencePiece Kudo and Richardson (2018), with a vocabulary size of 10k, which is the same as Ye et al. (2021)’s setup. For the training loss, we set contrastive temperature $\tau=0.02$ and weight of contrastive term $\lambda=1.5$ for German and Dutch, and $\lambda=1.0$ for the other languages.

Appendix B contains more detailed settings and explanations for the baseline models in Table 1. Appendix C shows the experiments on the choice of the hyper-parameters.

2 Main Results

Comparison with end-to-end ST models Table 1 shows the main results. Since many existing works regard “leveraging external data” to be one of their model’s features, their strong performances are largely predicated on the utilization of auxiliary data, especially large-scale MT data. For a relatively fair comparison, we investigate two cases: (1) without external MT data and (2) with external MT data. Without the external MT data, our method already gains an average improvement of 0.5 BLEU over the previous best models. Also when speech data is introduced for pre-training, our method works better than others (Self-training, W-Transf. and XSTNet). When extra MT data are introduced, our method also outperforms SOTA by an average of 0.6 BLEU. Among the benchmark models, with the same goal of closing two modality gaps, Chimera Han et al. (2021) constructed an extra fixed-length shared semantic space. However, the shared memory with fixed size actually compromises the MT performance, while our contrastive learning approach is more straightforward and effective.

Comparison with cascaded ST systems We compare our method with several cascade baselines, where Ye et al. (2021) and Xu et al. (2021) provided two strong cascade systems trained using MuST-C and external ASR and MT data (LibriSpeech, WMT, and Opensubtitles). From Table 2, we find that as an end-to-end model, ConST can outperform these strong cascade models. In Appendix 7, we provide a case study to show such improvement.

Analysis

With the same model architecture and the same pre-training + fine-tuning procedure, the main difference between ConST and XSTNet Ye et al. (2021) is whether we use the contrastive loss term during the fine-tuning or not. Comparing the BLEU results of w/o and w/ external MT data situations in Table 1, we find that ConST further improves $0.5$ and $0.6$ BLEU scores in terms of eight translation directions on average, which proves the effectiveness of the cross-modal contrastive learning. By gradually removing each losses in Eq.( 1), Table 3 shows the improvements bringing by the multi-task learning and the contrastive learning. For En-De translation direction, contrastive learning can bring an average improvement of 0.9 BLEU over the baseline models by only optimizing $\mathcal{L}_{\text{ST}}$ (corresponding to the last row of the Table 3), and multi-task learning can lead to a further improvement of about 1.2 BLEU on top of that.

2 Which layer to contrast on?

An intriguing question is which representations should be considered in the contrastive loss function. In the method part (Section 3.2), we use averaged audio representation $u$ for speech $\mathbf{s}$ (Eq.(2)) and averaged lexical embedding $v$ for the transcript $\mathbf{x}$ (Eq.(3)), denoted as low-level repr.. Whereas inspired by a recent study in multilingual MT (Pan et al., 2021), we also provide an alternative contrastive loss as a comparison, whose speech and text features are average-pooled semantic representations derived from the Transformer encoder, denoted as high-level repr..

Table 4 shows that contrastive learning using the low-level representations (Line 1) is better than using the high-level ones (Line 2). On the other hand, although the performance of Line 2 is relatively inferior, it still outperforms the multi-task model without the contrastive loss (Line 3). The detailed analysis of possible explanations will be shown in Section 6.2.

3 Is contrastive loss better than other losses?

Our goal for introducing the contrastive loss term (denoted as CTR Loss) is to close the distance between speech and text representations. Whereas, there are other options to achieve this goal, such as L2 loss and CTC loss.

L2 Loss: Without introducing any negative samples, L2 loss directly reduces the Euclidean distance between the representations of two modalities by minimizing $\mathcal{L}=\|u-v\|^{2}$. L2 loss can be viewed as an implementation based on the idea of knowledge distillation Heo et al. (2019); Dong et al. (2021b).

CTC Loss: The connectionist temporal classification (CTC) loss Graves et al. (2006) is commonly used in speech-related tasks Xu et al. (2021); Dong et al. (2021b). Unlike contrastive loss that cares about the representation, CTC loss connects the two modalities by establishing speech-text alignment and maximizing $p(\mathbf{x}|\mathbf{a})=\sum_{\pi\in\Pi_{\mathbf{s},\mathbf{a}}}\prod_{t=1}^{T}p_{t}(\pi_{t}|\mathbf{a})$, where $\Pi_{\mathbf{s},\mathbf{a}}$ is the set of all valid alignments.

Compared to the other two ways of bridging the modality gap, L2 and CTC loss, is the contrastive loss term better? The answer is yes according to the results in Table 5. Our explanation is that information on the negative samples benefits the contrastive loss, bringing the the distance between the speech and its corresponding transcription closer while pushing the distance to the irrelevant text farther.

4 Analysis on the hard example mining strategies

In Section 3.3, we proposed four methods to mine the hard examples for contrastive learning, namely span-masked augmentation (SMA), word repetition (Rep), sequence cut-off (SCut), and feature cut-off (FCut). In this section, we study how effective these methods are, and to do so, we consider the BLEU performances of their 15 combinations (Figure 4). Note that “Original” means the original contrastive loss in Eq.(4) without any additional hard examples mining operation, and the diagonal in the heat map represents only one strategy used. For an easy and fair comparison, we set the weight of the contrastive term to 1.0 uniformly. We have the following observations.

All the hard examples mining methods are effective. All the BLEU scores in Figure 4 exceed the strong multi-task model trained without contrastive learning ($27.1$). Among all the strategies, the combination of the original and SCut reaches the best result ($28.3$), and is better than the model without any expanded operations ($p<0.01$). Generally, to find the best model, we suggest adopting multiple strategies and choosing the best checkpoint on the dev-set.

The combinations of the hard examples mining methods and the “original” have relatively better performances. We argue that we need the original positive and negative examples to give more accurate representations (without any dropout) for contrastive learning. On the contrary, without the help of “original” loss, the performance with both sequence cut-off and feature cut-off is the worst in Figure 4, probably because too much information is lost by superimposing the two.

Why does cross-modal contrastive learning work? — Analysis on the Modality Gap

As mentioned earlier, the existing multi-task training models cannot address the speech-text modality gap. Does ConST reduce the representation gap between speech and text?

Does the speech-text modality gap exist without explicitly bridging the two? Speech-text modality gap means the discrepancy between the audio representations and transcription sentence embeddings. To visualize it, we plot the bivariate kernel density estimation Parzen (1962) (KDE) contour of their dim-reduced features, where T-SNE (Van der Maaten and Hinton, 2008) is used to reduce the dimension into two (Figure 5). Ideally, if the representations of speech and its corresponding transcript are similar, their KDEs will be similar, and thus the contour lines will overlap as much as possible. However, Figure 5(a) is the KDE contour of the multi-task framework without any explicit modeling to bring two modalities together Ye et al. (2021). It shows that the representations are so dissimilar that they are organically divided into two clusters, i.e. speech-text modality gap exists.

Does ConST reduce the modality gap? As shown in Figure 5(b), compared to the baseline model without contrastive learning, ConST with cross-modal contrastive learning is able to bring representations of different modalities much closer. This means that the audio representation contains more linguistic information similar to that of the textual transcription, which is more advantageous for the downstream ST generation through the shared Transformer encoder and Transformer decoder.

2 Cross-modal Retrieval

How good is the cross-modal representation space learned from ConST? To answer this question, we conduct a retrieval experiment, i.e. finding the nearest (smallest cosine similarity) transcript based on the speech representation. We compare ConST model with the baseline without cross-modal contrastive learning and report the top-1 retrieval accuracy using (1) the low-level representations and (2) the high-level semantic representations, in Table 7.

When retrieving the text using low-level representations, our method gains a substantial 79% increase over the baseline. In addition, we find that without explicit contrastive modeling, the baseline can achieve retrieval accuracy of more than $94\%$ according to the semantic representations outputted from the Transformer encoder. We believe that such high accuracy is automatically learned from the triple-supervised data itself under the multi-task learning framework. With such a degree of cross-modal alignment, if we construct the contrastive loss with semantic representations, its gain to the ST performance turns out to be limited, which exactly corroborates the findings in Section 5.2 – low-level representations are preferred in the cross-modal contrastive learning.

Case Analysis

In this section, we use several cases that ConST generates. We compare our model with the cascaded model and the previous end-to-end model, XSTNet Ye et al. (2021).

For this first case, the cascaded system fails to give a right translation due to the mis-punctuation issue (klingt is a verb), while the end-to-end model, XSTNet and ConST translate correctly. For the second case, the previous end-to-end XSTNet model cannot accurately translate the phrase “started exploring this idea of”, which performs worse than the cascaded one. Whereas ConST successfully conveys the meaning of “this idea” , and translates more accurately than XSTNet. We believe this improvement comes from the cross-modal contrastive learning.

Conclusion

In this paper, we propose ConST, a simple yet effective contrastive learning framework bridging the speech-text representation gap and facilitating the ST with limited data. We also provide feasible hard example mining methods to learn robust representations. The results on the MuST-C ST dataset prove the effectiveness of the method.

Broader Impact

This work improves the performance of ST tasks on public datasets by learning speech representations that are more similar to text representations, but the model is far from being achieved for industrial-grade implementations. In real scenarios, for example, the original voice is noisier and the distribution of speech lengths is more complex than in the public dataset, which cannot be handled by an end-to-end model alone. The shortcoming of this model is that it still needs a certain amount of labeled data for training, especially to learn better speech representation, and for the more than $7,000$ languages and dialects in the world, most of them do not have corresponding translations or even transcriptions, our method does not work in untranscribed scenarios. In this paper, we focus on the improvement brought by the better speech representation on the ST task, and obtained good results with hundreds of hours of speech data. We hope that our work achieves better results using more data (e.g. raw speech, raw text, ASR, MT data) in the future.

References

Appendix A Statistics of all datasets

Appendix B Experimental Details

Training and Implementation Details We use Adam optimizer ($\beta_{1}=0.9,\beta_{2}=0.98$) with learning rate $=1e^{-4}$ and warmup 25k steps during the ST training. We also implement the expanded setting with the introduction of external WMT to train the Transformer module. In the pre-training stage, we set the learning rate $=7e^{-4}$ and warmup 4000 steps. For robust training, we set label smoothing to $0.1$, and dropout rate to $0.1$. The hyper-parameters for different data augmentation methods are as follows: for masked audio span strategy, we set masking probability $p=0.25$ and masking span length $M=3600$ frames; for both sequence and feature cut-off, we set the cut-off dropout rate as $0.1$. We save the checkpoint with the best BLEU on dev-set and average the last 10 checkpoints. For decoding, we use a beam size of 10 and length penalty $0.7$ for German, $1.0$ for French, and $0.4$ for Russian. We train the models in 8 Nvidia Tesla V100 GPUs for each experiment. We use Fairseq Ott et al. (2019) as the code-base for our implementation.

Baseline Models In Table 1, we compared our method with end-to-end baseline models whose audio inputs are 80-channel log Mel-filter bank, including: FairseqST Wang et al. (2020a), NeurST Zhao et al. (2021a), Espnet ST Inaguma et al. (2020), Dual-decoder Transformer Le et al. (2020), SATE Xu et al. (2021), Speechformer Papi et al. (2021), self training Pino et al. (2020) and mutual learning Zhao et al. (2021b) method, STAST Liu et al. (2020b), bi-KD Inaguma et al. (2021), MLT method Tang et al. (2021b), Lightweight Adaptor Le et al. (2021), JT-S-MT Tang et al. (2021a), FAT-ST Zheng et al. (2021), TaskAware Indurthi et al. (2021), and STPT Tang et al. (2022). We also compare our method to baseline models that have pretrained Wav2vec2.0 as a module, including:

W-Transf. Ye et al. (2021): the model has the same structure as ours, but is only trained on parallel data.

Chimera-ST Han et al. (2021): the model that builds a shared semantic memory for both audio and text modalities.

XSTNet Ye et al. (2021): the model has the same structure as ours, and adopted a multi-task fine-tuning strategy.

STEMM Fang et al. (2022): the model that bridges the modality representation gap by minimizing the Jensen–Shannon divergence between the original speech representation and the manifold mix-up representation.

Appendix C The Choice for Hyper-parameters

Influence of Temperature In the contrastive loss, the temperature hyper-parameter is provided to control the smoothness of the distribution normalized by softmax operation. A high temperature helps to smooth the distribution, making it more difficult for the model to distinguish between positive and negative samples (corresponding to correct transcriptions and other transcriptions in this work), while the low temperature behaves just the opposite. We choose several temperature hyper-parameters ranging from $0.01$ to $0.5$, and Figure 6 shows their BLEUs on the test and dev sets . We find that (1) the choice of the temperature does not drastically affect the final BLEU score, and (2) we recommend that the temperature $\tau$ be set between 0.02 and 0.05 to ensure a relatively good ST performance. In the experiment, we use $\tau=0.02$.

Influence of Contrastive Loss Weight The total loss we optimize, Eq.(1), is a linear combination of the multi-task cross-entropy losses $\mathcal{L}_{\text{MLT}}$ and the contrastive term $\mathcal{L}_{\text{CTR}}$. To investigate how much the contrastive terms affect BLEU, we fix its temperature $\tau=0.02$, adjust the values of its loss weight $\lambda$ from 0.1 to 2.0, performed three experiments for each value, and test the average BLEU on En-De tst-COMMON set. Figure 7 depicts the performances. First, all objective functions containing $\mathcal{L}_{\text{CTR}}$, even if their weights $\lambda$ take different values, are apparently better than the baseline model with $\mathcal{L}_{\text{MLT}}$ only $\mathcal{L}_{\text{CTR}}$. Then, the best BLEU score is achieved at loss weight $\lambda=1.5$, corresponding to the results in Table 1. And when analyzing the effect of data augmentation strategies (Section 5.4), since we need to consider the combination between them, which is more complicated. Therefore, we set the loss weight to $1.0$ uniformly for simplicity. In general, we recommend that the weight hyper-parameter takes a value between $0.8$ and $1.5$.

Appendix D Data Scale for Fine-tuning

The experiments in the main paper show that our model can perform well by introducing external MT data pre-training. Here, we simulate the scenario with plenty of MT and speech data and limited ST triple-labeled data, and does ConST have the ability of low-resource learning? In the experiment, we reduce the labeled ST data to 1, 10, and 100 hours, corresponding to sentence counts of about 500, 5k, and 50k sentences. For a fair comparison, we use the same MT pre-trained Transformer module as in the main paper. We find the contrastive loss particularly helpful when the amount of speech data is extremely small, like only 1 hour of speech. Second, the multi-task training strategy is also very effective in improving the robustness of the model performance. We also find that by using easily accessible MT and speech pre-training, our model could reach the previous baseline results without pre-training using only $1/4$ of the original data, i.e. $100$ hours of labeled ST data.