Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias

Introduction

Text-to-speech (TTS) synthesis aims to generate human-like speech from text and has gained significant attention in the field of machine learning. Traditional TTS systems are usually trained on limited datasets, which impairs their models’ ability to produce diverse and generalizable results. In contrast, large-scale TTS systems are trained on tens of thousands of hours of speech data, which significantly improves their zero-shot capability . Current large-scale TTS systems typically encode the speech waveform into latent with neural codec models as the intermediate representation and model it with autoregressive language models (LM) or diffusion models .

As presented in Table 1, human speech can be decoupled into several attributes: content, timbre, prosody, phase, etc. However, current large-scale TTS systems directly use neural audio codec models to encode the entire speech into latent and ignore the following intrinsic nature of speech: 1) phase is highly dynamic and irrelevant to semantics, which means people are much less sensitive to perceive phase than to prosody and timbre, especially for monaural audio. Therefore, only one reasonable phase is needed for waveform reconstruction, and it is not necessary to model all possible phases. Modeling phase with LM or diffusion model can waste a lot of model parameters since they model the full distribution of phaseThat is why GAN-based vocoders are popular.. 2) Timbre should remain stable within the sentence as a global vector. Modeling timbre with time-varying latent is costlyOur method retains a small portion of time-varying timbre information in the latent code, while the majority is represented as the global vector.. 3) Prosody typically has both local and long-term dependencies and changes rapidly over time with a weak correlation to text, which makes conditional phoneme-level LLMs inherently ideal for generating prosody sequences. 4) Content has monotonic alignment with speech while the autoregressive language model cannot guarantee that, which can lead to repeating or missing word issues .

To make use of the large and wild training datasets while matching the inductive bias of the model and the intrinsic nature of speech, we propose a zero-shot text-to-speech model called Mega-TTS. Specifically, 1) considering the limitations of neural audio codec models, we select mel-spectrogram as the intermediate representation to separate the phase and other attributes. We adopt a GAN-based vocoder to reconstruct the phase information to improve our model’s efficiency. 2) To model timbre information, we employ global vectors since timbre is a global attribute that changes slowly over time. We extract the global information from a different speech of the same speaker with the global speaker encoder to decompose the timbre and content information. 3) To capture prosody information in a sentence, we adopt a VQGAN-based acoustic model to generate the mel-spectrogram and a latent code language model called P-LLM to fit the distribution of prosody. The P-LLM is capable of capturing both local and long-range dependencies for prosody modeling.

To evaluate the zero-shot performance of Mega-TTS, we perform experiments on VCTK , AISHELL-3 and LibriSpeech test-clean datasets. All of the test speakers are unseen in the training corpus. Our Mega-TTS surpasses the state-of-the-art zero-shot TTS systems in terms of speaker similarity, speech naturalness, and generation robustness, which demonstrates the superiority of introducing appropriate inductive biases. Moreover, Mega-TTS outperforms state-of-the-art models on speech editing and cross-lingual TTS tasks. The main contributions of this work are summarized as follows:

We propose Mega-TTS, a zero-shot text-to-speech system that considers intrinsic inductive biases. Instead of using latent encoded by audio codec as the intermediate representation , we decompose mel-spectrogram into content, timbre, prosody, and phase attributes and model each of them according to their intrinsic properties.

We train Mega-TTS on a multi-domain and multi-lingual dataset that contains 20k hours of speech data. It is worth noting that existing large-scale TTS systems are typically trained with speech corpora from audiobooks, while our system is trained on multi-domain speech corpora.

We evaluate Mega-TTS on 3 down-stream speech generation tasks (i.e., zero-shot TTS, speech editing, and cross-lingual TTS), demonstrating that Mega-TTS can be applied to various speech generation tasks. We also propose a novel sampling strategy for speech editing via the discrete prosody tokens extracted by Mega-TTS.

Background

In this section, we briefly overview the background of this work, including zero-shot text-to-speech (TTS) and generative models for speech synthesis.

Text-to-speech models usually generate mel-spectrogram from text and then synthesize speech waveform from the generated mel-spectrogram using a separately pre-trained vocoder , or directly generate waveform from text in an end-to-end manner . For decades, the increasing demand for personalized speech generation in various applications has posed challenges for TTS models , especially in zero-shot multi-speaker scenarios regarding domain shifts. Previous approaches can be categorized into speaker adaptation and speaker encoding methods. Traditional works are typically trained on small datasets , while some recent works are trained on large-scale datasets and demonstrate the effectiveness in zero-shot scenarios. These systems utilize the neural audio codec models to convert audio waveform into latent and consider it as the intermediate representation for speech generation. Among them, SPEAR-TTS splits the TTS task into two sequence-to-sequence tasks, which enables the training using abundant audio-only data. NaturalSpeech 2 uses a text-conditioned diffusion model to generate the latent vectors of the neural audio codec model. VALL-E proposes the first neural codec language model for text-to-speech, exhibiting strong in-context learning abilities to overcome challenges in zero-shot speech generation. However, these methods ignore the intrinsic property of speech and may lead to inferior or uncontrollable results (e.g., word skipping, repeating, and collapse ). Considering the nature of different speech attributes, the autoregressive language model is ideally suitable for prosody modeling. ProsoSpeech has proposed to improve the prosody modeling for TTS with latent prosody vectors predicted by a language model. Nevertheless, it lacks the in-context learning capacity, which restricts its application scenarios.

Generative models for speech synthesis.

Generative models, like language models , VAE , GAN , Normalizing flow , and diffusion model , have been applied to speech or audio synthesis for years. Previous works of autoregressive generative model mainly aim at waveform generation and continuous acoustic feature generation . Recently, speech generation systems like AudioLM and VALL-E propose to utilize neural audio codec models to convert audio waveform into discrete codes as the intermediate representation and design LLMs to generate these codes to achieve speech synthesis. Although good reconstruction quality can be achieved by neural audio codec models, they ignore the intrinsic nature of speech and may not be suitable to serve as the generator of intermediate representation for speech generation. The encoded latent contains the phase, content, and timbre attributes and language models are not suitable for predicting these due to the error propagation problem.

Method

To introduce proper inductive biases into large-scale TTS systems, we propose Mega-TTS, a zero-shot TTS system for natural and robust speech generation in various scenarios (i.e., zero-shot prompt-based TTS, speech editing, and cross-lingual TTS). As shown in Figure 1, Mega-TTS consists of a VQGAN-based TTS model and a prosody large language model (P-LLM). We carefully model different speech attributes in different ways. First, we choose the mel-spectrogram as the intermediate representation as it separates the phase from other attributes very well. Secondly, we extract the global vector from the random previous sentence of the same speaker with the global timbre encoder to disentangle the timbre and content information. Finally, we further use a VQGAN-based acoustic model to generate the mel-spectrogram and propose a latent code language model called P-LLM to fit the distribution of prosody, since language models are capable of capturing both local and long-range dependency. During inference, we propose to use the content from the given text sequence, the timbre extracted from the prompt speech, and the prosody predicted by our P-LLM to generate the target speech, which is a novel TTS decoding mechanism called prosody-oriented speech decoding. Finally, to demonstrate that our model can be applied to various scenarios, we design inference strategies for downstream tasks. We describe these designs and the training and inference procedures in detail in the following subsections.

To introduce appropriate inductive biases into different speech attributes, we need to separately express these attributes and carefully design different architectures for them. The overall model architecture of Mega-TTS is shown in Figure 1. We use three types of encoders to separately encode content, prosody, and timbre representations. Then we adopt a GAN-based mel-spectrogram decoder to generate mel-spectrograms with these representations. We describe the disentangling strategy and detailed design of the proposed encoders as follows.

We disentangle the mel-spectrogram into content, prosody, and timbre representations with the reconstruction loss of the autoencoder and a carefully designed bottleneck : 1) we feed the mel-spectrogram into the prosody encoder, and we also introduce carefully-tuned dimension reduction and phoneme-level downsampling to the prosody encoder to constrain the information flow; 2) the content encoder encodes the phoneme sequence into the content representation; 3) we feed the reference mel-spectrogram sampled from a different speech of the same speaker to disentangle the timbre and content information and temporally average the output of the timbre encoder to get a one-dimensional global timbre vector. The correctly-designed bottleneck will learn to remove the content information and the global timbre information from the output of the prosody encoder, which ensures the performance of disentanglement. Due to the limited page space, we put more details about the hyperparameter selection for the information bottleneck in Appendix D.

Architecture design of encoders.

1) The prosody encoder consists of two convolution stacks, a phoneme-level pooling layer, and a vector quantization (VQ) bottleneck. The first convolution stacks compress mel-spectrograms into phoneme-level hidden states according to the phoneme boundary and the second stacks capture phoneme-level correlations. The vector quantization layer then utilizes these hidden states to obtain phoneme-level prosody codes $\mathbf{u}=\{u_{1},u_{2},...,u_{T}\}$ and hidden states $H_{prosody}$. To ease the difficulty of disentanglement, only the low-frequency band of the mel-spectrogram (the first 20 bins in each mel-spectrogram frame) is used as input, as it contains almost complete prosody and significantly less timbre/content information compared to the full band ; 2) The content encoder is composed of several feed-forward Transformer layers. To achieve the monotonic alignment between the speech content and generated speech, we adopt the duration predictor and length regulator following common practice in non-autoregressive TTS systems . Differently, we feed the prosody information extracted by the prosody encoder to the duration predictor in order to ease the one-to-many mapping problem ; 3) The timbre encoder is designed to extract a global vector $H_{timbre}$ that contains the speaker identity of the given speech. The timbre encoder consists of several stacks of convolution layers. To ensure the stability of timbre information across the time axis, we temporally average the output of the timbre encoder to get a one-dimensional timbre vector $H_{timbre}$.

To keep good perceptual quality, we introduce a GAN-based mel-spectrogram decoder. We adopt the multi-length discriminator based on random windows of different lengths as the discriminator. Overall, the first-stage training loss $\mathcal{L}$ of Mega-TTS can be formulated as:

2 P-LLM

The P-LLM is a latent code language model that captures local and long-range dependency for prosody modeling. We describe the prosody-oriented speech decoding mechanism and details of the P-LLM as follows.

Generating prosody codes.

3 Speech prompting for inference

To facilitate in-context learning for various speech generation tasks, we design different speech prompting mechanisms to encourage Mega-TTS to follow the information in the speech prompt.

Inference for speech editing.

In speech editing, the predicted prosody codes should achieve smooth transitions at both the left and right boundaries of the masked region. Previous works like EditSpeech propose to perform left and right autoregressive inferences separately and concat the mel-spectrogram at the least L2-norm difference fusion point. However, the L2-norm difference of the mel-spectrogram is far from human perception, leading to poor audio naturalness. Since the prosody representations in Mega-TTS is discrete, we can solve the transition problem by operating on discrete prosody representations. First, we regard the area on the left side of the mask as a prompt to generate $N$ candidate paths with top-k random sampling strategy. Secondly, the $N$ generated paths are used as new prompts to generate the probability matrix of the area on the right side of the mask and the ground-truth prosody codes are used to obtain the probabilities of each decoding step from the probability matrix. In the third stage, we sum up the log probabilities of each decoding step for the candidate paths. Finally, we choose the path that achieves the maximum probability in the second step as the predicted result. The decoding strategy for speech editing can be formulated as follows:

where $L$ and $R$ are the left and right boundaries of the mask. $T$ is the length of the mel-spectrogram. $u^{i}$ is the prosody code in the i-th candidate path. $u_{t}^{gt}$ is the ground-truth prosody codes. Since our decoding strategy considers the prosody information of the boundaries on both sides, the edited region can achieve smooth transitions.

Experiments

In this section, we present the evaluation results of Mega-TTS and the comparison with baselines in terms of the objective and subjective metrics.

We use GigaSpeech and WenetSpeech as the training corpora, which contains 20k hours of multi-domain speeches in English and Chinese in total. Since the speech in GigaSpeech and WenetSpeech does not have speaker identities and multiple speakers may appear in a speech clip, we process the datasets with an open-source automatic speaker diarization modelhttps://huggingface.co/pyannote/speaker-diarization . We also extract the phoneme-level alignments with the external alignment toolhttps://github.com/MontrealCorpusTools/Montreal-Forced-Aligner. More information can be found in Appendix A.3.

Evaluation datasets.

We employ two datasets for evaluation: 1) VCTK dataset , an English dataset that contains 108 speakers; 2) LibriSpeech test-clean, an English dataset that contains 40 speakers. For each of these datasets, we randomly sample 10 utterances for each of the 40 speakers, resulting in a subset of 400 utterances for evaluation; Specifically, to synthesize each sample, we randomly select a different utterance of the same speaker to form the speech prompt. Note that all speakers in the evaluation datasets are unseen during training.

Model configuration.

Our Mega-TTS consists of three encoders, a prosody large language model, a mel decoder, and a discriminator. The prosody encoder, timbre encoder, and mel generator consist of 5 convolutional blocks with 320 hidden size, 5 convolution 1D kernel size. The content encoder is a 4-layer Transformer with 2 attention heads, 320 embedding dimensions, 1280 1D convolution filter size, and 5 convolution 1D kernel size. The duration predictor is a 3-layer 1D convolution with ReLU activation and layer normalization, which have 320 hidden size. The discriminator follows the architecture proposed in SyntaSpeech . The P-LLM model is a decoder-only architecture that contains 8 Transformer layers with 8 attention heads, 512 embedding dimensions, 2048 1D convolution filter size, and 5 convolution 1D kernel size. The overall number of model parameters is 222.5M. We add more detailed model configurations in Appendix A.1.

Training and inference.

In the training stage, we train Mega-TTS on 8 NVIDIA A100 GPUs, with a batch size of 30 sentences on each GPU. We use the Adam optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.98$, $\epsilon=10^{-9}$ and follow the same learning rate schedule in . It takes 320k steps for the VQ-GAN TTS model’s training and 100K steps for the P-LLM’s training until convergence. The predicted mel-spectrograms are transformed into audio samples using pre-trained HiFi-GAN V1https://github.com/jik876/hifi-gan . In the inference stage, we use the top-5 random sampling scheme to sample diverse results.

Objective metrics.

We evaluate the pitch distance and speaker similarity for zero-shot TTS. In terms of the pitch distance, we compute the average dynamic time warping (DTW) distances between the pitch contours of ground-truth speech and synthesized speech. And for the cosine speaker similarity, we use the WavLM model finetuned for speaker verificationhttps://huggingface.co/microsoft/wavlm-base-plus-sv to compute the cosine speaker similarity score between the ground-truth speech and synthesized speech. The similarity score is in the range of $\left[-1,1\right]$, where a larger value indicates a higher similarity of input samples. In addition, we also evaluate the word error rate (WER) for cross-lingual TTS. We use the ASR system from the released HuBERT-Large model to transcribe the generated speech into text. Then, the WER between the transcribed text and the original target text is measured. We use all samples in the test set for the objective evaluation. We put more information in Appendix A.4 and Appendix A.5.

Subjective metrics.

We conduct the MOS (mean opinion score) and CMOS (comparative mean opinion score) evaluation on the test set to measure the audio naturalness via Amazon Mechanical Turk. We keep the text content and prompt speech consistent among different models to exclude other interference factors. We randomly choose 50 samples from the test set of each dataset for the subjective evaluation and each audio is listened to by at least 20 testers. We analyze the MOS in three aspects: MOS-Q (Quality: clarity, high-frequency, and original timbre reconstruction), MOS-P (Prosody: naturalness of pitch, energy, and duration), and MOS-S (Speaker similarity). We also analyze the CMOS in terms of audio quality and speech prosody. We tell the tester to focus on one corresponding aspect and ignore the other aspect when scoring. We put more information about the subjective evaluation in Appendix A.2.

2 Results of zero-shot synthesis

We compare the zero-shot synthesis performance of Mega-TTS with baseline systems, including: 1)YourTTS , a powerful zero-shot TTS model trained on 1k hours of speech dataset. We use the official code and released checkpointhttps://github.com/Edresson/YourTTS; 2) VALL-E, a large-scale zero-shot TTS model using the audio codec model to generate discrete speech codes and LLM to generate them. For VALL-E, we directly download the first 16 utterances from the VALL-E demo page. The audio samples consist of 8 samples from LibriSpeech and 8 samples from VCTKVALL-E does not release its code officially. The unofficial implementations and our implementation are deficient, which would make us difficult to fairly compare our system with VALL-E.. As shown in Table 2, Mega-TTS significantly outperforms YourTTS in terms of audio quality and speech prosody. And in terms of speaker similarity, Mega-TTS significantly outperforms YourTTS with +0.51 MOS-S on VCTK and +0.68 MOS-S on LibriSpeech, demonstrating the effectiveness of Mega-TTS in zero-shot scenarios. Besides, as shown in Table 3, Mega-TTS outperforms VALL-E in all metrics. It can be seen that Mega-TTS is able to generate more natural speeches than VALL-E, demonstrating the effectiveness of introducing intrinsic inductive biases. To further investigate the performance of disentanglement, we also visualize the distribution of the timbre and prosody representations in Appendix C.

3 Results of zero-shot speech editing

We compare the quality of generated audio samples of our Mega-TTS with SOTA speech editing baselines, including 1) EditSpeech ; 2) A3T . Since the text content of the generated speech has been edited in the speech editing evaluation, the ground truth is missing. Therefore, we only conduct the subjective evaluation. We manually define modification operations (i.e., insertion, replacement, and deletion) of the test samples. We then conduct the experiments on the VCTK dataset. We evaluate the audio quality, speech prosody, and speaker similarity for each audio sample. The results are presented Table 4. It can be seen that Mega-TTS achieves the highest perceptual quality, prosody, and speaker similarity score, which demonstrates the effectiveness of our proposed speech prompting mechanism for speech editing and the powerful in-context learning capability of Mega-TTS.

4 Results of zero-shot cross-lingual TTS

To compare Mega-TTS with the zero-shot cross-lingual TTS models VALL-E X , we directly download the utterances from the VALL-E X demo page, which consists of 6 speech pairs from LibriSpeech, EMIME, and AISHELL-3. Since YourTTS is built only for English TTS, we evaluate the performance of English TTS with Chinese samples as prompts. The results are listed in Table 5. It can be seen that Mega-TTS surpasses VALL-E X in terms of audio quality, speech prosody, and speaker similarity scores, which further demonstrates the superiority of introducing proper inductive biases to different speech attributes. For objective evaluations, we use all of the text samples in the LibriSpeech test-clean set as the target sentences and randomly select one audio from AISHELL-3 as the speech prompt for each target sentence. The results show that Mega-TTS achieves a significantly lower WER than YourTTS, demonstrating the effectiveness of our method.

5 Results of robustness evaluation

To further evaluate the robustness of the proposed model, we adopt the 50 particularly hard sentences following FastSpeech . As shown in Table 6, Tacotron and VALL-E show poor robustness on these complicated sentences. As a comparison, our Mega-TTS shows equivalent robustness to the non-autoregressive models, such as FastSpeech , without any repeat or skip issues. It can be seen that directly modeling the discrete speech tokens with LLMs like VALL-E would cause robustness issues. As a comparison, Mega-TTS not only leverages the in-context learning capability of LLMs, but also maintains good robustness by introducing the proper inductive bias to each speech component.

Conclusion

In this paper, we proposed Mega-TTS, which aims to introduce proper inductive biases into large-scale zero-shot TTS systems. We disentangle speech into different attributes (i.e., content, timbre, prosody, and phase) and model different attributes in different ways. We train Mega-TTS with 20K hours of multi-domain speech data and evaluate its performance on unseen datasets. Our experimental results on three speech synthesis tasks show that Mega-TTS outperforms state-of-the-art zero-shot TTS models regarding audio quality, speech prosody, speaker similarity, and robustness. Due to limited page space, we discuss the limitations and future works in Appendix F and the broader impacts in Appendix G.

References

Appendix A Detailed Experimental Settings

We list the model hyper-parameters of Mega-TTS in Table 7.

A.2 Details in Subjective Evaluation

We perform the audio quality, speech prosody, and speaker similarity evaluations on Amazon Mechanical Turk (MTurk). For each dataset, we randomly select 50 samples from the test set and use the TTS systems to generate the audio samples. Each audio has been listened to by at least 20 listeners. For MOS, each tester is asked to evaluate the subjective score of a sentence on a 1-5 Likert scale. For CMOS, listeners are asked to compare pairs of audio generated by systems A and B, indicating which of the two audio they prefer, and choose one of the following scores according to the degree of superiority: 0 indicating no difference, 1 indicating source slightly better, 2 indicating source mostly better and 3 indicating source completely better. For audio quality evaluation (MOS-Q and CMOS-Q), we tell listeners to “Please focus on the audio quality and ignore other factors”. For prosody evaluations (MOS-P and CMOS-P), we tell listeners to “Please focus on the prosody and style, and ignore the differences of grammar, audio quality, or other factors. ”. For speaker similarity evaluations (MOS-S), we tell listeners to “ Please focus only on the similarity of the speaker to the reference, and ignore the differences of content, grammar, prosody, audio quality, or other factors.”.

The screenshots of instructions for testers are shown in Figure 3. We paid $12 to participants hourly and totally spent about$1000 on participant compensation. We tell the participants that the data will be used in scientific research.

A.3 Details of Speaker Diarization Model

To obtain the speaker information from GigaSpeech and WenetSpeech, we use a released automatic speaker diarization model called pyannote.audiohttps://huggingface.co/pyannote/speaker-diarization, which achieves DER=11.24% on the VoxConverse dataset and DER=14.09% on the AISHELL-4 dataset. We only assign the speaker ID to the audio clip when its activation score is higher than 70% and abandon other audio clips. We also abandon the audio clips that contain multiple speakers speaking simultaneously.

A.4 Details of Speaker Similarity Model

To measure the speaker similarity, we use the WavLM model finetuned for speaker verification from https://huggingface.co/microsoft/wavlm-base-plus-sv to extract the speaker embedding. Then the cosine similarity between the synthesized speech’s speaker embedding and the ground-truth speech’s speaker embedding is calculated as the speaker similarity score. The WavLM model is pretrained on 94,000 hours of speech data and finetuned on the VoxCeleb1 dataset using an X-Vector head with an Additive Margin Softmax loss, which achieves 0.84%, 0.928%, and 1.758% EER (Equal Error Rate) on the Vox1-O, Vox1-E, and Vox1-H trial lists.

A.5 Details of ASR Model

To measure the audio quality and speech intelligibility for cross-lingual TTS systems, we evaluate the word error rate (WER) metric. We use the finetuned HuBERT-Large model to transcribe the synthesized speech into text and calculate the WER between the transcribed text and the original target text. The finetuned HuBERT-Large model from https://huggingface.co/facebook/hubert-large-ls960-ft is finetuned on 960h of Librispeech and achieves 1.5%, 3.0%, 1.9%, and 3.3% WER on the dev-clean, dev-other, test-clean, and test-other set of Librispeech.

A.6 Error Bars and Random Seeds

For the subjective evaluations, we report confidence intervals of the results of MOS tests in Table 2, Table 3, Table 4, and Table 5. For the objective evaluations, we ran the experiments 10 times with 10 different random seeds ($$) and obtained the averaged results.

Appendix B Visualizations of Mel-Spectrograms

We put more visualizations of mel-spectrograms with different random seeds in Figure 4. We can see that with different random seeds, Mega-TTS can generate diverse results that have different prosody and frequency details.

Appendix C Visualization of Representations

To validate the effectiveness of disentanglement for speech components in Section 3.1, we adopt T-SNE to visualize timbre embedding and prosody embedding for unseen speakers on the VCTK dataset. We randomly select 10 speakers and directly use the encoders proposed in Section 3.1 to extract the timbre and prosody information from their audio samples. The results are shown in Figure 5 and Figure 6. It can be seen that the timbre embeddings are ideally located according to the speaker ID. However, the prosody embeddings of different speakers have similar distributions. It shows that our proposed prosody and timbre encoders are able to disentangle the corresponding representations from the mel-spectrograms, which further ensures the effectiveness of our P-LLM.

Appendix D Hyperparameter Selection for the Information Bottleneck

In this section, we describe the details of the hyperparameter selection for the information bottleneck proposed in Section 3.1. The information bottleneck of Mega-TTS mainly contains two key hyperparameters: the channel size and the embedding size of the vector quantization (VQ) layer. When the channel size and the embedding size are too small or large, the performance of disentanglement will be poor. Therefore, we should carefully select these hyperparameters. We train the VQGAN-based TTS models with different VQ hyperparameters and evaluate their pitch distance and speaker similarity following Section 4. Differently, we use the proposed encoders to extract the timbre, content, and prosody embeddings of the test samples. Then, we randomly shuffle the timbre embedding sequence and reconstruct the mel-spectrogram with the original content, original prosody, and shuffled timbre information. We calculate the pitch distance between the ground-truth speech and the generated speech, but we calculate the speaker similarity between the shuffled ground-truth speech and the generated speech. As shown in Table 8, when the channel size is 256 and the embedding size is 2048, the VQGAN-based TTS model shows the best pitch accuracy and speaker similarity, i.e., the disentanglement performance is the best.

Appendix E Ablation Studies of Dataset Size and Model Size

In this section, we evaluate the influences of the training dataset size and model size on the zero-shot TTS task for Mega-TTS. We evaluate the pitch distance, speaker similarity, and the average absolute duration error in milliseconds on the LibriSpeech test-clean set. As shown in Table 9, when the dataset size grows, the zero-shot performance of Mega-TTS is significantly improved. Moreover, from Table 10, we can see that when the hidden size of P-LLM grows, the pitch distance significantly drops, demonstrating that the in-context learning capability of P-LLM can be greatly improved by the size of the model.

Appendix F Limitations and Future Works

Although achieving superior performance on various zero-shot speech synthesis tasks, Mega-TTS still suffers from two main limitations.

Although we use 20K hours of multi-domain data for training, our model still cannot cover everyone’s voice. Especially for some speakers with extremely heavy accents, our model cannot imitate their speaking style very well. In the future, we will further scale up the training data to 200K hours to further improve the performance of the model.

Reconstruction Robustness.

Although the reconstruction quality of the proposed VQGAN-based TTS model is satisfying on the clean dataset, it will be influenced by the background music or the extremely loud reverberation. In future work, we will explore a new model structure that is more robust against the acoustic environment noises.

Appendix G Broader Impacts

Mega-TTS improves the quality and efficiency of zero-shot speech synthesis, which makes it easier for people to synthesize personalized speeches. In most cases, people will utilize this technique to facilitate movies, games, podcasts, and other services only. However, it may carry potential risks in misuse of the model, such as spoofing voice or other deepfake-related usages. To handle this, potential solutions like building a corresponding deepfake detection model should be considered. We also plan to include restrictions in the open-source license of the Mega-TTS project to prevent the misuse of the model.