PortaSpeech: Portable and High-Quality Generative Text-to-Speech

Introduction

Recently, deep learning-based text-to-speech (TTS) has attracted a lot of attention in speech community . Among neural network-based TTS systems, some of them generate mel-spectrograms autoregressively from text and suffer from slow inference speed and robustness (word skipping and repeating) problems , while others generate mel-spectrograms in parallel with comparable quality using non-autoregressive architecture, called NAR-TTS, which enjoys fast inference and avoids robustness issues in the meanwhile. In general, modern TTS models aim to achieve the following goals:

Fast: to reduce the cost of computational resources and apply the model to real-time applications, the inference speed of TTS model should be fast.

Lightweight: to deploy the model to mobile or edge devices, the model size should be small and the runtime memory footprint should be low.

High-quality: to improve the naturalness of synthesized speech, the model should capture the details (frequency bins between two adjacent harmonics, unvoiced frames and high-frequency parts) in natural speech.

Expressive: to generate expressive and dynamic speech, the model should use powerful prosody modeling methods to accurately model the fundamental frequency and duration of speech.

Diverse: to prevent the synthesized speech from being too dull and tedious when generating long speech, the model should be able to generate diverse speech samples with different intonations given one text input sequence.

To achieve the above goals, in this work, we propose PortaSpeech, a portable and high-quality generative text-to-speech model, which generates mel-spectrograms with natural details and expressive prosody using a lightweight architecture. Specifically,

Through some preliminary experiments (see Section 4.2), we find that VAE is good at capturing the long-range semantics features (e.g., prosody), while normalizing flow is good at reconstructing the frequency bin-wise details. Based on these observations, we adopt VAE with an enhanced prior followed by a flow-based post-net as the main model architecture of PortaSpeech, which helps PortaSpeech generate high-quality and expressive results. In addition, PortaSpeech can generate diverse speech by sampling latent variables from the prior of VAE and post-net.

Through the experiments, we also find that even when the model is very small, VAE is still good at capturing the prosody, making it possible for PortaSpeech to reduce its model size using a lightweight VAE. Besides, we introduce the grouped parameter sharing mechanism to the post-net to compress its model size. By doing these, PortaSpeech can be very lightweight and fast at a small performance cost.

To model the prosody better and generate more expressive speech, we introduce a linguistic encoder with mixture alignment, which combines hard word-level alignment and soft phoneme-level alignment. Our proposed linguistic encoder also reduces the dependence on fine-grained (phoneme-level) alignment and alleviates the burden of the speech-to-text aligner.

Experiments on the LJSpeech dataset show that PortaSpeech outperforms other state-of-the-art TTS models with comparable model parameters in voice quality and prosody in terms of both subjective and objective evaluation metrics. When compressing the model size, our PortaSpeech shows only a slight performance degradation but enjoys the benefits of a much smaller number of model parameters (about 4x model size reduction) and lower memory footprints (about 3x memory reduction) compared with FastSpeech 2. The main contributions of this work are summarized as follows:

We analyze the characteristics of VAE and normalizing flow when applied to TTS and combines the advantages of VAE and normalizing flow to generate mel-spectrograms with rich details and expressive prosody.

We propose mixture alignment in the linguistic encoder, which improves the prosody and reduces the dependence on fine-grained (phoneme-level) hard alignment.

Using lightweight VAE and introducing the grouped parameter sharing mechanism to the post-net, PortaSpeech can generate high-quality speech with a small number of model parameters and small runtime memory footprints.

Background

In this section, we describe the background of TTS and the basic knowledge of VAE and normalizing flow. We also review the existing applications of VAE and normalizing flow in non-autoregressive TTS and analyze their advantages and disadvantages.

Text-to-speech (TTS) models convert input text or phoneme sequence into mel-spectrogram (e.g., Tacotron , FastSpeech ), which is then transformed to waveform using vocoder (e.g., WaveNet ), or directly generate waveform from text (e.g., FastSpeech 2s and EATS ). End-to-end text-to-speech models have gradually developed from autoregressive to non-autoregressive architecture: early autoregressive text-to-speech models generate each mel-spectrogram frame conditioned on previous ones, resulting in high inference latency and low robustness. Recently, several non-autoregressive TTS works have been proposed, which generate mel-spectrogram frames in parallel. FastSpeech and ParaNet are the first non-autoregressive TTS models, which use pre-trained autoregressive TTS teacher models to extract text-to-spectrogram alignments from the training data to bridge the length gap between text and speech for non-autoregressive student model. FastSpeech 2 introduces more variation information of speech, including pitch and energy, to alleviate the one-to-many mapping problem in TTS. While these methods need external text-to-spectrogram alignment models or tools, Glow-TTS directly searches for the most probable monotonic alignment between text and the latent representation of speech using normalizing flows and dynamic programming. In addition to improving the performance of non-autoregressive models, some works focus on lightweight and portable model designs: SpeedySpeech replaces the self-attention layers with fully convolutional blocks to reduce the computational complexity. LightSpeech leverages neural architecture search (NAS) to automatically design more lightweight models, while the training of NAS consumes huge resources. In this work, we save the model parameters by taking advantage of the characteristics of VAE and normalizing flow and introducing the grouped parameter sharing mechanism.

VAE

The VAE is a generative model in the form of $p_{\theta}(\mathbf{x},\mathbf{z})=p(\mathbf{z})p_{\theta}(\mathbf{x}|\mathbf{z})$, where $p(\mathbf{z})$ is a prior distribution over latent variables $\mathbf{z}$ and $p_{\theta}(\mathbf{x}|\mathbf{z})$ is the likelihood function that generates data $x$ given latent variables $\mathbf{z}$ which can be considered as a decoder. It is parameterized by a neural network $\theta$. Since the true posterior $p_{\theta}(\mathbf{x},\mathbf{z})$ over the latent variables of a VAE is usually analytically intractable, we approximate it with a variational distribution $q_{\phi}(\mathbf{z}|\mathbf{x})$, which can be viewed as an encoder. The parameters $\theta$ and $\phi$ can be optimized by maximizing the evidence lower bound (ELBO):

Recently, some works successfully apply VAE to TTS. One of them is BVAE-TTS , which adopts a bidirectional-inference variational autoencoder that learns hierarchical latent representations using both bottom-up and top-down paths to increase its expressiveness. Thanks to the hierarchical structure and latent modeling, BVAE-TTS can capture the dynamism and variability of ground-truth prosody. However, its generated mel-spectrograms are very blurry and over-smoothing, resulting in unnatural sounds, due to the posterior collapse and the reconstruction loss term used in BVAE-TTS, which has independency assumption of generated frequency bins given latent variables.

Normalizing Flow

Normalizing flow is a kind of generative models which has several advantages including exact log-likelihood evaluation and fully-parallel sampling. In generation, normalizing flows transform the latent variable $\mathbf{z}$ into a datapoint $\mathbf{x}$ through a composition of invertible functions $\mathbf{f}=\mathbf{f}_{1}\circ\mathbf{f}_{2}\circ\cdots\circ\mathbf{f}_{K}$ and we assume a tractable prior $p_{\boldsymbol{\theta}}(\mathbf{z})$ over latent variable $\mathbf{z}$ sampled from a simple distribution (e.g., a Gaussian distribution). In training, the log-likelihood of a datapoint $\mathbf{x}$ can be computed exactly using the change of variables rule:

where $\mathbf{h}_{0}=\mathbf{x}$, $\mathbf{h}_{i}=\mathbf{f}_{i}(\mathbf{h}_{i-1})$, $\mathbf{h}_{K}=\mathbf{z}$ and $|\det(d\mathbf{h}_{i}/d\mathbf{h}_{i-1})|$ is the Jacobian determinant. We learn the parameters of $\mathbf{f}_{1}\dots\mathbf{f}_{K}$ by maximizing Equation (1) over the training data. Given $\mathbf{g}=\mathbf{f}^{-1}$, we can now generate a sample $\hat{\mathbf{x}}$ by sampling $\mathbf{z}\sim p_{\boldsymbol{\theta}}(\mathbf{z})$ and computing $\hat{\mathbf{x}}=\mathbf{g}(\mathbf{z})$.

There are several normalizing flow-based non-autoregressive TTS methods: Flow-TTS is an early flow-based TTS method, which replaces the decoder in FastSpeech with Glow and jointly learns the alignment and mel-spectrogram generation through a single network. Then Glow-TTS is proposed, which combines the normalizing flow and dynamic programming-based monotonic alignment to enable fast, diverse and controllable speech synthesis. These methods handle the blurry mel-spectrogram problems well due to the nature of the normalizing flow. However, according to our experiments (see Section 4.2), flow-based NAR-TTS model usually requires a huge model capacity to achieve good performance, and the performance can drop notably when reducing the number of model parameters.

PortaSpeech

Considering the characteristics of VAE and normalizing flow mentioned in Section 2, to build a TTS system that can meet the goals described in Section 1, we propose PortaSpeech, which combines the advantages of VAE and normalizing flows and overcomes their deficiencies. As shown in 1(a), PortaSpeech is composed of a linguistic encoder with mixture alignment, a variational generator with enhanced prior and a flow-based post-net with the grouped parameter sharing mechanism. First, the text sequence with word-level boundary is fed into the linguistic encoder to extract the linguistic features in both phoneme and word level. Secondly, to model the expressiveness and variability of speech with lightweight architecture, we train the VAE-based variational generator to maximize the ELBO over the ground-truth mel-spectrograms conditioned on the linguistic features, whose prior distribution is modeled by a small volume-preserving normalizing flow. Finally, to refine and enhance the natural speech details in the generated mel-spectrograms, we train the post-net by maximizing the likelihood of ground-truth mel-spectrograms conditioned on both the linguistic features and the outputs of the variational generator. During inference, the text is transformed to mel-spectrograms by successively passing through the linguistic encoder, the decoder of the variational generator and the reversed flow-based post-net. We describe these designs and the training and inference procedures in detail in the following subsections. We put more details in Appendix A.

To expand the lengths of linguistic features (outputs of the linguistic encoder), previous non-autoregressive TTS models introduce a duration predictor to predict the number of frames of each phoneme (phoneme duration) and the ground-truth phoneme duration (hard alignment) is obtained by external models/tools (e.g., FastSpeech and FastSpeech 2 ) or jointly monotonic alignment training (e.g., Glow-TTS and BVAE-TTS ). However, phoneme-level hard alignment has several issues: since some of the boundaries between two phonemes are naturally uncertain It could be difficult to determine the exact boundary between two phonemes in millisecond level even for manually labeling., it is challenging for the alignment model to obtain very accurate phoneme-level boundaries, which inevitably introduces errors and noises. Further, these alignment errors and noises can affect the training of duration predictor, which hurts the prosody of the generated speech in inference. To tackle these problems, we introduce mixture alignment to the linguistic encoder, which uses soft alignment in phoneme level and keeps hard alignment in word level.

As shown in 1(b), our linguistic encoder consists of a phoneme encoder, a word encoder, a duration predictor and a word-to-phoneme attention module and detailed architecture of these modules are put in Appendix A.1. Suppose we have an input phoneme sequence together with the word boundary (for example, "HH AE1 Z | N EH1 V ER0", where "|" denotes the word boundary in phoneme sequence). First, we encode the phoneme sequence into phoneme hidden states $\mathcal{H}_{p}$. Then we apply word-level pooling on $\mathcal{H}_{p}$ to obtain the input representation of the word encoder, which averages the phoneme hidden states inside each word according to the word boundary. The word encoder then encodes the word-level hidden states into word-level hidden states and expanded them to match the length of the target mel-spectrogram (denoted as $\mathcal{H}_{w}$) using length regulator with the word-level duration. Finally, to add fine-grained linguistic information, we introduce a word-to-phoneme attention module, which takes $\mathcal{H}_{w}$ as the query and $\mathcal{H}_{p}$ as the key and the value. In addition, due to the monotonic nature of text-to-spectrogram alignment, to encourage the attention to be close to the diagonal, we add a word-level relative positional encoding embedding to both $\mathcal{H}_{p}$ and $\mathcal{H}_{w}$ before they are fed into the attention module. To predict the word-level duration, we use the duration predictor which takes $\mathcal{H}_{p}$ as input and then sums the predicted duration of the phonemes in each word as the word-level duration In training, the ground-truth word-level duration can be obtained by external forced alignment tools or autoregressive TTS models.. Our mixture alignment mechanism avoids the uncertain and noisy phoneme-level alignment extraction and duration prediction while keeping fine-grained, soft and close-to-diagonal text-to-spectrogram alignment.

2 Variational Generator with Enhanced Prior

To achieve expressive and diverse speech generation with lightweight architecture, we introduce VAE as the mel-spectrogram generator, called variational generator. However, traditional VAE uses simple distribution (e.g., Gaussian distribution) as the prior, which results in strong constraints on the posterior: optimizing with Gaussian prior pushes the posterior distribution towards the mean, limiting diversity and hurting the generative power . To enhance the prior distribution, inspired by , we introduce a small volume-preserving normalizing flow For simplicity and convenience, we use volume-preserving flow (VP-Flow), which does not need to consider the Jacobian term when calculating the data log-likelihood. We find that volume-preserving is powerful enough for modeling the prior., which transforms simple distributions (e.g., Gaussian distribution) to complex distributions through a series of K invertible mappings (a stack of WaveNet residual blocks with dilation 1). Then we take the complex distributions as the prior of the VAE. When introducing normalizing flow-based enhanced prior, the optimization objective of the mel-spectrogram generator becomes:

where $\phi$, $\theta$ and $\bar{\theta}$ denote the model parameters of VAE encoder, VAE decoder and the normalizing flow-based enhanced prior, respectively; $c$ denotes the outputs of linguistic encoder. Due to the introduction of normalizing flows, the KL term in Equation (2) no longer offers a simple closed-form solution. So we estimate the expectation w.r.t. $q_{\phi}(\mathbf{z}|\mathbf{x},c)$ via Monte-Carlo method by modifying the KL term:

As shown in 1(c), in training, the posterior distribution $N(\mu_{q},\sigma_{q})$ is encoded by the encoder of the variational generator. Then $z_{q}$ is sampled from the posterior distribution using reparameterization and is passed to the decoder of the variational generator (the right dotted line). In the meanwhile, the posterior distribution is fed into the VP-Flow to convert it to a standard normal distribution (the middle dotted line). In inference, VP-Flow converts a sample in the standard normal distribution into a sample $z_{p}$ in the prior distribution of the variational generator and we pass the $z_{p}$ to the decoder of the variational generator.

3 Flow-based Post-Net

To generate high-quality mel-spectrograms, normalizing flows have been widely proved to be effective. Unlike simple loss-based (L1 or MSE-based) or VAE-based methods that often generate blurry outputs, flow-based models can overcome the over-smoothing problem and generate more realistic outputs. To model rich details in ground-truth mel-spectrograms, we introduce a flow-based post-net with strong condition inputs to refine the outputs of the variational generator. As shown in 1(d), the architecture of the post-net adopts Glow and is conditioned on the outputs of the variational generator and the linguistic encoder. In training, the post-net transforms the mel-spectrogram samples into latent prior distribution (isotropic multivariate Gaussian) and calculates the exact log-likelihood of the data using the change of variables. In inference, we sample the latent variables from the latent prior distribution and pass them into the post-net reversely to generate the high-quality mel-spectrogram.

However, flow-based models suffer from large model footprints. Since the conditional inputs contain the text and prosody information, our post-net only focuses on modeling the details in mel-spectrograms, greatly reducing requirements for model capacity. To further reduce the model size and keep the modeling power, we introduce the grouped parameter sharing mechanism to the affine coupling layer, which shares some model parameters among different flow steps ($\mathbf{f}_{i}$, $\mathbf{f}_{i+1}$, …, $\mathbf{f}_{j}$). As shown in Figure 2, we divide all flow steps ($\mathbf{f}_{1}$, $\mathbf{f}_{2}$, …, $\mathbf{f}_{K}$) into several groups and share the model parameters of $NN$ (WaveNet-like network, see Appendix A.3) in the coupling layers among flow steps in a group. Our grouped parameter sharing mechanism is similar to the shared neural density estimator proposed in with some differences that: 1) we simplify the model by removing the flow indication embedding since the unshared conditional projection layer in different flow steps can help the model to indicate the position of the step; 2) instead of sharing the parameters among all flow steps, we generalize the sharing mechanism by sharing the parameters among flow steps in a group, making it easier to adjust the number of trainable model parameters without changing the model architecture.

4 Training and Inference

In training, the final loss of PortaSpeech consists of the following loss terms: 1) duration prediction loss $L_{dur}$: MSE between the predicted and the ground-truth word-level duration in log scale; 2) reconstruction loss of variational generator $L_{VG}$: MAE between the ground-truth mel-spectogram and that generated by the variational generator; 3) the KL-divergence of variational generator $L_{KL}$ = $\log q_{\phi}(\mathbf{z}|\mathbf{x},c)-\log p_{\bar{\theta}}(\mathbf{z}|c)$, where $\mathbf{z}\sim q_{\phi}(\mathbf{z}|\mathbf{x},c)$, according to Equation (3); and 4) the negative log-likelihood of the post-net $L_{PN}$. In inference, the linguistic encoder first encodes the text sequence, predicts the word-level duration and expand the hidden states via mixture alignment to obtain the linguistic hidden states $\mathcal{H}_{L}$. Secondly, we sample $\mathbf{z}$ from the enhanced prior, and then the decoder of the variational generator generates the coarse-grained mel-spectrograms $\bar{M}_{c}$ (the output mel-spectrograms before post-net) conditioned on the linguistic hidden states $\mathcal{H}_{L}$. Thirdly, the post-net converts randomly sampled latent into fine-grained mel-spectrograms $\bar{M}_{f}$ conditioned on $\mathcal{H}_{L}$ and $\bar{M}_{c}$. Finally, $\bar{M}_{f}$ is transformed to waveform using a pre-trained vocoder. Since we use hard word-level alignment in PortaSpeech, absolute durations for individual words can also be specified at inference time like FastSpeech . As for silences, we add a word boundary symbol as an extra special word such as "SIL" between two words in training. In this way, we can adjust the duration of silences via modifying the duration of the special word "SIL".

Experiments

We evaluate PortaSpeech on LJSpeech dataset , which contains 13100 English audio clips and corresponding text transcripts. Following FastSpeech 2 , we split LJSpeech dataset into three subsets: 12229 samples for training, 348 samples (with document title LJ003) for validation and 523 samples (with document title LJ001 and LJ002) for testing. We randomly choose 50 samples in the test set for subjective evaluation and use all testing samples for objective evaluation. We convert the text sequence to the phoneme sequence with an open-source grapheme-to-phoneme tool https://github.com/Kyubyong/g2p. We transform the raw waveform with the sampling rate 22050 into mel-spectrograms following with the frame size 1024 and the hop size 256.

Model Configuration

Our PortaSpeech consists of an encoder, a variational generator and a post-net. The encoder consists of multiple feed-forward Transformer blocks with relative position encoding following Glow-TTS . The encoder and decoder in variational generator are 2D-convolution networks. The post-net adopts the architecture of Glow . We conduct experiments on two settings with different model sizes: PortaSpeech (normal) and PortaSpeech (small). We add more detailed model configurations of these two settings in Appendix B.

Training and Evaluation

We train the PortaSpeech on 1 NVIDIA 2080Ti GPU, with batch size of 64 sentences on each GPU. We use the Adam optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.98$, $\varepsilon=10^{-9}$ and follow the same learning rate schedule in . It takes 320k steps for training until convergence. The output mel-spectrograms of our model are transformed into audio samples using HiFi-GAN https://github.com/jik876/hifi-gan trained in advance. We conduct the MOS (mean opinion score) and CMOS (comparative mean opinion score) evaluation on the test set to measure the audio quality via Amazon Mechanical Turk. We keep the text content consistent among different models to exclude other interference factors, only examining the audio quality or prosody. Each audio is listened by at least 20 testers. We analyze the MOS and CMOS in two aspects: prosody (naturalness of pitch, energy and duration) and audio quality (clarity, high-frequency and original timbre reconstruction), and score MOS-P/CMOS-P and MOS-Q/CMOS-Q corresponding to the MOS/CMOS of prosody and audio quality. We tell the tester to focus on one aspect and ignore the other aspect when scoring MOS/CMOS of this aspect. We put more information about the subjective evaluation in Appendix B.2.

2 Preliminary Analyses on VAE and Flow

In image generation tasks, VAE is good at capturing the overall image structure information (low-frequency parts) while discarding small sharp textures/details (high-frequency parts). Similarly, in mel-spectrograms, low-frequency parts correspond to the shape of harmonics, which determines the pitch and prosody of speech. Thus we can intuitively infer that VAE is good at modeling the prosody while not good at modeling the details in speech. While flow-based models can generate high-quality images at the cost of very large model size and huge computation complexity and we may infer that flow-based models can model the details in speech well with large model size.

To verify our hypothesis and explore the characteristic of VAE and flow-based models in TTS, we conduct audio quality (MOS-Q) and prosody (MOS-P) comparisons among several VAE and flow-based NAR-TTS models with different model sizes: 1) big: more than 40M model parameters; 2) middle: about 10M model parameters; and 3) small: about 5M model parameters. We keep the architecture of the encoders in three models consistent. The detailed model architecture and configurations are put in Appendix A.4. The results are shown in Table 1. From the table, we can see that 1) when reducing the model capacities, the prosody quality of flow-based models drops significantly. In contrast, that of VAE-based model only drops slightly, according to MOS-P. This phenomenon inspires us to apply VAE-based mel-spectrogram decoder (variational generator) to our lightweight TTS model. 2) Compared with flow-based models, VAE-based model has poorer audio quality upper bound according to MOS-Q, which motivates us to make up for shortcomings of VAE by introducing a flow-based post-net to refine the mel-spectrograms generated by VAE.

3 Performance

We compare the quality of generated audio samples, inference latency, model size The model parameters do not include the encoder of VAE in BVAE-TTS and PortaSpeech. and memory footprint We profile the peak GPU memory using MemReporter in pytorch_memlab (https://github.com/Stonesjtu/pytorch_memlab) and find the maximum ”active_bytes” as the peak memory during inference. of our PortaSpeech (normal and small model size) with other systems, including 1) GT, the ground truth audio; 2) GT (Mel + HiFi-GAN), where we first convert the ground truth audio into mel-spectrograms, and then convert the mel-spectrograms back to audio using HiFi-GAN; 3) Tacotron 2 ; 4) Transformer TTS ; 5) FastSpeech ; 6) FastSpeech 2 ; 7) Glow-TTS and 8) BVAE-TTS We fail in reproducing the performance of BVAE-TTS reported in the original paper, so we use hard text-to-speech alignment in their model and obtain reasonable results.. The results are shown in Table 2. We have the following observations:

For audio quality, PortaSpeech (normal) outperforms previous TTS models in both audio quality (MOS-Q) and prosody (MOS-P), and only has slight performance degradation when reducing the model size, which shows the superiority of our proposed method.

For model size and memory footprint, PortaSpeech (small) has the smallest model size and memory footprint. Compared with FastSpeech 2, PortaSpeech (small) achieves 4x model size and 3x memory footprint compression ratios.

For inference speed, PortaSpeech (small) speeds up the end-to-end speech generation by 5.5x and 45.9x compared with Tacotron 2 and TransformerTTS and achieves similar RTF with other NAR-TTS models.

Besides, we conduct some experiments on the multi-speaker dataset and draw similar conclusions (see Appendix C). We also conduct robustness evaluation on both single-speaker and multi-speaker dataset in Appendix D and find that PortaSpeech achieves comparable robustness performance with state-of-the-art NAR-TTS models.

4 Visualizations

We then visualize the mel-spectrograms generated by the above systems in Figure 3. We can see that PortaSpeech can generate mel-spectrograms with rich details in frequency bins between two adjacent harmonics, unvoiced frames and high-frequency parts, which results in natural sounds. Besides, we visualize the diverse mel-spectrograms generated by PortaSpeech in Appendix F. In conclusion, our experiments demonstrate that PortaSpeech achieves the goals described in Section 1 (fast, lightweight, high-quality, expressive and diverse).

5 Ablation Studies

We conduct ablation studies to demonstrate the effectiveness of designs in PortaSpeech, including the enhanced prior, our post-net and the mixture alignment. We put more analyses on the grouped parameter sharing mechanism in Appendix G. We conduct CMOS evaluation for these ablation studies. The results are shown in Table 3.

To demonstrate the effectiveness of enhanced normalizing flow-based prior, we compare our models with those with simple Gaussian prior as the original VAE. The results are shown in row 2 in Table 3. We can see that CMOS-P drops when removing the enhanced prior, indicating that the enhanced prior can improve the prosody. Since the prosody is mainly modeled by VAE, compared with simple Gaussian prior, the enhanced prior has weaker assumptions and restrictions on the shape of the VAE prior distribution.

Post-Net

To demonstrate the effectiveness and necessity of flow-based post-net, we compare PortaSpeech with that without the post-net and that with convolutional post-net, which is widely used in previous TTS models, such as Tacotron 2 . The results are shown in row 3 and row 4 in Table 3. From row 3, it can be seen that CMOS-Q drops significantly when removing our post-net, demonstrating that our post-net can improve the audio quality of the generated mel-spectrograms. From row 4, we can see that our flow-based post-net outperforms the commonly used convolutional post-net.

Mixture Alignment

To demonstrate the effectiveness of mixture alignment, we replace the mixture alignment in the linguistic encoder with the phoneme-level hard alignment proposed in FastSpeech . The results are shown in row 5 in Table 3. We can see that PortaSpeech with mixture alignment outperforms that with phoneme-level hard alignment in terms of both CMOS-P and CMOS-Q. These results demonstrate that 1) mixture alignment can improve the prosody, which may benefit from more accurate duration extraction and prediction; 2) mixture alignment can also improve the generated voice quality since the soft alignment helps the end-to-end model optimization. Then we calculate the average absolute duration error in word and sentence level on the test set for PortaSpeech (small) with and without mixture alignment. The results are shown in Table 4. It can be seen that the linguistic encoder with mixture alignment predicts more accurate duration, also demonstrating the effectiveness of the mixture alignment. We visualize the attention alignments generated by our linguistic encoder in Appendix A.1, showing that PortaSpeech can create reasonable alignments which is close to the diagonal.

Conclusion

In this paper, we proposed PortaSpeech, a portable and high-quality generative text-to-speech model. PortaSpeech uses a variational generator with an enhanced prior followed by a flow-based post-net with grouped parameter sharing mechanism as the main model architecture. We also proposed a new linguistic encoder with mixture alignment to improve the prosody and reduce the dependence on the hard fine-grained alignment, which combines the hard word-level and soft phoneme-level alignments. Our experimental results show that PortaSpeech outperforms other TTS models in voice quality and prosody and shows only a slight performance degradation when reducing the model size. We also conduct comprehensive ablation studies to verify the effectiveness of each component in PortaSpeech. However, to take advantage of the merits of VAE and normalizing flow, we sacrifice at the cost of more complicated model designs than previous NAR-TTS models: the overall architecture, which cascades linguistic encoder, VAE and post-net, is somewhat complicated. In the future, we will verify the effectiveness of PortaSpeech on multi-speaker and multilingual scenarios. We will also try to tap its potential on other tasks, such as voice conversion and end-to-end text-to-waveform generation.

Acknowledgments

This work was supported in part by the National Key R&D Program of China under Grant No.2020YFC0832505, National Natural Science Foundation of China under Grant No.61836002, No.62072397, Zhejiang Natural Science Foundation under Grant LR19F020006 and Baidu Scholarship Program.

References

Appendix A Details of Models

In this section, we describe details in the linguistic encoder, variational generator, post-net and the models we used in Section 4.2.

As shown in Figure 4, our linguistic encoder consists of a phoneme encoder, a word encoder, a duration predictor and a word-to-phoneme attention module. The phoneme encoder and the word encoder are both stacks of feed-forward Transformer layers with relative position encoding , as shown in 4(b). The duration predictor, as shown in 4(c), consists of two 1D-convolutional layers, each of which is followed by ReLU activation and layer normalization, and a linear layer to project the hidden states in each timestep to a scalar, which is the predicted phoneme duration. The word-level pooling averages the phoneme hidden states inside each word according to the word boundary, as shown in 4(d). The word-to-phoneme attention module is a multi-head attention with 2 heads and we apply a word-to-phoneme mapping mask to the attention weight to force each query (Q) to only attend to the phonemes belongs to the word corresponding to this query. We also add a well-designed positional encoding to the inputs of word-to-phoneme attention module: for K and V, the positional encoding is: $\frac{i}{L_{w}}E_{kv}$, where $i$ is the position of the corresponding phoneme in the word $w$; $L_{w}$ is the number of phonemes in word $w$; $E_{kv}$ is a learnable embedding; and $i\in\{0,1,...,L_{w}-1\}$. For Q, the positional encoding becomes: $\frac{j}{T_{w}}E_{q}$, where $j$ is the position of the corresponding frame in the word $w$; $T_{w}$ is the number of frames in word $w$; $E_{q}$ is another learnable embedding; and $j\in\{0,1,...,T_{w}-1\}$.

A.2 Variational Generator

As shown in Figure 5, our variational generator consists of an encoder, a decoder and a volume-preserving (VP) flow-based prior model. The encoder, as shown in 5(a), is composed of a 1D-convolution with stride 4 followed by ReLU activation and layer normalization, and a non-causal WaveNet. The decoder, as shown in 5(b), consists of a non-causal WaveNet and a 1D transposed convolution with stride 4, also followed by ReLU and layer normalization. The prior model, as shown in 5(c), is a volume-preserving normalizing flow, which is composed of a residual coupling layer (5(d)) and a channel-wise flip operation.

A.3 Post-Net

We use non-causal WaveNet as the main architecture of NN in the affine coupling layer. We introduce the number of shared groups $N_{g}$, for example, when $N_{g}=2$, NNs in flow steps ($\mathbf{f}_{1}$, $\mathbf{f}_{2}$, …, $\mathbf{f}_{K/2}$) and ($\mathbf{f}_{K/2+1}$, $\mathbf{f}_{K/2+2}$, …, $\mathbf{f}_{K}$) share the parameters separately. In inference, we can sample $z$ from $N(0,T^{2})$, where $T$ is the temperature and use $T=0.8$ by default.

A.4 Models Used in Section 4.2

We use FastSpeech as the backbone for preliminary analyses in Section 4.2. We replace the decoder of FastSpeech with flow-based decoder and VAE-based decoder to explore the characteristics of them. The flow-based decoder is mainly adopted from Glow and WaveGlow , which uses the expanded encoder outputs as the condition, as shown in 6(a). The VAE-based decoder is similar to the variational generator in our proposed PortaSpeech, except that it does not use the flow-based prior. The model hyperparameters of different model configurations are listed in Table 5.

Appendix B Detailed Experimental Settings

In this section, we describe more model configurations and details in subjective evaluation.

We list the model hyper-parameters of PortaSpeech (normal) and PortaSpeech (small) in Table 6 and total number of parameters of each module in Table 7.

B.2 Details in Subjective Evaluation

For MOS, each tester is asked to evaluate the subjective naturalness of a sentence on a 1-5 Likert scale. For CMOS, listeners are asked to compare pairs of audio generated by systems A and B and indicate which of the two audio they prefer and choose one of the following scores: 0 indicating no difference, 1 indicating small difference, 2 indicating a large difference and 3 indicating a very large difference. For audio quality evaluation (MOS-Q and CMOS-Q), we tell listeners to "focus on examining the naturalness of prosody and rhythm, and ignore the differences of audio quality (e.g., environmental noise, timbre)". For prosody evaluations (MOS-P and CMOS-P), we tell listeners to "focus on examining the naturalness of prosody and rhythm, and ignore the differences of audio quality (e.g., environmental noise, timbre)". The screenshots of instructions for testers are shown in Figure 7. We paid $8 to participants hourly and totally spent about$750 on participant compensation.

Appendix C Results on Multi-Speaker Dataset

We conduct the MOS evaluation on the multi-speaker dataset: LibriTTS. The results are shown in Table 8 (we use a pre-trained Parallel WaveGAN for LibriTTS as the vocoder). We can draw similar conclusions as that on LJSpeech that PortaSpeech can achieve good prosody and audio quality in terms of MOS-P and MOS-Q, even in more complicated (multi-speaker) scenarios.

Appendix D Robustness Evaluation

We conduct the robustness evaluation on LJSpeech and LibriTTS datasets. We select 50 sentences that are particularly hard for TTS systems following FastSpeech . The results are shown in Tables 9 and 10. We can see that PortaSpeech achieves comparable robustness performance with state-of-the-art non-autoregressive TTS models.

Appendix E Visualization of Attention Weights

We put some word-to-phoneme attention visualizations in Figure 8. We can see that PortaSpeech can create reasonable phoneme-to-spectrogram alignments which are close to the diagonal, which helps the end-to-end training.

Appendix F More Visualizations of Mel-Spectrograms

We put more visualizations of mel-spectrograms with different sampling temperatures of post-net and different random seeds on PortaSpeech (normal) in Figure 9 and Figure 10. We have several observations: 1) From Figure 9, we can see that when $T=0.8$, our model can generate natural sound perceptually with reasonable details in mel-spectrograms. 2) From Figure 10, we can see that with different random seeds, PortaSpeech can generate diverse results, which have different prosody and mel-spectrogram details.

Appendix G Analyses on the Grouped Parameter Sharing Mechanism

In this section, we conduct the subjective evaluation to compare the audio quality with different numbers of shared groups ($N_{g}$) for PortaSpeech (normal) and PortaSpeech (small). The results are shown in Table 11. It can be seen that the audio quality drops significantly when sharing parameters among all flow steps, demonstrating the effectiveness of our grouped parameter sharing mechanism.

Appendix H Potential Negative Societal Impacts

PortaSpeech lowers the requirements for speech synthesis service deployment (memory and CPU performance) and synthesizes high-quality speech voice, which may cause unemployment for people with related occupations such as broadcaster and radio host. In addition, there is the potential for harm from non-consensual voice cloning or the generation of fake media and the voices of the speakers in the recordings might be overused than they expect.