Sequence-to-Sequence Acoustic Modeling for Voice Conversion

I Introduction

Voice conversion aims to modify the speech signal of a source speaker to make it sound like being uttered by a target speaker, while keeping the linguistic contents unchanged . The potential applications of this technique include entertainment, personalized text-to-speech, and so on .

Building statistical acoustic models for feature mapping is a popular approach to voice conversion nowadays. At the training stage of the conventional voice conversion pipeline, acoustic features are first extracted from the waveforms of source and target utterances. Then, the features of parallel utterances are aligned frame by frame using alignment algorithms, such as dynamic time wrapping (DTW) . Next, an acoustic model for conversion is trained using the acoustic features of paired source-target frames. The acoustic model can be a joint density Gaussian mixture model (JD-GMM) or a deep neural network (DNN) , both of which are universal function approximators . At the conversion stage, a mapping function is derived from the built acoustic model that converts the acoustic features of source speaker into those of target speaker. Finally, waveforms are recovered from the converted acoustic features using a vocoder.

This conventional pipeline for voice conversion has its limitations. First, most previous work focused on the conversion of spectral features and simply adjusted $F_{0}$ trajectories linearly in the logarithm domain . Besides, the durations of converted utterances were kept the same as the ones of source utterances since the acoustic models were built on a frame-by-frame basis. However, the production of human speech is a highly dynamic process and the frame-by-frame assumption constrains the modeling ability of mapping functions .

This paper proposes an acoustic modeling method for voice conversion based on the sequence-to-sequence neural network framework . A Sequence-to-sequence ConvErsion NeTwork (SCENT) is designed to directly describe the conditional probabilities of target acoustic feature sequences given source ones without explicit frame-to-frame alignment. The SCENT model follows the widely-used architecture of encoder-decoder with attention . The encoder network first transforms the input feature sequences into hidden representations which are suitable for the decoder to deal with. At each decoder time step, the attention module selects encoder outputs softly by attention probabilities and produces a context vector. Then, the decoder predicts output acoustic features frame by frame using context vectors. Furthermore, a post-filtering network is designed to enhance the accuracy of the converted acoustic features. Finally, a speaker-dependent WaveNet is utilized to recover time-domain waveforms from the predicted sequences of acoustic features.

In our proposed method, Mel-scale spectrograms are adopted as acoustic features, which do not rely on the source-filter assumption of speech production. Therefore, $F_{0}$ and spectral features are converted jointly in a single model. Additional bottleneck features derived using an automatic speech recognition (ASR) model are appended to the source Mel-spectrograms, which are expected to improve the pronunciation correctness of the converted speech. Attention module learns the soft alignments between the pairs of source-target feature sequence implicitly. Facilitated by attention module, our proposed method is capable of predicting target acoustic sequences with durations different from source ones at conversion stage.

Experimental results show that our proposed method achieved better objective and subjective performance than the GMM-based and DNN-based baseline systems. This proposed method also outperformed our previous work which achieved the top rank in Voice Conversion Challenge 2018 . It is worth noting that our proposed method can achieve appropriate duration conversion, which contributes to higher similarity and is difficult in conventional methods. Ablation studies were further conducted and the results confirmed the effectiveness of several key components in our proposed method.

In this paper, we focus on one-to-one voice conversion, i.e., one model is trained for each speaker pair. It should be noticed that our proposed method can also be adapted to other cases rather than one-to-one conversion. For example, the proposed method can be extended to multiple speaker pairs by conditioning on codes of speaker identities, which can be obtained from the outputs of a speaker encoder .

The rest of this article is organized as follows. Section II reviews the related work on seq2seq modeling, voice cloning and WaveNet vocoders. Section III introduces our proposed method for voice conversion. Details and results of experiments are presented in Section IV. The article is concluded in Section V.

II Related Work

Text-to-speech (TTS) methods based on seq2seq learning have emerged recently and attracted much attention . Our work is inspired by the success of applying seq2seq models to TTS. However, voice conversion is different from TTS in several aspects. First, the inputs of a voice conversion model are frame-level acoustic features rather than phone-level or character-level linguistic features. Typically, linguistic features are discrete, while acoustic features are continuous. In addition to linguistic information, acoustic features also contain speaker identity information which should be processed during voice conversion. Second, the input-output alignment in voice conversion task is different from that in TTS. Speech generation in TTS is a decompressing process and the alignment between text and acoustic frames is usually a one-to-many mapping. While the alignment can be either one-to-many or many-to-one in voice conversion, depending on the characteristics of speaker pairs and the dynamic characteristics of acoustic sequences. Third, the training data available for voice conversion is typically smaller than that for TTS.

II-B Relationship with voice cloning

Voice cloning is a task that learns the voice of unseen speakers from a few speech samples for text-to-speech synthesis. Unlike voice conversion, voice cloning takes text as model input. Arik et al. evaluated two techniques of voice cloning, i.e., speaker adaptation and speaker encoding, based on Deep Voice 3 . Jia et al. proposed a transfer learning method for voice cloning. A speaker-discriminative embedding network was first trained to achieve a speaker verification task. Then, the built network was transferred to a conditional Tacotron model to generate speech for a variety of speakers. Nachmani et al. extended the Voice Loop model to fit new voices by incorporating a fitting network. Instead of using text as model input in these studies, we utilize a separate ASR model for extracting linguistic-related features and the input of our model is only the speech of source speakers. Also, instead of generating speech of unseen speakers, we focus on voice conversion for one pair of speakers. It should be noticed that the techniques developed for voice cloning are potentially useful for extending our proposed method from one-to-one conversion to many-to-many conversion, which will be a part of our future study.

II-C Sequence-to-sequence learning for voice conversion

To the best of our knowledge, Ramos made the first attempt to convert spectral features using a sequence-to-sequence model with attention. However, as stated in Section 5.5 of Ramos’s thesis , the model was not capable of using its own predictions to generate a real valued output prediction. Kaneko et al. proposed a CNN-based seq2seq spectral conversion method. Because of the lack of attention module in their method, the DTW algorithm was still utilized in order to obtain frame-level aligned feature sequences during training data preparation. Miyoshi et al. proposed a method of mapping context posterior probabilities using seq2seq models. In their method, an RNN-based encoder-decoder converted the source posterior probability sequence to the target one for each phone, and the phone durations of natural target speech were necessary at conversion stage.

Our work is most similar to Ramos’s one , where an utterance-level seq2seq with attention model is built for acoustic feature conversion. Different from previous methods, Mel-spectrograms are adopted as acoustic features in our method. Thus, $F_{0}$ and spectral features are transformed jointly. Our method has the ability of modeling pairs of input and output utterance without dependency on DTW alignment. During conversion, the durations of generated target acoustic sequences are determined automatically and the probability of completion is predicted at each decoder time step.

II-D Voice conversion using WaveNet

WaveNet , as a neural network-based waveform generation model, has been successfully applied to TTS and voice conversion areas . Studies have shown that WaveNet vocoders outperformed traditional vocoders such as WORLD and STRAIGHT in terms of the quality of reconstructed speech . Voice conversion methods using WaveNet models have also been studied in recent years. Kobayashi et al. proposed a GMM-based voice conversion method with WaveNet-based waveform generation. Liu et al. proposed building WaveNet vocoders for voice conversion with limited data by model adaptation. Directly mapping source acoustic features into target speaker’s waveforms using WaveNet has also been proposed .

In this paper, WaveNet vocoders are used to reconstruct the waveforms of target speakers. WaveNet vocoders accept Mel-spectrograms as input conditions and are trained in a speaker-dependent way without using the adaptation technique described in .

III Proposed Method

Figure 1 shows the diagram of our proposed method when converting an input utterance. The conversion process can be divided into two main stages. One is a Seq2seq ConvErsion NeTwork (SCENT) for acoustic feature prediction, the other is a WaveNet neural vocoder for waveform generation. Mel-spectrograms are adopted as acoustic features in this paper. Bottleneck features extracted by an ASR model from source speech are concatenated with acoustic features to form the input sequences of the SCENT model. The SCENT model converts input sequence into Mel-spectrograms of the target speaker. Then, the target speaker’s speech is synthesized by passing the predicted Mel-spectrograms through the WaveNet vocoder.

III-B Feature extraction

Mel-spectrograms are computed through a short-time Fourier transform (STFT) on waveforms. The STFT magnitudes are transformed to Mel-frequency scale using Mel-filterbanks followed by a logarithmic dynamic range compression. In order to extract bottleneck features, a recurrent neural network (RNN) based ASR model is trained on a separate speech recognition dataset. For each input frame, bottleneck features, i.e., the activations of the last hidden layer before the softmax output layer of the ASR model, are extracted. Such bottleneck features can provide additional linguistic-related descriptions which are expected to benefit the conversion process. It should be noticed that these bottleneck features are still automatically extracted from the acoustic signals of source utterances and no text transcriptions are necessary. The Mel-spectrograms and bottleneck features at each frame are concatenated to form the input sequence $\bm{x}=[\bm{x}_{1},\dots,\bm{x}_{T_{x}}]$ of the SCENT model, where $T_{x}$ is the frame number of source speech.

III-C Structure of SCENT

A SCENT model contains an encoder-decoder with attention network which predicts acoustic feature in an uni-directional left-to-right way, and a bi-directional post-filtering network which refines the generation results. Figure 2 shows the network structure of a SCENT model.

Let $\bm{y}=[\bm{y}_{1},\dots,\bm{y}_{T_{y}}]$ denote the output Mel-spectrogram sequence of the encoder-decoder network, where $T_{y}$ is the frame number of target speech. The encoder-decoder network models the mapping relationship between input and output feature sequences using conditional distributions of each output frame $\bm{y}_{t}$ given previous output frames $\bm{y}_{<t}=[\bm{y}_{1},\dots,\bm{y}_{t-1}]$ and the input $\bm{x}$ as

where $W_{enc}$ and $W_{dec}$ are parameters of the encoder-decoder network. As shown in Figure 2, the encoder transforms the concatenated Mel-spectrograms and bottleneck features of source speech into a high-level and abstract representation $\bm{h}=[\bm{h}_{1},\dots,\bm{h}_{T_{h}}]$ as

$T_{h}$ is the frame number of hidden representations and $T_{h}<T_{x}$ because of the pyramid structure of encoder. The decoder with attention mechanism utilizes $\bm{h}$ and produces a probability distribution over output frames as

The generation process of the decoder network is uni-directional. In order to make use the bi-directional context information, a post-filtering network (i.e., PostNet) is further employed to enhance the accuracy of prediction. Let $\bm{z}=[\bm{z}_{1},\dots,\bm{z}_{T_{z}}]$ represent the PostNet output sequence, which is the final prediction of the SCENT model. In this paper, the frame rates of decoder outputs and PostNet outputs are the same, i.e. $T_{z}=T_{y}$. The distribution of feature sequence $\bm{z}$ given the output of the encoder-decoder network $\bm{y}$ is modeled as

where $W_{pos}$ denotes the parameters of the PostNet.

Next, we will describe each part of SCENT in details.

The encoder network is constructed based on the pyramid bidirectional LSTM architecture , which processes the sequence with lower time resolution at higher layers. In a conventional deep bidirectional LSTM (BLSTM) architecture, the output at the $n$-th time step of the $j$-th layer is computed as

In a pyramid BLSTM (pBLSTM), the outputs at consecutive steps of a lower layer are concatenated and fed into the next layer to decrease the sampling rate of input sequence. The general calculation of pBLSTM hidden units can be formulated as

where $M$ is ratio of downsampling. The technique of layer normalization is applied to the encoder LSTM cells. Then, a location code $\bm{l}_{n}=[l_{n}(0),\dots,l_{n}(d-1)]^{\top}$ is added to the top output layer of pBLSTMs to form the hidden representation $\bm{h}$. Let $d$ be the dimension of each $\bm{h}_{n}$. The location code is composed of sine and cosine functions of different frequencies as

where $n$ is the time step in sequence $\bm{h}$ and $i\in[0,\dots,\left.d\middle/2\right.-1]$ is the dimension index. The base 10000 in Eq. (7) follows the configuration in the original paper which proposed the location code. This location code is useful since it gives the model explicit information of which portion of the sequence is currently processed. The effectiveness of the location code will be demonstrated by ablation tests in our experiments.

The pyramid structure of our encoder network results in shorter hidden representation than original input sequence. For the voice conversion task, we expect that the encoder network should exclude speaker-dependent information of the source speech and extract hidden representation $\bm{h}$ which is high-level and linguistic-related. Because one phone usually corresponds to tens of acoustic frames, it is reasonable to derive hidden representation with lower sampling rate than the frame-level input sequence. Furthermore, hidden representation with lower sampling rate makes the attention module easier to converge, since this leads to fewer encoding states for attention calculation at each decoding step. This pyramid structure also reduces the computational complexity by shortening the length of $\bm{h}$ for attention calculation and speeds up training and inference significantly.

III-C2 Decoder with attention mechanism

The decoder is an auto-regressive RNN which predicts the output acoustic features from the hidden representation $\bm{h}$. Non-overlapping $r$ frames are predicted at each decoder step. This trick divides the total decoding steps by $r$, which further reduces training and inference time . In Figure 2, the decoder is illustrated with $r=1$ for clarity. The prediction of previous time step $\bm{y}_{t-1}$ is first passed through a pre-processing network (i.e., PreNet), which is a two-layer MLP with ReLU activation and dropout in our implementation. The MLP outputs are sent into an LSTM layer with attention mechanism. A context vector $\bm{c}$ is calculated at each decoder step using attention probabilities as

where $\bm{\alpha}_{t}=[\alpha_{t}^{1},...,\alpha_{t}^{T_{h}}]$ are attention probabilities, $t$ is decoder time step, and $n$ is the index of encoder outputs.

where $\bm{v}$, $\bm{b}$, $W$ and $U$ are trainable parameters of the model. As we can see from Eq. (10), the calculation of the hybrid attention takes two parts into consideration. The first part of Eq. (10) measures the relationship between the query vector and different entries of encoder outputs. The second part of Eq. (10) is computed based on the alignment of previous decoder step $\bm{\alpha}_{t-1}$ and provides a constraint on current attention probabilities. The convolution matrix is employed to filter $\bm{\alpha}_{t-1}$ for extracting useful features as shown in Eq. (9). The features are further integrated into the calculation of attention scores as shown in Eq. (10).

Furthermore, the forward attention method proposed in our previous work is adopted to stabilize the attention alignment and speed up the convergence of attention alignment. In the forward attention method, the attention probability $\alpha_{t}^{n}$ is calculated as

The motivation of forward attention is to follow the monotonic nature of alignments in human speech generation . Therefore, a forward variable which only takes the monotonic alignment paths into consideration is designed. This forward variable is derived from the original attention probabilities $\hat{e}_{t}^{n}$ and it can be computed recursively as Eq. (12). Then, the normalized forward variables $\alpha_{t}^{n}$ are used to replace original attention probabilities $\hat{e}_{t}^{n}$ for summarizing the encoder outputs as shown in Eq. (8). In addition, a location code is also added to the auto-regressive input of the decoder at each time step.

The context vector $\bm{c}$ and query vector $\bm{q}$ are concatenated and fed into a stack of two-layer decoding LSTMs. The concatenation of $\bm{c}$, $\bm{q}$ and the outputs of decoding LSTMs are linearly projected to produce the Mel-spectrogram output of the decoder network. In parallel, the concatenation of $\bm{c}$ and $\bm{q}$ are linearly projected to a scalar and passed through a sigmoid activation to predict the completion probability $p_{end}$, which indicates whether the converted sequence reaches the last frame.

III-C3 Post-filtering network

The PostNet refines the Mel-spectrograms predicted by the decoder using bi-directional context information. The PostNet is a convolutional neural network (CNN) with a residual connection from network input to the final output. The first layer of the PostNet is composed of 1-D convolution filter banks in order to extract rich context information. The outputs of the convolution banks are stacked together and further passed through a two-layer 1-D CNN. The outputs of the final layer are added to the input Mel-spectrograms to produce the final results.

III-D Loss function of SCENT

We train the SCENT model by multi-task learning and the total loss is the weighted sum of three sub-losses as

where $w_{dec}$, $w_{post}$ and $w_{end}$ are the weights of the three components. $L_{dec}$ and $L_{post}$ denote the losses of Mel-spectrogram prediction given by the decoder and the PostNet respectively. $L_{end}$ is a binary cross-entropy loss for evaluating the predicted completion probabilities.

Two types of criteria are investigated for $L_{dec}$. One is the minimum square error (MSE) between the predicted and ground truth acoustic features. The other is the maximum likelihood (ML) criterion based on Gaussian mixture model (GMM). For GMM-ML, the network outputs are adopted to parameterize a GMM following the framework of mixture density networks (MDN) .

More specifically, the likelihood function in GMM-ML takes the form of a GMM as

Then, the GMM parameters in Eq. (16) can be derived from the vector $\bm{o}(\bm{x},W_{enc},W_{dec})$ as

where $\bm{\sigma}_{i}(\bm{x})$ is a vector composed of the diagonal elements of $\bm{\Sigma}_{i}(\bm{x})$. For GMM-ML, $L_{dec}$ is defined as the negative log-likelihood (NLL) function, i.e.,

Under both MSE and GMM-ML criteria, natural acoustic histories of target speech are sent into the decoder at training time. The MSE criterion is actually a special case of GMM-ML which uses single mixture with fixed unit variance and predicted mean vector . Theoretically, GMM-ML is more flexible since it models more general probability distributions and the MSE criterion usually leads to over-smoothed prediction because of the averaging effect .

When applying the GMM-ML criterion to $L_{dec}$, the mean vector of the component with maximum prior probability is used to generate the output sample at both training and testing stages. At training time, the gradients from the PostNet are only back-propagated through the sampled mean vectors given by the decoder output layer.

Only the MSE criterion is applied to $L_{post}$ in our implementation. For calculating $L_{end}$, only the last decoder step of a natural target sequence is labelled as 1 (i.e., completed) and the rest steps are labelled as 0 (i.e., incompleted).

III-E WaveNet-based vocoder

As shown in Figure 1, a WaveNet-based vocoder is adopted to reconstruct time-domain waveforms given the predicted Mel-spectrogram features.

In our WaveNet model, the Mel-spectrogram features are first passed through a ConditionNet consisting of stack of dilated 1-D convolution layers with parametric ReLU activation (PReLU) . The outputs of ConditionNet are upsampled to be consistent with the sampling rate of waveforms by simply repeating. Then, the sequence of condition vectors are fed into each dilated convolution block of the WaveNet to control the waveform generation. Our WaveNet model is trained only using the target speech data for building the SCENT model and the adaptation technique is not used in this paper.

IV Experiments

Two datasets were used in our experiments. The first one contained 1060 parallel Mandarin Chinese utterances from one male speaker (about 53 mins) and one female speaker (about 72 mins). This dataset was separated into a training set with 1000 utterances, a validation set with 30 utterances and a test set with 30 utterances. For the second dataset, speech data of one male (rms, about 62 mins) and one female (slt, about 52 mins) from the CMU ARCTIC database was adopted. This dataset contained 1132 parallel English utterances, which were separated into a training set with 1000 utterances, a validation set with 66 utterances and a test set with 66 utterances. Our analytical experiments in Section IV-B and Section IV-D only adopted the Mandarin dataset, and the main objective and subjective evaluations in Section IV-C adopted both datasets.

The recordings of both dataset were sampled at 16kHz. 80-dimensional Mel-scale spectrograms were extracted every 10 ms with Hann windowing of 50 ms frame length and 1024-point Fourier transform. 512-dimensional bottleneck features were extracted using an ASR model every 40 ms and were then upsampled by repeating to match the frame rate of Mel-spectrograms.

The speaker-independent ASR model was trained using internal datasets of iFLYTEK company, which contained recordings of about 10,000 hours for Mandarin and recordings of about 3,000 hours for English. Our ASR model was an LSTM-HMM-based one. The LSTM was bidirectional with 6 hidden layers and 1024 units in each direction. The classification targets of the LSTM model were clustered triphones, i.e., senones. For the Mandarin dataset, the phoneme set included 26 initials and 140 tonal finals. We evaluated the performance of the ASR model on the parallel dataset for voice conversion. The frame classification accuracies for the female and male speakers were 72.3% and 78.4% respectively. For the English dataset, there were 62 phonemes and the frame classification accuracies for the female and male speakers were 76.4% and 75.9% respectively.

The details of our model configurations are listed in TABLE I. In our implementation, two frames were predicted at one decoding step (i.e., $r=2$) and only the last frame was fed back into the PreNet for the generation at next step. In the loss function for training the SCENT model, $w_{dec}$ was heuristically set as 1.0 or 0.01 if MSE or GMM-ML training criterion was adopted for $L_{dec}$. $w_{post}$ and $w_{end}$ were heuristically set as 1.0 and 0.005 respectively. Zoneout with probability of 0.2 were used at LSTM layers for regularization. Residual connections were adopted for the LSTM layers of encoder and decoder to speed up model convergence. We used Adam optimizer with learning rate of $10^{-3}$ for the first 20 epochs. After 50 epochs, the learning rate was exponentially decayed by 0.95 for each epoch. $L_{2}$ regularization with weight $10^{-6}$ was also applied. The batch size was 4. For WaveNet training, the $\mu$-law companded waveforms were quantized into 10 bits, i.e., 1024 levels. A speaker-dependent WaveNet vocoder was trained using each speaker’s waveforms with random initialization and a learning rate of $10^{-4}$ until the loss converge.

Three kind of baseline methods were adopted for comparison in our experiments. 41-dimensional Mel-cepstral coefficients (MCCs), 1-dimensional fundamental frequency ($F_{0}$) and 5-dimensional band aperiodicities (BAPs) were extracted every 5 ms by STRAIGHT as acoustic features in our baseline systems. The descriptions of these methods are as followsSamples of audio are available at https://jxzhanggg.github.io/Seq2SeqVC..

JD-GMM: Gaussian mixture models with full-covariance matrices were utilized for modeling the joint spectral feature vectors of source and target speakers. For each speaker, static and delta spectral features were used. The number of mixtures $m$ was tuned on validation set with $m\in$. Maximum likelihood parameters generation (MLPG) with global variance (GV) enhancement were used for spectral parameter generation. $F_{0}$ was converted by Gaussian normalization in the logarithm domain . BAPs were not converted but directly copied from the source, since previous research showed that converting aperiodic component did not make a statistically significant difference to the quality of converted speech . Waveforms were reconstructed by STRAIGHT vocoder from the converted acoustic features.

DNN: The DNN-based voice conversion models were implemented based on Merlin toolkit . The static, delta and acceleration components of MCCs, $F_{0}$ and BAPs were transformed jointly using a DNN. In addition to use the acoustic features of the source speaker as model input, we also concatenated the input acoustic features with the bottleneck features used in our proposed method. This approach was named bn-DNN in the rest of this paper. The ReLU activation function was used at DNN hidden units. A grid search using validation set was adopted in order to pick up the optimal depth $d$ and width $w$ of the DNN with $d\in$ and $w\in$. MLPG and GV techniques were used for acoustic parameter generation. Waveform was reconstructed by STRAIGHT vocoder from the converted acoustic features.

VCC2018: This baseline method followed the framework of our previous work , which achieved the top rank on naturalness and similarity in Voice Conversion Challenge 2018. A speaker-dependent acoustic feature predictor was trained by adapting a pre-trained speaker-independent model using the data of the target speaker. The predictor was an LSTM model which predicted MCCs, $F_{0}$ and BAPs of the target speaker from bottleneck features frame-by-frame. At the training stage, bottleneck features were extracted from the target speaker as model inputs. At the conversion stage, bottleneck features were obtained from the speech of the source speaker and were sent into the acoustic feature predictor of the target speaker for conversion. In this method, a speaker-dependent WaveNet vocoder conditioned on MCCs, $F_{0}$ and BAPs features was built for waveform reconstruction.

IV-B Comparison between different decoder loss functions

As introduced in Section III-D, either MSE or GMM-ML criterion was applied to define the loss function $L_{dec}$ of the decoder output in our implementation. We evaluated the objective performance of these loss functions by experiments on both female-to-male and male-to-female conversions using the Mandarin dataset.

The Mel-cepstral distortion (MCD) and root mean square error (RMSE) of $F_{0}$ on validation set were adopted as metrics. Because Mel-spectrograms were adopted as acoustic features, it’s not straightforward to extract $F_{0}$ and MCCs features from the converted acoustic features. Therefore, $F_{0}$ and 25-dimensional MCCs features were extracted by STRAIGHT from the reconstructed waveforms for evaluation. Then the extracted features were aligned to those of the reference utterances in the validation set in order to compute MCD and $F_{0}$ RMSE values. The $F_{0}$ RMSE was calculated only using the frames which were both voiced in the converted and reference utterances.

TABLE II summarizes the objective evaluation results on validation set. From the table, we can see that the model using the GMM-ML criterion with 2 mixture components achieves the best performance on validation set among all settings except the MCD of male-to-female conversion. A further examination shows that using the GMM-ML criterion with mixture components more than 2 may lead to the instability of attention alignment. Some cases of attention failures, such as getting stuck in one frame, can be observed for MX6 and MX8. We tried to re-optimize the weighting factors in Eq. (15) for the MX6 and the MX8. The experimental results showed that changing the coefficients for models with more mixtures could slightly improve the alignment quality while the overall performances of the models were still worse than the MX2 model. One possible reason is that larger mixture numbers may increase the number of parameters and the difficulty of model training. Thus, the GMM-ML criterion with 2 mixtures was adopted for $L_{dec}$ in following experiments.

The SCENT network models pairs of source and target utterance directly. During training, alignments of utterance pairs are learned by attention module implicitly. An example of the alignment between an utterance pair using the SCENT model is shown in Figure 3, where each column denotes the attention probabilities corresponding to different encoder states for one decoder step. The DTW algorithm was also conducted based on the input and output Mel-spectrogram sequences and the resulting path was plotted as the red dashed line for comparison. From this figure, we can see that these two alignments matched well. Comparing with the DTW path which denotes hard and deterministic alignment, the attention alignment is soft and changes smoothly along consecutive decoder time steps.

IV-C Comparison between baseline and proposed methods

Objective evaluations were first carried out to compare the MCD and $F_{0}$ RMSE performance of our proposed method and the baseline methods introduced in Section IV-A, including JD-GMM, DNN, bn-DNN and VCC2018. In order to compensate the duration differences between source and target speakers, we also tried to linearly interpolate the source feature sequences before sending them into the conversion models according to the average ratio between the training set durations of the two speakers. We only interpolated the static part of the source features and the dynamic features were recalculated based on the interpolated static features. This led to four additional methods, named i-JD-GMM, i-DNN, i-bn-DNN and i-VCC2018, in our evaluations. The MCDs and $F_{0}$ RMSEs were calculated following the way introduced in Section IV-B. For fair comparison, $F_{0}$ and MCCs were re-extracted by STRAIGHT from the converted waveforms for all methods when computing MCDs and $F_{0}$ RMSEs.

The proposed and baseline methods were evaluated on both the Mandarin dataset and the English CMU ARCTIC dataset. When using the English CMU ARCTIC dataset, the same procedure of tuning the decoder output layer as described in Section IV-B was conducted and the GMM output layer with 2 mixtures was also chosen.

TABLE III shows the objective evaluation results of baseline and proposed methods on test set of the Mandarin dataset. We can see that the MCDs and $F_{0}$ RMSEs of baseline methods with interpolation were close to or slightly worse than those without interpolation. Appending bottleneck features as inputs was beneficial for improving the objective performance of the DNN-based method. Our proposed method outperformed all baseline methods, which obtained the lowest MCD and $F_{0}$ RMSE.

TABLE IV shows the results evaluated on the English CMU ARCTIC dataset. The proposed method achieved best performance on $F_{0}$ RMSE, while its performance on MCD was not as good as some baseline methods. Considering that the MCD measurement may be inconsistent with human perception , some subjective evaluations were further conducted and will be introduced later.

One advantage of our proposed method is that it can convert the duration of source speech using an unified acoustic model. In order to investigate the performance of duration conversion, the scatter diagrams of test utterance durations are drawn in Figure 5 and Figure 5 for female-to-male and male-to-female conversions using the Mandarin dataset. For each test utterance, the durations of speech converted using different baseline methods were the same, i.e., the duration of the source speech. For the baseline methods with source feature interpolation, the same global interpolation ratio was shared by all baseline methods. Therefore, “i-Baseline” and “Baseline” in these two figures stand for all baseline methods with and without interpolation respectively.

From these figures, we can see that the male speaker had higher speaking rate and shorter utterance durations than the female speaker in the Mandarin dataset. The simple linear interpolation made the length of converted speech closer to the target.

Furthermore, the average absolute differences between the durations of the converted and target utterances (DDUR) are calculated using both Mandarin and English datasets and are presented in TABLE V. Results show that our proposed method can generate speech with lower duration errors than the baseline methods without duration modification or with global speaking rate compensation.

Figure 6 plots the $F_{0}$ contours and spectrograms of one test utterance converted using different methods and the natural target reference in the Mandarin dataset. From this figure, we can see that our proposed method can generate speech with more similar $F_{0}$ contours to the natural reference than the other two baseline methods. Furthermore, our proposed method can also modify the duration of source speech towards the natural reference appropriately as shown in this figure.

IV-C2 Subjective evaluation

Subjective evaluations were conducted to compare the performance of our proposed method with the baseline methods in terms of the naturalness and similarity of converted speech. In this evaluation, twenty utterances in the test set were randomly selected and converted using our proposed method and three baseline methods, including i-JD-GMM, i-bn-DNN, and i-VCC2018.

For the experiments conducted on the Mandarin dataset, ten native listeners participated in the evaluation. For the experiments conducted on the English CMU ARCTIC dataset, evaluations were conducted on the Amazon Mechanical Turk https://www.mturk.com (AMT), a platform designed to facilitate crowdsourcing. At least twenty native English listeners took part in the evaluation. In both evaluations, the listeners were asked to use headphones and the samples were shown to them in random order. The listeners were asked to give a 5-scale opinion score (5: excellent, 4: good, 3: fair, 2: poor, 1: bad) on both similarity and naturalness for each converted utterance.

The results of the subjective evaluations are presented in TABLE VI. From the table, we can see that the i-bn-DNN method achieved similar naturalness and similarity to the i-JD-GMM method. This is consistent with previous studies on DNN-based voice conversion methods . It should be noticed that the i-bn-DNN method accepted additional bottleneck features as inputs, which may benefit the performance of this method. Compared with the i-bn-DNN method, the i-VCC2018 method did not use acoustic features as inputs. However, this method achieved the best performance among the three baseline methods, especially on the naturalness of converted speech. One important reason is that the i-VCC2018 method adopted WaveNet vocoder instead of conventional STRAIGHT vocoder to reconstruct speech waveforms from the converted acoustic features.

Our proposed method outperformed the i-VCC2018 method on both naturalness and similarity, also on both Mandarin and English datasets. These experimental results proved the effectiveness of our proposed method and the improvement brought by our proposed method was not limited to a specific language. One possible reason is that at the conversion stage of the i-VCC2018 method, bottleneck features extracted from source speech were fed to the acoustic predictor. While the model was trained with the bottleneck features of the target speaker as inputs . This inconsistency may degrade the similarity of converted speech. Another reason can be attributed to the duration conversion ability of our proposed method as introduced in the objective evaluations. Therefore, the prosody similarity and naturalness of our proposed method were better than simply adjusting speaking rate globally.

IV-D Ablation tests

In order to further analyze the effectiveness of some key components in our model, ablation tests on model inputs, attention module and location code were conducted. In this subsection, only the Mandarin dataset was adopted for evaluation.

In order to investigate the necessity of using Mel-spectrograms and bottleneck features, we removed each one of them and built SCENT models utilizing only source bottleneck features or Mel-spectrograms as inputs respectively. Objective evaluation results of MCD and $F_{0}$ RMSE on test set are presented in TABLE VII.

From this table, we can see that Mel-spectrograms are beneficial for the model to achieve more accurate prediction of acoustic features. It also can be found that removing bottleneck features led to higher $F_{0}$ RMSE and MCD on test set, and its degradation on $F_{0}$ RMSE was more serious than removing Mel-spectrograms. Listening to the converted audio samples without using bottleneck features, we found they suffered from serious mispronunciation problem. The bottleneck features extracted by an ASR model contain high-level and linguistic-related information. The experimental results indicate that they were essential for achieving stable voice conversion results in our proposed method.

$F_{0}$ contours and spectrograms of one test utterance converted by the proposed method and the proposed method without bottleneck features are presented in Figure 7 (a) and Figure 7 (b) respectively. Compared to the method without using bottleneck features, the $F_{0}$ contour of the utterance converted by our proposed method is more similar to that of the natural reference in Figure 7 (d). Also, a significant spectral distortion can be observed at the $1\sim 2s$ interval of the spectrogram generated by the “w/o-bn” method.

IV-D2 Attention module

The attention module in a SCENT model helps to achieve the alignment between input and output feature sequences at the training stage and to predict target durations at the conversion stage. In order to investigate how the attention module contributed to the overall performance of our proposed method, we modified the SCENT model to a frame-by-frame transformation model without attention mechanism for comparison. Once the attention module was removed, the LSTM layer with attention in the decoder became a plain uni-directional LSTM. In order to get frame aligned sequence pairs for model training, the input sequences were wrapped towards the target ones using DTW algorithm and MCCs features. The other parts of the SCENT model were kept unchanged.

Our experiments compared three methods, including the proposed method, the proposed method without attention (w/o-att) and the proposed method without attention but using source interpolation at conversion time (i-w/o-att). TABLE VIII shows the MCDs and $F_{0}$ RMSEs of these three methods. We can see that the prediction errors increased in the absence of the attention module. $F_{0}$ contours and spectrograms of one test utterance converted by proposed method and “i-w/o-att” method are presented in Fig. 7 (a) and Fig. 7 (c) respectively. This figure again shows the effectiveness of the attention module for generating speech with duration and $F_{0}$ contour closer to that of the natural target speech.

Furthermore, a group of preference tests were conducted to compare the subjective performance of these three methods. Because the duration conversion achieved by the attention module significantly affected on the similarity, the preference tests focused on the similarity aspect of converted speech. Ten native listeners were involved in evaluation and the experimental results are presented in TABLE IX. This table shows that the strategy of global speaking rate adjustment by source interpolation can improve the similarity of converted speech in both conversion pairs. The proposed method with attention module outperformed the method without attention but using source interpolation. These results further confirmed the effectiveness of the attention module.

IV-D3 Location code

Ablation tests were conducted for investigating how the location code affected the performance of the model. In the experiments, the location code was removed and the models were built in the same conditions. MCD, $F_{0}$ RMSE and DDUR were calculated and are presented in TABLE X. A slight raise of MCD and $F_{0}$ RMSE after removing the location code can be observed from this table. Furthermore, the DDURs in female-to-male and male-to-female conversions increased by 5.4% and 15.3% respectively. These experimental results demonstrated the positive effects of using the location code.

IV-E Discussions

As discussed in Section II, directly implementing seq2seq models at utterance level is difficult for the voice conversion task. The input and output sequences in voice conversion are composed of frame-level features and are relatively long thus it is a challenge for the attention mechanism to search for the correct hidden entries to pay attention to. Once there are abnormal skips or repetitions in the sequence of attention probabilities, mistakes of converted speech may occur.

These difficulties are considered when designing the SCENT model. In order to improve attention stability, the techniques of forward attention and adding location features are used when calculating attention probabilities. The bottleneck features can also provide linguistic-related information to help the attention-based alignment between input and output feature sequences. However, errors still can not be completely avoided in the converted speech. Additional 100 non-parallel utterances of both speakers in the Mandarin dataset, which were out of the dataset used for previous experiments, were adopted for error analysis. The utterances of the male speaker contained 2747 phonemes, while the utterances of the female speaker had 2538 phonemes. We conducted male-to-female and female-to-male conversions for these utterances and identified different categories of conversion errors subjectively. In male-to-female conversion, there were 1 skipping phoneme error, 2 completion prediction errors, 34 phoneme pronunciation errors, 31 tone defects and 10 phoneme quality defects. In female-to-male conversion, there were 19 phoneme pronunciation errors, 20 tone defects and 17 phoneme quality defects.

Several reasons may lead to these errors. First, the proposed model contains about 7.5 M trainable parameters thus is complex and needs to be trained in a data-driven way. Therefore, the insufficiency of training data may cause the model’s lack of generalization ability when dealing with unseen utterances. Also, the extracted bottleneck features may also be misleading due to the accuracy limitation of the ASR model. To further reduce conversion errors and to produce more reliable conversion results using seq2seq models will be an important task of our future work.

V Conclusion

This paper presents SCENT, a sequence-to-sequence neural network, for acoustic modeling in voice conversion. Mel-spectrograms are used as acoustic features. Bottleneck features extracted by an ASR model are taken as additional linguistic-related descriptions and are concatenated with the source acoustic features as network inputs. Taking advantage of the attention mechanism, the SCENT model does not rely on the preprocessing of DTW alignment and the duration conversion can be achieved simultaneously. Finally, the converted acoustic features are passed through a WaveNet vocoder to reconstruct speech waveforms. Objective and subjective experimental results demonstrated the superiority of our proposed method compared with baseline methods, especially in durational aspect. Ablation tests further proved the benefits of inputting Mel-spectrograms and the necessity of bottleneck features. The importance of the attention module and the positive effect of the location code were also proved in our ablation studies. To investigate the influence of training set size on the performance of our proposed method and to reduce conversion errors by improving attention calculation will be our work in the future.

References