End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction

Introduction

Recent years have witnessed exciting advances towards solving the cocktail party problem. The inventions of deep clustering , deep attractor networks and permutation free training have dramatically improved the performance of single-channel speaker-independent multi-speaker speech separation, demonstrating overwhelming advantages over previous methods including graphical modeling approaches , spectral clustering approaches , and CASA methods .

However, all of these conduct separation on the magnitude in the time-frequency (T-F) domain and directly use the mixture phase for time-domain re-synthesis, largely because phase is difficult to estimate. It is well-known that this incurs a phase inconsistency problem , especially for speech processing, where there is typically at least half overlap between consecutive frames. This overlap makes the STFT representation of a speech signal highly redundant. As a result, the enhanced STFT representation obtained using the estimated magnitude and mixture phase would not be in the consistent STFT domain, meaning that it is not guaranteed that there exists a time-domain signal having that STFT representation.

To improve the consistency, one stream of research is focused on iterative methods such as the classic Griffin-Lim algorithm , multiple input spectrogram inverse (MISI) , ISSIR , and consistent Wiener filtering , which can recover the clean phase to some extent starting from the mixture phase and a good estimated magnitude by iteratively performing STFT and iSTFT . There are some previous attempts at naively applying such iterative algorithms as a post-processing step on the magnitudes produced by deep learning based speech enhancement and separation . However, this usually only leads to small improvements, even though the magnitude estimates from DNNs are reasonably good. We think that this is possibly because the T-F masking is performed without being aware of the later phase reconstruction steps and hence may not produce spectral structures that are appropriate for iterative phase reconstruction.

This study hence proposes a novel end-to-end speech separation algorithm that trains through iterative phase reconstruction via T-F masking for signal-level approximation. On the publicly-available wsj0-2mix corpus, our algorithm reaches 12.6 dB scale-invariant SDR, which surpasses the previous best by a large margin and is comparable to the oracle 12.7 dB result obtained using the so-called ideal ratio mask (IRM). Our study shows, for the first time and based on a large open dataset, that deep learning based phase reconstruction leads to tangible and large improvements when combined with state-of-the-art magnitude-domain separation.

Chimera++ Network

To elicit a good phase via phase reconstruction, it is necessary to first obtain a good enough magnitude estimate. Our recent study proposed a novel multi-task learning approach combining the regularization capability of deep clustering with the ease of end-to-end training of mask inference, yielding significant improvements over the individual models.

Our recent study suggests that an alternative loss function, which whitens the embedding in a k-means objective, leads to better separation performance.

To learn the embeddings, bi-directional LSTM (BLSTM) is usually used to model the context information from past and future frames. The network architecture is shown at the bottom of Fig. 1, where the DC embedding layer is a fully-connected layer with a non-linearity such as a logistic sigmoid, followed by unit-length normalization for each frequency.

Another permutation-free training scheme was proposed for mask-inference networks first in , and was later found to be working very well in and . The idea is to train a mask-inference network to minimize the minimum loss over all permutations. Following , the phase-sensitive mask (PSM) is used as the training target. It is common in phase-sensitive spectrum approximation (PSA) to truncate the unbounded mask values. Using $\operatorname{T}_{a}^{b}(x)=\min(\max(x,a),b)$, the truncated PSA (tPSA) objective is

where $\angle X$ is the mixture phase, $\angle S_{c}$ the phase of the $c$-th source, $\mathcal{P}$ the set of permutations on $\{1,\dots,C\}$, $|X|$ the mixture magnitude, $\hat{M}_{c}$ the $c$-th estimated mask, $|S_{c}|$ the magnitude of the $c$-th reference source, $\odot$ denotes element-wise matrix multiplication, and $\gamma$ is a mask truncation factor. Sigmoidal activation together with $\gamma=1$ is commonly used in the output layer of T-F masking. To endow the network with more capability, multiple activation functions that can work with $\gamma>1$ will be discussed in Section 3.4.

Following , our recent study proposed a chimera++ network combining the two approaches via multi-task learning, as illustrated in the bottom of Fig. 1. The loss function is a weighted sum of the deep clustering loss and the mask inference loss.

Only the MI output is needed to make predictions at run time.

Proposed Algorithms

There are multiple target sources to be separated in each mixture in our study. The Griffin-Lim algorithm only performs iterative reconstruction for each source independently. In , we therefore proposed to utilize the MISI algorithm (see Algorithm 1) to reconstruct the clean phase of each source starting from the estimated magnitude of each source and the mixture phase, where the sum of the reconstructed time-domain signals after each iteration is constrained to be the same as the mixture signal. Note that the estimated magnitudes remain fixed during iterations, while the phase of each source are iteratively reconstructed. In , the phase reconstruction was only added as a post-processing, and it was not part of the objective function during training, which remained computed on the time-frequency representation of the estimated signal, prior to resynthesis. In this paper, we go several steps further.

2 Waveform Approximation

The first step in phase reconstruction algorithms such as MISI is to reconstruct a waveform from a time-frequency domain representation using the inverse STFT. We thus consider a first objective function computed on the waveform reconstructed by iSTFT, denoted as waveform approximation (WA), and represent iSTFT as various layers on top of the mask inference layer, so that end-to-end optimization can be performed. The label permutation problem is resolved by minimizing the minimum $L_{1}$ loss of all the permutations at the waveform level. We denote the model trained this way as WA. The objective function to train this model is

where $s_{c}$ denotes the time-domain signal of source $c$, and $\hat{s}_{c}^{(0)}$ denotes the $c$-th time-domain signal obtained by inverse STFT from the combination of the $c$-th estimated magnitude and the mixture phase. Note that mixture phase is still used here and no phase reconstruction is yet performed. This corresponds to the initialization step in Algorithm 1.

In , a time-domain reconstruction approach is proposed for speech enhancement. However, their approach only trains a feed-forward mask-inference DNN through iDFT separately for each frame using squared error in the time domain. By Parseval’s theorem, this is equivalent to optimizing the mask for minimum squared error in the complex spectrum domain, when using the noisy phases, as in , proposed in the same conference. A follow-up work of supplies clean phase during training. However, this makes their approach equivalent to conventional magnitude spectrum approximation , which does not perform as well as the phase-sensitive mask . Closest to the above WA objective, an adaptive front-end framework was recently proposed in which the STFT and its inverse are subsumed by the network, along with the noisy phase, so that training is effectively end-to-end in the time-domain. The proposed method then replaces the STFT and its inverse by trainable linear convolutional layers. Unfortunately the paper does not compare training through the STFT to the conventional method so the results are uninformative about this direction.

3 Unfolded Iterative Phase Reconstruction

We further unfold the iterations in the MISI algorithm as various deterministic layers in a neural network. This can be achieved by further growing several layers representing STFT and iSTFT operations on top of the mask inference layer. By performing end-to-end optimization that trains through MISI, the network can become aware of the later iterative phase reconstruction steps and learn to produce estimated magnitudes that are well-suited to that subsequent processing, hence producing better phase estimates for separation. The model trained this way is denoted as WA-MISI-K, where $K\geq 1$ is the number of unfolded MISI iterations. The objective function is

where $\hat{s}_{c}^{(K)}$ denotes the $c$-th time-domain signal obtained after $K$ MISI iterations as described in Algorithm 1. The whole separation network, including unfolded phase reconstruction steps at the output of the mask inference head of the Chimera++ network, is illustrated in Fig. 1. The STFT and iSTFT can be easily implemented using modern deep learning toolkits as deterministic layers efficiently computed on a GPU and through which backpropagation can be performed.

A recent study by Williamson et al. proposed a complex ratio masking approach for phase reconstruction and speech enhancement, where a feed-forward DNN is trained to predict the real and imaginary components of the ideal complex filter in the STFT domain, i.e., $M_{c}=S_{c}/X=|S_{c}|e^{j(\angle S_{c}-\angle X)}/|X|$ for source $c$ for example. The real component is equivalent to the earlier proposed phase-sensitive mask , which contains patterns clearly predictable from energy-based features . However, recent studies along this line suggest that the patterns in the imaginary component are too random to predict , possibly because it is difficult for a learning machine to determine the sign of $\sin(\angle S_{c}-\angle X)$ only from energy-based features. In contrast, the $\cos(\angle S_{c}-\angle X)$ in the real component is typically much smaller than one for T-F units dominated by other sources and close to one otherwise, making itself predictable from energy-based features. The proposed method thus only focuses on estimating a mask in the magnitude domain and uses the estimated magnitude to elicit better phase through iterative phase reconstruction.

Another recent trend is to avoid the phase inconsistency problem altogether by operating in the time domain, using convolutional neural networks , WaveNet , generative adversarial networks , or encoder-decoder architectures . Although they are promising approaches, the current state-of-the-art approach for supervised speech separation is via T-F masking . The proposed approach is expected to produce even better separation if the phase can be reconstructed.

4 Activation Functions with Values Beyond One

Sigmoidal units are dominantly used in the output layer of deep learning based T-F masking , partly because they can model well data with bi-modal distribution , such as the IRM and its variants . Restricting the possible values of the T-F mask to lie in $$ is also reasonable when using the mixture phase for reconstruction: indeed, T-F mask values larger than one would in theory be needed in regions where interferences between sources result in a mixture magnitude smaller than that of a source; but the mixture phase is also likely to be different from the phase of that source in such regions, in which case it is more rewarding in terms of objective measure to oversuppress than to go even further in a wrong direction. This is no longer valid if we consider phase reconstruction in the optimization. Moreover, capping the mask values to be between zero and one is more likely to take the enhanced magnitude further away from the consistent STFT domain, posing potential difficulties for later phase reconstruction.

To obtain clean magnitudes, the oracle mask should be $|S_{c}|/|X|$ (also known as the FFT mask in or the ideal amplitude mask in ). Clearly, this mask can go beyond one, because the underlying sources, although statistically independent, may have opposite phase at a particular T-F unit, therefore cancelling with each other and producing a mixture magnitude that is smaller than the magnitude of a given source. It is likely much harder to predict the mask values of such T-F units, but we believe that it is still possible based on contextual information.

In our study, we truncate the values in PSM to the range $$(i.e.,$\gamma=2$ in Eq. (3)), as only a small percentage of mask values goes beyond this range. Multiple activation functions can be utilized in the output layer. We here consider:

doubled sigmoid: sigmoid non-linearity multiplied by 2;

clipped ReLU: ReLU non-linearity clipped to $$;

convex softmax: the output non-linearity is a three-dimensional softmax for each source at each T-F unit. It is used to compute a convex sum between the values 0, 1, and 2: $y=[x_{0},x_{1},x_{2}]^{T}$ where $[x_{0},x_{1},x_{2}]$ is the output of the softmax. This activation function is designed to model the three modes concentrated at 0, 1 and 2 in the histogram of the PSM.

Experimental Setup

We validate the proposed algorithms on the publicly-available wsj0-2mix corpus , which is widely used in many speaker-independent speech separation tasks. It contains 20,000, 5,000 and 3,000 two-speaker mixtures in its 30 h training, 10 h validation, and 5 h test sets, respectively. The speakers in the validation set (closed speaker condition, CSC) are seen during training, while the speakers in the test set (open speaker condition, OSC) are completely unseen. The sampling rate is 8 kHz.

Our neural network contains four BLSTM layers, each with 600 units in each direction. A dropout of $0.3$ is applied on the output of each BLSTM layer except the last one. The network is trained on 400-frame segments using the Adam algorithm. The window length is 32 ms and the hop size is 8 ms. The square root Hann window is employed as the analysis window and the synthesis window is designed accordingly to achieve perfect reconstruction after overlap-add. A 256-point DFT is performed to extract 129-dimensional log magnitude input features. We first train the chimera++ network with $\alpha$ set to 0.975. Next, we discard the deep clustering branch (i.e., we set $\alpha$ to 0) and train the network with $\mathcal{L}_{\text{WA}}$. Subsequently, the network is trained using $\mathcal{L}_{\text{WA-MISI-1}}$, then $\mathcal{L}_{\text{WA-MISI-2}}$, and all the way to $\mathcal{L}_{\text{WA-MISI-K}}$, where here $K=5$, as performance saturated after five iterations in our experiments. We found this curriculum learning strategy to be helpful. At run time, for the models trained using $\mathcal{L}_{\text{WA-MISI-5}}$, we run MISI with $5$ iterations, while results for other models are obtained without phase reconstruction unless specified.

We report the performance using scale-invariant SDR (SI-SDR) , as well as the SDR metric computed using the bss_eval_sources software because it is used by other groups. We believe SI-SDR is a more proper measure for single-channel instantaneous mixtures .

Evaluation Results

Table 1 reports the SI-SDR results on the wsj0-2mix dataset. We first present the results using sigmoidal activation. The chimera++ network obtains significantly better results than the individual models (11.2 dB vs. 10.4 dB and 10.0 dB SI-SDR). With the mixture phase and estimated magnitudes, performing five iterations of MISI pushes the performance to 11.5 dB, while 11.3 dB is obtained when applying five iterations of Griffin-Lim on each source independently, as is reported in . Performing end-to-end optimization using $\mathcal{L}_{\text{WA}}$ improves the results to 11.6 dB from 11.2 dB, without requiring phase reconstruction post-processing. Further applying MISI post-processing for five iterations (MISI-5) on this model however does not lead to any improvements, likely because the mixture phase is used during training and the model compensates for it without expecting further processing. In contrast, training the network through MISI using $\mathcal{L}_{\text{WA-MISI-5}}$ pushes the performance to 12.2 dB.

Among the three proposed activation functions, the convex softmax performs the best, reaching 12.6 dB SI-SDR. It thus seems effective to model the multiple peaks in the histogram of the truncated PSM, and important to produce estimated magnitudes that are closer to the consistent STFT domain. As expected, activations going beyond $1$ only become beneficial when training through phase reconstruction.

In Fig. 2, we show the evolution of the SI-SDR performance of the convex softmax models trained with different objective functions against the number of MISI iterations at test time ( to $5$). Training with $\mathcal{L}_{\text{WA}}$ leads to a magnitude that is very well suited to iSTFT, but not to further MISI iterations. As we train for more MISI iterations, performance starts lower, but reaches higher values with more test-time iterations.

Table 2 lists the performance of competitive approaches on the same corpus, along with the performance of various oracle masks with or without applying MISI for five iterations. The first three algorithms use mixture phase directly for separation. The fourth one, time-domain audio separation network (TasNet), operates directly in the time domain. Our result is 1.1 dB better than the previous state-of-the-art by in terms of both SI-SDR and SDR.

Concluding Remarks

We have proposed a novel end-to-end approach for single-channel speech separation. Significant improvements are obtained by training the T-F masking network through an iterative phase reconstruction procedure. Future work includes applying the proposed methods to speech enhancement, considering the joint estimation of magnitude and an initial phase that improves upon the mixture phase, and improving the estimation of the ideal amplitude mask. We shall also consider alternatives to the waveform-level loss, such as errors computed on the magnitude spectrograms of the reconstructed signals.

References