TasNet: time-domain audio separation network for real-time, single-channel speech separation

Introduction

Real-world speech communication often takes place in crowded, multi-talker environments. A speech processing system that is designed to operate in such conditions needs the ability to separate speech of different talkers. This task which is effortless for humans has proven very difficult to model in machines. In recent years, deep learning approaches have significantly advanced the state of this problem compared to traditional methods .

A typical neural network speech separation algorithm starts with calculating the short-time Fourier transform (STFT) to create a time-frequency (T-F) representation of the mixture sound. The T-F bins that correspond to each source are then separated, and are used to synthesize the source waveforms using inverse STFT. Several issues arise in this framework. First, it is unclear whether Fourier decomposition is the optimal transformation of the signal for speech separation. Second, because STFT transforms the signal into a complex domain, the separation algorithm needs to deal with both magnitude and the phase of the signal. Because of the difficulty in modifying the phase, the majority of proposed methods only modify the magnitude of the STFT by calculating a time-frequency mask for each source, and synthesize using the masked magnitude spectrogram with the original phase of the mixture. This imposes an upper bound on separation performance. Even though several systems have been proposed to use the phase information to design the masks, such as the phase-sensitive mask and complex ratio mask , the upper bound still exists since the reconstruction is not exact. Moreover, effective speech separation in STFT domain requires high frequency resolution which results in relatively large time window length, which is typically more than 32 ms for speech and more than 90 ms for music separation . Because the minimum latency of the system is bounded by the length of the STFT time window, this limits the use of such systems when very short latency is required, such as in telecommunication systems or hearable devices.

A natural way to overcome these obstacles is to directly model the signal in the time-domain. In recent years, this approach has been successfully applied in tasks such as speech recognition, speech synthesis and speech enhancement , but waveform-level speech separation with deep learning has not been investigated yet. In this paper, we propose Time-domain Audio Separation Network (TasNet), a neural network that directly models the mixture waveform using an encoder-decoder framework, and performs the separation on the output of the encoder. In this framework, the mixture waveform is represented by a nonnegative weighted sum of $N$ basis signals, where the weights are the outputs of the encoder, and the basis signals are the filters of the decoder. The separation is done by estimating the weights that correspond to each source from the mixture weight. Because the weights are nonnegative, the estimation of source weights can be formulated as finding the masks which indicate the contribution of each source to the mixture weight, similar to the T-F masks that are used in STFT systems. The source waveforms are then reconstructed using the learned decoder.

This signal factorization technique shares the motivation behind independent component analysis (ICA) with nonnegative mixing matrix and semi-nonnegative matrix factorization (semi-NMF) . However unlike ICA or semi-NMF, the weights and the basis signals are learned in a nonnegative autoencoder framework , where the encoder is a 1-D convolutional layer and the decoder is a 1-D deconvolutional layer (also known as transposed convolutional). In this scenario, the mixture weights replace the commonly used STFT representations.

Since TasNet operates on waveform segments that can be as small as 5 ms, the system can be implemented in real-time with very low latency. In addition to having lower latency, TasNet outperforms the state-of-art STFT-based system. In applications that do not require real-time processing, a noncausal separation module can also be used to further improve the performance by using information from the entire signal.

Model Description

The problem of single-channel speech separation is formulated as estimating $C$ sources $s_{1}(t),\ldots,s_{c}(t)$, given the discrete waveform of the mixture $x(t)$

In comparison to other matrix factorization algorithms such as ICA where the basis signals are required to have distinct statistical properties or explicit frequency band preferences, no such constraints are imposed here. Instead, the basis signals are jointly optimized with the other parameters of the separation network during training. Moreover, the synthesis of the source signal from the weights and basis signals is done directly in the time-domain, unlike the inverse STFT step which is needed in T-F based solutions.

2 Network design

Figure 1 shows the structure of the network. It contains three parts: an encoder for estimating the mixture weight, a separation module, and a decoder for source waveform reconstruction. The combination of the encoder and the decoder modules construct a nonnegative autoencoder for the waveform of the mixture, where the nonnegative weights are calculated by the encoder and the basis signals are the 1-D filters in the decoder. The separation is performed on the mixture weight matrix using a subnetwork that estimates a mask for each source.

This step is motivated by the gated CNN approach that is used in language modeling , and empirically it performs significantly better than using only ReLU or Sigmoid in our system.

2.2 Separation network

Starting from the second LSTM layer, an identity skip connection is added between every two LSTM layers to enhance the gradient flow and accelerate the training process.

2.3 Decoder for waveform reconstruction

2.4 Training objective

Since the output of the network are the waveforms of the estimated clean signals, we can directly use source-to-distortion ratio (SDR) as our training target. Here we use scale-invariant source-to-noise ratio (SI-SNR), which is used as the evaluation metric in place of the standard SDR in , as the training target. The SI-SNR is defined as:

Experiments

We evaluated our system on two-speaker speech separation problem using WSJ0-2mix dataset , which contains 30 hours of training and 10 hours of validation data. The mixtures are generated by randomly selecting utterances from different speakers in Wall Street Journal (WSJ0) training set si_tr_s, and mixing them at random signal-to-noise ratios (SNR) between 0 dB and 5 dB. Five hours of evaluation set is generated in the same way, using utterances from 16 unseen speakers from si_dt_05 and si_et_05 in the WSJ0 dataset. To reduce the computational cost, the waveforms were down-sampled to 8 kHz.

2 Network configuration

The parameters of the system include the segment length $L$, the number of basis signals $N$, and the configuration of the deep LSTM separation network. Using a grid search, we found optimal $L$ to be 40 samples (5 ms at 8 kHz) and $N$ to be 500. We designed a 4 layer deep uni-directional LSTM network with 1000 hidden units in each layer, followed by a fully-connected layer with 1000 hidden units that generates two 500-dimensional mask vectors. For the noncausal configuration with bi-directional LSTM layers, the number of hidden units in each layer is set to 500 for each direction. An identical skip connection is added between the output of the second and last LSTM layers.

During training, the batch size is set to 128, and the initial learning rate is set to $3e^{-4}$ for the causal system (LSTM) and $1e^{-3}$ for the noncausal system (BLSTM). We halve the learning rate if the accuracy on validation set is not improved in 3 consecutive epochs. The criteria for early stopping is no decrease in the cost function on the validation set for 10 epochs. Adam is used as the optimization algorithm. No further regularization or training procedures were used.

We apply curriculum training strategy in a similar fashion with . We start the training the network on 0.5 second long utterances, and continue training on 4 second long utterances afterward.

3 Evaluation metrics

For comparison with previous studies, we evaluated our system with both SI-SNR improvement (SI-SNRi) and SDR improvement (SDRi) metrics used in , where the SI-SNR is defined in Section 2.2.4, and the standard SDR is proposed in .

4 Results and analysis

Table 1 shows the performance of our system as well as three state-of-art deep speech separation systems, Deep Clustering (DPCL++, ), Permutation Invariant Training (PIT, ), and Deep Attractor Network (DANet, ). Here TasNet-LSTM represents the causal configuration with uni-directional LSTM layers. TasNet-BLSTM corresponds to the system with bi-directional LSTM layers which is noncausal and cannot be implemented in real-time. For the other systems, we show the best performance reported on this dataset.

We see that with causal configuration, the proposed TasNet system significantly outperforms the state-of-art causal system which uses a T-F representation as input. Under the noncausal configuration, our system outperforms all the previous systems, including the two-stage systems DPCL++ and uPIT-BLSTM-ST which have a second-stage enhancement network. Note that our system does not contain any regularizers such as recurrent dropout (DPCL++) or post-clustering steps for mask estimation (DANet).

Table 2 compares the latency of different causal systems. The latency of a system $T_{tot}$ is expressed in two parts: $T_{i}$ is the initial delay of the system that is required in order to receive enough samples to produce the first output. $T_{p}$ is the processing time for a segment, estimated as the average per-segment processing time across the entire test set. The model was pre-loaded on a Titan X Pascal GPU before the separation of the first segment started. The average processing speed per segment in our system is less than 0.23 ms, resulting in a total system latency of 5.23 ms. In comparison, a STFT-based system requires at least 32 ms time interval to start the processing, in addition to the processing time required for calculation of STFT, separation, and inverse STFT. This enables our system to preform in situation that can tolerate only short latency, such as hearing devices and telecommunication applications.

Conclusion

In this paper, we proposed a deep learning speech separation system that directly operates on the sound waveforms. Using an autoencoder framework, we represent the waveform as nonnegative weighted sum of a set of learned basis signals. The time-domain separation problem then is solved by estimating the source masks that are applied to the mixture weights. Experiments showed that our system was 6 times faster compared to the state-of-art STFT-based systems, and achieved significantly better speech separation performance. Audio samples are available at .

Acknowledgement

This work was funded by a grant from National Institute of Health, NIDCD, DC014279, National Science Foundation CAREER Award, and the Pew Charitable Trusts.

References