Convolutional-Recurrent Neural Networks for Speech Enhancement

Introduction

Speech enhancement is one of the corner stones of building robust automatic speech recognition (ASR) and communication systems. The problem is of especial importance nowadays where modern systems are often built using data-driven approaches based on large scale deep neural networks . In this scenario, the mismatch between clean data used to train the system and the noisy data encountered when deploying the system will often degrade the recognition accuracy in practice, and speech enhancement algorithms work as a preprocessing module that help to reduce the noise in speech signals before they are fed into these systems.

Speech enhancement is a classic problem that has attracted much research efforts for several decades in the community. By making assumptions on the nature of the underlying noise, statistical based approaches, including the spectral subtraction method , the minimum mean-square error log-spectral method , etc., can often obtain analytic solutions for noise suppression. However, due to these unrealistic assumptions, most of these statistical-based approaches often fail to build estimators that can well approximate the complex scenarios in real-world. As a result, additional noisy artifacts are usually introduced in the recovered signals .

Related Work. Due to the availability of high-quality, large-scale data and the rapidly growing computational resources, data-driven approaches using regression-based deep neural networks have attracted much interests and demonstrated substantial performance improvements over traditional statistical-based methods . The general idea of using deep neural networks, or more specifically, the MLPs for noise reduction is not new , and dates back at least to . In these works, MLPs are applied as general nonlinear function approximators to approximate the mapping from noisy utterance to its clean version. A multivariate regression-based objective is then optimized using numeric methods to fit model parameters. To capture the temporal nature of speech signals, previous works also introduced recurrent neural networks (RNNs) , which removes the needs for the explicit choice of context window in MLPs.

Contributions. We propose an end-to-end model based on convolutional and recurrent neural networks for speech enhancement, which we term as EHNet. EHNet is purely data-driven and does not make any assumptions about the underlying noise. It consists of three components: the convolutional component exploits the local patterns in the spectrogram in both frequency and temporal domains, followed by a bidirectional recurrent component to model the dynamic correlations between consecutive frames. The final component is a fully-connected layer that predicts the clean spectrograms. Compared with existing models such as MLPs and RNNs, due to the sparse nature of convolutional kernels, EHNet is much more data-efficient and computationally tractable. Furthermore, the bidirectional recurrent component allows EHNet to model the dynamic correlations between consecutive frames adaptively, and achieves better generalization on both seen and unseen noise. Empirically, we evaluate the effectiveness of EHNet and compare it with state-of-the-art methods on synthetic dataset, showing that EHNet achieves the best performance among all the competitors on all the 5 metrics. Specifically, our model leads up to a 0.6 improvement of PESQ measure on seen noise and 0.64 improvement on unseen noise.

Models and Learning

In this section we introduce the proposed model, EHNet, in detail and discuss its design principles as well as its inductive bias toward solving the enhancement problem. At a high level, we view the enhancement problem as a multivariate regression problem, where the nonlinear regression function is parametrized by the network in Fig. 1. Alternatively, the whole network can be interpreted as a complex filter for noise reduction in the frequency domain.

Under this setting, the key is to find a parametric family for denoising function $g_{\theta}$ such that it is both rich and data-efficient.

2 Convolutional Component

One choice for the denoising function $g_{\theta}$ is vanilla multilayer perceptrons, which has been extensively explored in the past few years . However, despite being universal function approximators , the fully-connected network structure of MLPs usually cannot exploit the rich patterns existed in spectrograms. For example, as we can see in Fig. 1, signals in the spectrogram tend to be continuous along the time dimension, and they also have similar values in adjacent frequency bins. This key observation motivates us to apply convolutional neural networks to efficiently and cheaply extract local patterns from the input spectrogram.

It is worth pointing out that without padding, with unit stride, the size of each feature map $h_{\mathbf{z}}(\mathbf{x})$ is $(d-b+1)\times(t-w+1)$. However, in order to recover the original speech signal, we need to ensure that the final prediction of the model have exactly the same length in the time dimension as the input spectrogram. To this end, we choose $w$ to be an odd integer and apply a zero-padding of size $d\times\lfloor w/2\rfloor$ at both sides of $\mathbf{x}$ before convolution is applied to $\mathbf{x}$. This guarantees that the feature map $h_{\mathbf{z}}(\mathbf{x})$ has $t+2\times\lfloor w/2\rfloor-w+1=t+w-1-w+1=t$ time steps, matching that of $\mathbf{x}$.

On the other hand, because of the local similarity of the spectrogram in adjacent frequency bins, when convolving with the kernel $\mathbf{z}$, we propose to use a stride of size $b/2$ along the frequency dimension. As we will see in Sec. 3, such design will greatly reduce the number of parameters and the computation needed in the following recurrent component, without losing any prediction accuracy.

Remark. We conclude this section by emphasizing that the application of convolution kernels is particularly well suited for speech enhancement in the frequency domain: each kernel can be understood as a nonlinear filter that detects a specific kind of local patterns existed in the noisy spectrograms, and the width of the kernel has a natural interpretation as the length of the context window. On the computational side, since convolution layer can also be understood as a special case of fully-connected layer with shared and sparse connection weights, the introduction of convolutions can thus greatly reduce the computation needed by a MLP with the same expressive power.

3 Bidirectional Recurrent Component

In other words, we vertically concatenate $\{h_{\mathbf{z}_{j}}(\mathbf{x})\}_{j=1}^{k}$ along the feature dimension to form a stacked 2D feature map $H(\mathbf{x})$ that contains all the information from the previous convolutional feature map.

In EHNet, we use deep bidirectional long short-term memory (LSTM) as our recurrent component due to its ability to model long-term interactions. At each time step $t$, given input $H_{t}\vcentcolon=H_{t}(\mathbf{x})$, each unidirectional LSTM cell computes a hidden representation $\overrightarrow{H}_{t}$ using its internal gates:

4 Fully-connected Component and Optimization

As discussed in Sec. 2.1, the last step is to define the mean-squared error between the predicted spectrogram $\hat{\mathbf{y}}$ and the clean one $\mathbf{y}$, and optimize all the model parameters simultaneously. Specifically, we use AdaDelta with scheduled learning rate to ensure a stationary solution.

Experiments

To demonstrate the effectiveness of EHNet on speech enhancement, we created a synthetic dataset, which consists of 7,500, 1,500 and 1,500 recordings (clean/noisy speech) for training, validation and testing, respectively. Each recording is synthesized by convolving a randomly selected clean speech file with one of the 48 room impulse responses available and adding a randomly selected noise file. The clean speech corpus consists of 150 files containing ten utterances with male, female, and children voices. The noise dataset consists of 377 recordings representing 25 different types of noise. The room impulse responses were measured for distances between 1 and 3 meters. A secondary noise dataset of 32 files, with noises that do not appear in the training set, is denoted UnseenNoise and used to generate another test set of 1,500 files. The randomly generated speech and noise levels provide signal-to-noise ratio between 0 and 30 dB. All files are sampled with 16 kHz sampling rate and stored with 24 bits resolution.

As a preprocessing step, we first use STFT to extract the spectrogram from each utterance. The spectrogram has 256 frequency bins $(d=256)$ and $\sim 500$ frames $(t\approx 500)$ frames. To throughly measure the enhancement quality, we use the following 5 metrics to evaluate different models: signal-to-noise ratio (SNR, dB), log-spectral distortion (LSD), mean-squared-error on time domain (MSE), word error rate (WER, $\%$), and the PESQ measure. To measure WER, we use the DNN-based speech recognizer, described in . The system is kept fixed (not fine-tuned) during the experiment. We compare our EHNet with the following state-of-the-art methods:

MS. Microsoft’s internal speech enhancement system used in production, which uses a combination of statistical-based enhancement rules.

DNN-Symm . DNN-Symm contains 3 hidden layers, all of which have 2048 hidden units. It uses a symmetric context window of size 11.

DNN-Causal . Similar to DNN-Symm, DNN-Causal contains 3 hidden layers of size 2048, but instead of symmetric context window, it uses causal context window of size 7.

RNN-Ng . RNN-Ng is a recurrent neural network with 3 hidden layers of size 500. The input at each time step covers frames in a context window of length 3.

The architecture of EHNet is as follows: the convolutional component contains 256 kernels of size $32\times 11$, with stride $16\times 1$ along the frequency and the time dimensions, respectively. We use two layers of bidirectional LSTMs following the convolution component, each of which has 1024 hidden units. To train EHNet, we fix the number of epochs to be 200, with a scheduled learning rate $\{1.0,0.1,0.01\}$ for every 60 epochs. For all the methods, we use the validation set to do early stopping and save the best model on validation set for evaluation on the test set. EHNet does not overfit, as both weight decay and dropout hurt the final performance. We also experiment with deeper EHNet with more layers of bidirectional LSTMs, but this does not significantly improve the final performance. We also observe in our experiments that reducing the stride of convolution in the frequency dimension does not significantly boost the performance of EHNet, but greatly incurs additional computations.

2 Results and Analysis

Experimental results on the dataset is shown in Table 1. On the test dataset with seen noise, EHNet consistently outperforms all the competitors with a large margin. Specifically, EHNet is able to improve the perceptual quality (PESQ measure) by 0.6 without hurting the recognition accuracy. This is very surprising as we treat the underlying ASR system as a black box and do not fine-tune it during the experiment. As a comparison, while all the other methods can boost the SNR ratio, they often decrease the recognition accuracy. More surprisingly, EHNet also generalizes to unseen noise as well, and it even achieves a larger boost (0.64) on the perceptual quality while at the same time increases the recognition accuracy.

To have a better understanding on the experimental result, we do a case study by visualizing the denoised spectrograms from different models. As shown in Fig. 2, MS is the most conservative algorithm among all. By not removing much noise, it also keeps most of the real signals in the speech. On the other hand, although DNN-based approaches do a good job in removing the background noise, they also tend to remove the real speech signals from the spectrogram. This explains the reason why DNN-based approaches degrade the recognition accuracies in Table 1. RNN does a better job than DNN, but also fails to keep the real signals in low frequency bins. As a comparison, EHNet finds a good tradeoff between removing background noise and preserving the real speech signals: it is better than DNN/RNN in preserving high/low-frequency bins and it is superior than MS in removing background noise. It is also easy to see that EHNet produces denoised spectrogram that is most close to the ground-truth clean spectrogram.

Conclusion

We propose EHNet, which combines both convolutional and recurrent neural networks for speech enhancement. The inductive bias of EHNet makes it well-suited to solve speech enhancement: the convolution kernels can efficiently detect local patterns in spectrograms and the bidirectional recurrent connections can automatically model the dynamic correlations between adjacent frames. Due to the sparse nature of convolutions, EHNet requires less computations than both MLPs and RNNs. Experimental results show that EHNet consistently outperforms all the competitors on all 5 different metrics, and is also able to generalize to unseen noises, confirming the effectiveness of EHNet in speech enhancement.

References