Data augmentation and loss normalization for deep noise suppression

Introduction

Speech enhancement using neural networks has recently seen large attention and success in research and is being implemented in commercial applications also targeting real-time communication. An exciting property of deep learning-based noise suppression is that it also reduces highly non-stationary noise and background sounds such as barking dogs, banging kitchen utensils, crying babys, construction or traffic noise, etc. This has not been possible so far using single-channel statistical model-driven speech enhancement techniques that often only reduce quasi-stationary noise . Notable approaches towards real-time implementations have been proposed e. g. in .

The dataset is a key part of data-driven learning approaches, especially for supervised learning. It is a challenge to build a dataset that is large enough to generalize well, but still represents the expected real-world data sufficiently. Data augmentation techniques can not only help to control the amount of data, but is also necessary to synthesize training data that represents all effects encountered in practice.

While in many publications, data corpus generation is only roughly outlined due to lack of space, or often exclude several key practical aspects, we direct this paper on showing contributions on several augmentation techniques when synthesizing a noisy and target speech corpus for speech enhancement. In particular, we show the effects of increasing the SNR range and using a continuous instead of discrete distribution, spectral augmentation by applying random spectral shaping filters to speech and noise, and finally level augmentation to increase robustness of the network against varying input signal levels.

As we found that level augmentation can decrease the performance when using signal-level dependent losses, we propose a normalization technique for the loss computation that can be generalized to any other signal-based loss. We show in experiments on the CHIME-2 challenge dataset that the augmentation techniques and loss normalization substantially improve the training procedure and the results.

In this paper, we first introduce a the general noise suppression task in Section 2. In Section 3, we describe the used real-time noise suppression system based on a recurrent network, that works on a single frame in - single frame out basis, i. e. requires no look-ahead and memory buffer, and describe the training setup. In Section 4, we describe the used loss function and propose a normalization to remove the signal level dependency of the loss. In Section 5, we describe augmentation techniques for signal-to-noise ratio (SNR), spectral shaping, and sequence level dynamics. The experiments are shown in Section 6, and Section 7 concludes the paper.

Deep Learning Based Noise Suppression

In a pure noise reduction task, we assume that the observed signal is an additive mixture of the desired speech and noise. We denote the observed signal $X(k,n)$ directly in the short-time Fourier transform (STFT) domain, where $k$ and $n$ are the frequency and time frame indices as

where $S(k,n)$ is the speech and $N(k,n)$ is the disturbing noise signal. Note that the speech signal $S(k,n)$ can be reverberant, and we only aim at reducing additive noise.

The objective is to recover an estimate $\widehat{S}(k,n)$ of the speech signal by applying a filter $G(k,n)$ to the observed signal by

The filter $G(k,n)$ can be either a real-valued suppression gain, or a complex-valued filter. While the former option (also known as mask) only recovers the speech amplitude, a complex filter could potentially also correct the signal phase. In this work, we use a suppression gain.

Network and Training

We use a rather straightforward recurrent network architecture based on gated recurrent units and feed forward (FF) layers, similar to the core architecture of without convolutional encoder layers. Input features are the logarithmic power spectrum $P=\log_{10}(|X(k,n)|^{2}+\epsilon)$, normalized by the global mean and variance of the training set. We use a STFT size of 512 with 32 ms square-root Hann windows and 16 ms frame shift, but feed only the relevant 255 frequency bins into the network, omitting 0th and highest (Nyquist) bins, which do not carry useful information. The network consists of a FF embedding layer, two GRUs, and three FF mapping layers. All FF layers use rectified linear unit (ReLU) activations, except for the last output layer. When estimating a real-valued suppression gain, a Sigmoid activation is used to ensure positive output. The network architecture is shown in Fig. 1, and has 2.8 M parameters.

The network was trained using the AdamW optimizer with an initial learning rate of $10^{-4}$, which was dropped by a factor of 0.9 if the loss plateaued for 5 epochs. The training was monitored every 10 epochs using a validation subset. The best model was chosen based on the highest perceptual evaluation of speech quality (PESQ) on the validation set. All hyper-parameters were optimized by a grid search and choosing the best performing parameter for PESQ on the validation set.

Level Invariant Normalized Loss Function

The speech enhancement loss function is typically a distance metric between the enhanced and target spectral representations. The dynamically compressed loss proposed in has been shown to be very effective. A compression exponent of $0<c\leq 1$ is applied to the magnitudes, while the compressed magnitudes are combined with the phase factors again. Furthermore, the magnitude only loss is blended with the complex loss with a factor $0\leq\alpha\leq 1$.

A common drawback of all similar signal-based loss functions is the dependency on the level of the signals $S$ and $\widehat{S}$. This might have an impact on the loss when computing the loss over a batch of several sequences, which exhibit large dynamic differences. It could be that large signals dominate the loss, creating less balanced training.

Data Augmentation Techniques

Especially for small and medium-scale datasets, augmentation is a powerful tool to improve the results. In the case of supervised speech enhancement training, where the actual noisy audio training data is generated synthetically by mixing speech and noise, there are some augmentation steps, which are essential to mimic effects on data encountered in the wild. Disregarding reverberation, we need to be able to deal with different SNRs, audio levels, and filtering effects that can be caused e.g. by acoustics (room, occlusion), or the recording device (microphone, electronic transfer functions).

Our augmentation pipeline is shown in Fig. 2. Before mixing speech with noise, we applied random biquad filters to each noise sequence and speech sequences separately to mimic different acoustic transmission effects. From these signals, active speech and noise levels are computed using a level threshold-based voice activity detector (VAD). Speech and noise sequences are then mixed with a given SNR on-the-fly during training. After mixing, the mixture is scaled using a given level distribution. The clean speech target is scaled by the same factor as the mixture. The data generation and augmentation procedure is depicted in Fig. 2.

Experiments

We used the CHIME-2 WSJ-20k dataset , which is currently, while only being of medium size, the only realistic self-contained public dataset including matching reverberant speech and noise conditions. The dataset contains 7138, 2418, and 1998 utterances for training, validation and testing, respectively. The utterances are reverberant using binaural room impulse responses, and noise from the same rooms was added with SNRs in the range of -6 to 9 dB in the validation and test sets. We used only the left channel for our single-channel experiments.

The spectral augmentation filters are designed as proposed in by

with $r_{i}$ being uniformly distributed in $[-\frac{3}{8},\frac{3}{8}]$. For SNR augmentation, the mixing SNRs were drawn from a Gaussian distribution on the logarithmic scale with mean 5 dB and standard deviation 10 dB. The signal levels for dynamic range augmentation were drawn from a Gaussian distribution with mean -28 dBFS and variance 10 dB.

We evaluate our experiments on the development and test set from the CHIME-2 challenge. As objective metrics, we use PESQ , short-time objective intelligibility (STOI) , scale-invariant signal-to-distortion ratio (SI-SDR) , cepstral distance (CD), and frequency-weighted segmental SNR (fwSegSNR) .

2 Results

Fig. 3 shows the training progression in terms of PESQ and SI-SDR on the validation set. The blue curve shows training on the original data without any augmentation, mixed the 6 different SNR levels between -6 and 9 dB. We can see that the validation metrics decrease after 150 epochs. When applying SNR augmentation (red curve) with a broader and continuous distribution, we prevent the early validation decrease and can train for 280 epochs. Further, adding spectral augmentation increases the validation PESQ slightly. However, the level augmentation when training with standard loss (3) (purple curve) decreases the performance compared to SNR and spectral augmentation only. We attribute this effect to the large level imbalance per batch, which affects the standard level-dependent loss function. When computing the loss from normalized signals (green curve), this drawback is overcome and we obtain similar or even slightly better results than the yellow curve, but making the system robust to varying input signal levels.

The validation SI-SDR shows similar behavior as PESQ.

Tab. 1 shows the results on the CHIME-2 test set. The enhancement systems improve all results substantially over the noisy input. Adding SNR augmentation adds a gain of 0.05 PESQ. As in the development set, spectral augmentation adds an additional minor improvement. Interestingly, on the test set, the normalized loss shows no influence on the results. We assume this is due to that fact that the given test set does not exhibit largely varying signal levels.

Conclusion

We have shown the effectivity of data augmentation techniques for supervised deep learning-based speech enhancement by augmenting the SNR, spectral shapes, and signal levels. As level augmentation degrades the performance of the learning algorithm when using level-dependent losses, we proposed a normalization technique for the loss, which is shown to overcome this issue. The experiments were conducted using a real-time capable recurrent neural network on the reverberant CHIME-2 dataset. Future work will also investigate augmentation for acoustic conditions with reverberant impulse responses.

References