Very Deep Convolutional Neural Networks for Raw Waveforms
Wei Dai, Chia Dai, Shuhui Qu, Juncheng Li, Samarjit Das
Introduction
Acoustic modeling is traditionally divided into two parts: (1) designing a feature representation of the audio data, and (2) building a suitable predictive model based on the representation. However, it is often challenging and time-intensive to find the right representation in the so-called “feature-engineering” process, and the often heuristically designed features might not be optimal for the predictive task. Deep neural networks, which have achieved state-of-the-art performances in acoustic scene recognition and speech recognition , have increasingly blurred the line between representation learning and predictive modeling. Instead of using the hand-tuned Gaussian Mixture Model features and Mel-frequency cepstrum coefficients, neural network models can directly take as input features such as spectrograms and even raw waveforms . By using simpler features, deep neural networks can be viewed as extracting feature representation jointly with classification, rather than separately . This joint optimization is highly effective in speech recognition and image classification , among others.
A fundamental building block of these models is the convolutional neural networks (CNNs), which can learn spatially or temporally invariant features from pixels or time-domain waveforms. CNNs have famously achieved performance competitive or even surpassing human-level performance in the visual domains, such as object recognition and face recognition . A common theme among these powerful CNN models is that they are usually very deep, with the number of layers ranging from tens to even over a hundred. Nonetheless, designing and training a deep network suitable for a new application domain remain challenging.
Recent works have applied CNNs to audio tasks such as environmental sound recognition and speech recognition and found that CNNs perform well with just the raw waveforms . In one case, CNNs with time-domain waveforms can match the performance of models using conventional features like log-mel features . These works, however, have mostly considered only less deep networks, such as two convolutional layers .
In this work, we propose and study very deep architectures with up to 34 weight layers, directly using time-series waveforms as the input. Our deep networks are efficient to optimize over long sequences (e.g., vector of length 32000), necessary for processing raw audio waveforms. Our architectures use a very small receptive field in the convolutional layers, but a large receptive field in the first layer chosen based on the audio sampling rate to mimic bandpass filter. Our models are fully convolutional, without fully connected layers and dropout, in order to maximize the representation learning in the convolutional layers and can be applied to audio of varying lengths. By applying batch normalization , residual learning , and a careful design of down-sampling layers, we overcome the difficulties in training very deep models while keeping the computation cost low.
On an environmental sound recognition task , we show that deep networks improve the performance of networks with 2 convolutional layers by over 15% in absolute accuracy. We further demonstrate that the performance of deep models using just the raw signal is competitive with models using log-mel features . To our knowledge, this is the first report of a parity performance between log-mel features and raw time signal for environmental sound recognition.
Very Deep Convolutional Networks
Table 1 outlines the 5 architectures we consider. Our architectures take as input time-series waveforms, represented as a long 1D vector, instead of hand-tuned features or specially designed spectrograms. Key design elements are:
Deep architectures. To build very deep networks, we use very small receptive field 3 for all but the first 1D convolutional layersSmall receptive fields were first popularized by for 2D images.. This reduces the number of parameters in each layer and control the model sizes and computation cost as we go deeper. Furthermore, we aggressively reduce the temporal resolution in the first two layers by 16x with large convolutional and max pooling strides to limit the computation cost in the rest of the network . After the first two layers, the reduction of resolution is complemented by a doubling in the number of feature mapsIn the visual domain this change in resolution and the number of features maps leads to more specialized filters at the higher layers (e.g., feature maps responding to faces) and more basic filters at the bottom (e.g., feature maps responding diagonal lines).. We use rectified linear units (ReLU) for lower computation cost, following .
Fully convolutional networks. Most deep convolutional networks for classification use 2 or more fully connected (FC) layers of high dimensions (e.g., 4096 in ) for discriminative modeling, leading to a very high number of parameters. We hypothesize that most of the learning occurs in the convolutional layers, and with a sufficiently expressive representation from convolutional layers, no FC layer is necessary. We therefore adopt a fully convolutional design for our network construction . Instead of FC layers, we use a single global average pooling layer which reduces each feature map into one float by averaging the activation across the temporal dimension. By removing FC layers, the network is forced to learn good representation in the convolutional layers, potentially leading to better generalization. We support this design decision in our evaluation and demonstrate that fully convolutional networks perform comparably or better compared with their counterparts endowed with FC layers.
First layer receptive field. Time-domain waveforms at a reasonable sampling rate (e.g. 8000Hz) over a few seconds could have very large number of samples along a single dimension. If we exclusively use small receptive field for all convolutional layers such as in , which uses 3x3 in pixel for all layers, our model would need many layers in order to abstract high level features, which could be computationally expensive. Furthermore, audio sampling rate could affect the receptive field size in the first layer, since a field size of 80 at 8kHz sampling rate is at a different length scale than at 16kHz sampling rate. We thus choose our first layer receptive field to cover a 10-millisecond duration, which is similar to the window size for many MFCC computation. In Section 3 we show that a much smaller or larger receptive field gives poor performance.
Batch Normalization. We adopt auxiliary layers called batch normalization (BN) that alleviates the problem of exploding and vanishing gradients, a common problem in optimizing deep architectures. BN normalizes the output of the previous layer so the gradients are well-behaved. This makes possible training very deep networks (M18, M34-res) that were not studied previously . Following , we apply BN on the output of each convolutional layer before applying ReLU non-linearity.
Residual Learning. Residual learning is a recently proposed learning framework to ease the training of very deep networks. Normally we train a block of neural network layers to fit a desired mapping of ( being the the input to the layers). In the residual framework, we instead let the block of layers approximate , the residual mapping. Residual learning is achieved through a skip connection in the residual block (“res-block”, Figure 1b). We apply residual learning in M34-res (Table 1).
Experiment Details
We use UrbanSound8k dataset which contains 10 environmental sounds in urban areas, such as drilling, car horn, and children playing . The dataset consists of 8732 audio clips of 4 seconds or less, totalling 9.7 hours. We use the official fold 10 to be our test set, and the rest for training and validation. For computational speed, the audio waveforms are down-sampled to 8kHz and standardized to 0 mean and variance 1. We shuffle the training data but do not perform data augmentation.
Additional Models. To aid analysis, we train variants of models in Table 1. The “fc” models replace global average pooling layer with 2 fully connected (FC) layers of dimension 1000 (Table 5), since many conventional deep convolutional networks use 2 FC layers of dimension in the thousands . Following these works we also use a dropout layer between each fully connected layers for regularization, with a dropout rate of 0.3. We insert a batch normalization layer after each fully connected layers to aid training. These models have substantially more parameters than the original models due to the FC layers (Table 5). Additionally, M3-big and M5-big (Table 4) are variants of M3 and M5, respectively, with 50% and 100% more filters (e.g., 384/256 filters in the first convolutional layer in M3-big/M5-big).
Results and Analyses
Table 2 shows the test accuracies and training time for models in Table 2. We first note that M3 perform very poorly compared with the other models, indicating that 2-layered CNNs are insufficient to extract discriminative features from raw waveforms for sound recognition. This is in contrast with models using the spectrogram as input, which achieve good performance with just 2 convolutional layers , and shows that applying CNN directly on time-series data is challenging. M3-big, a variant of M3 with 50% more filters and 2.5x more parameters, does not significantly improve the performance (Table 4), showing that shallow models have limited capacity to capture time-series inputs even with a larger model.
Deeper networks (M5, M11, M18, M34-res) substantially improve the performance. The test accuracy improves with increasing network depth for M5, M11, and M18. Our best model M18 reaches 71.68% accuracy that is competitive with the reported test accuracy of CNNs on spectrogram input using the same dataset Figure 4 in reports 68% accuracy using a baseline CNN model. We point out that we have a different evaluation scheme: we use the 10-th fold as test set, while performs 10-fold evaluation. Also we use sound at 8kHz sampling rate while they use the original 44.1kHz.. The performance increases cannot be simply attributed to the larger number of parameters in the deep models. For example, M5-big has 2.2M parameters (Table 4) but only achieves 63.30% accuracy, compared with the 69.07% by M11 (1.8M parameters). By using a very deep architecture, M18 outperforms M3 by as much as 15.56% in absolute accuracy, which shows that deeper architectures substantially improve acoustic modeling using waveforms. Furthermore, by using an aggressive down-sampling in the initial layers, very deep networks can be economical to train (Table 2 Time column). When we use stride 1 instead of 4 in the first convolutional layer for M11, we observe a 3.5x increase in training time but a lower test accuracy (67.37%) after 10 hours of training, compared with 68.42% test accuracy reached in 2 hours by M18.
Interestingly, the performance improves with depth up to M18, at 71.68% test accuracy. M34-res only achieves 63.47% test accuracy. This is due to overfitting. We observe that with residual learning we have no problem optimizing deep networks like M34-res, and M34-res reaches an extremely high training accuracy of 99.21%, compared with 96.72% training accuracy by M18. We also observe overfitting in a residual variant of M11 network (not shown here) which reaches higher training accuracy but a lower test accuracy (by 0.17%). Overfitting caused by very deep networks is well documented . We believe that our dataset is too small to train M34-res without further regularization. Nonetheless, M34-res still outperforms M3 and M5.
We compare our fully convolutional network with conventional networks that use large fully connected layers (FC) for classification. Table 5 shows that FC layers can increase number of parameters significantly and increase training time by 295%. However, FC layers do not improve test accuracy, and in the cases of M3-fc and M11-fc the additional FC layers lead to lower test accuracy (i.e., poorer generalization). We believe that the lack of FC layers in our network design pushes learning down to convolutional layers, leading to better representation and generalization.
To understand the effect of the receptive field (RF) size in the first convolutional layer, we train M11-srf and M18-srf, variants of M11 and M18 with RF 8, and M11-lrf and M18-lrf with RF 320. Table 4 shows that the performance degrades significantly by up to 6.6% compared with M11 and M18 with RF 80. Previous works have shown that the first convolutional layer, when trained on raw waveforms, mimics wavelet transforms . Our results suggest that a small RF popularized by vision models is insufficient to capture the necessary bandpass filter characteristics in the first convolutional layer, while a large RF smooths out local structures and cannot effectively detect local impulse patterns.
We study the effect of batch normalization (BN) in optimizing very deep networks (Table 6). Without BN, both M11-no-bn and M18-no-bn can be optimized to high training accuracy. Note that M18-no-bn results in lower test accuracy, indicating that BN has a regularization effect . M34-no-bn could not be optimized without BN and performs close to random guess (10%) after 159 epochs of training.
Fig. 2 shows the learned kernels for M18 variants with different RF sizes in the first convolution layer. All of them learn a filter bank of bandpass filter. M18 (Fig. 2 left) has well-distributed filters. In contrast, the small RF model (Fig. 2 middle) has much more dispersed bands, and thus lower frequency resolution for subsequent layers. Conversely, large RF model (Fig. 2 right) has fine-grained filters, but does not have sufficient filters in the high frequency range, showing that it cannot effectively respond to local high frequency impulses.
Conclusion
In this work, we propose very deep convolutional neural networks that operate directly on acoustic waveform inputs. Our networks, up to 34 weight layers, are efficient to optimize, thanks to the combination of batch normalization, residual learning, and down-sampling. We use a broad receptive field (RF) in the first convolutional layer and narrow RFs in the rest of the network. Our results show that a deep network with 18 weight layers outperforms networks with 2 convolutional layers by 15.56% accuracy absolutely and achieves accuracy, competitive with CNNs using log-mel spectrogram inputs . Our fully convolutional networks compare favorably with those with fully connected layers. Our proposed deep architectures hold the promise to improve CNNs for speech recognition and other time-series modeling.
Acknowledgement
This work is supported by contract FA8702-15-D-0002 with Software Engineering Institute, a center sponsored by the United States Department of Defense.