AudioCLIP: Extending CLIP to Image, Text and Audio

Introduction

The latest advances of the sound classification community provided many powerful audio-domain models that demonstrated impressive results. Combination of widely known datasets – such as AudioSet , UrbanSound8K and ESC-50 – and domain-specific and inter-domain techniques conditioned the rapid development of audio-dedicated methods and approaches .

Previously, researchers were focusing mostly on the classification task using the audible modality exclusively. In the last years, however, popularity of multimodal approaches in application to audio-related tasks has been increasing . Being applied to audio-specific tasks, this implied the use of either textual or visual modalities in addition to sound. While the use of an additional modality together with audio is not rare, combination of more than two modalities is still uncommon in the audio domain. However, the restricted amount of qualitatively labeled data is constraining the development of the field in both, uni- and multimodal directions. Such a lack of data has challenged the research and sparked a cautious growth of interest for zero- and few-shot learning approaches based on contrastive learning methods that rely on textual descriptions .

In our work, we propose an approach to combine a high-performance audio model – ESResNeXt – into a contrastive text-image model, namely CLIP , thus, obtaining a tri-modal hybrid architecture. The base CLIP model demonstrates impressive performance and strong domain adaptation capabilities that are referred as “zero-shot inference” in the original paper . To keep consistency with the CLIP terminology, we use the term “zero-shot” in the sense defined in .

As we will see, the joint use of three modalities during the training results in out-performance of previous models in environmental sound classification task, extends zero-shot capabilities of the base architecture to the audio modality and introduces an ability to perform cross-modal querying using text, image and audio in any combination.

The remainder of this paper is organized as follows. In Section 2 we discuss the current approaches to handle audio in a standalone manner as well as jointly with additional modalities. Then, we describe models that serve as a base of our proposed hybrid architecture in Section 3, its training and evaluation in Section 4 and the obtained results in Section 5. Finally, we summarize our work and highlight follow-up research directions in Section 6.

Related Work

In this section, we provide an overview of the audio-related tasks and approaches that are intersecting in our work. Beginning with description of the environmental sound classification task, we connect it to the zero-shot classification through the description of existing methods to handle multiple modalities in a single model.

The environmental sound classification task implies an assignment of correct labels given samples belonging to sound classes that surround us in the everyday life (e.g., “alarm clock”, “car horn”, “jackhammer”, “mouse clicking”, “cat”). To successfully solve this task, different approaches were proposed that included the use of one- or two-dimensional Convolutional Neural Networks (CNN) operating on static or trainable time-frequency transformation of raw audio. While the first approaches relied on the task-specific design of models, the latter results confirmed that the use of domain adaptation from visual domain is beneficial . However, the visual modality was used in a sequential way, implying the processing of only one modality simultaneously.

The joint use of several modalities occurred first in video-related tasks and was adapted to the sound classification task later . However, despite the multimodal design, such approaches utilized two modalities simultaneously at most, while recent studies suggest that the use of more modalities is beneficial .

The multimodal approaches described above share a common key idea of contrastive learning. Such a technique belongs to the branch of self-supervised learning that, among other features, helps to overcome the lack of qualitatively labeled data. That makes it possible to apply contrastive learning-based training to the zero-shot classification tasks .

Summarizing, our proposed model employs contrastive learning to perform training on textual, visual and audible modalities, is able to perform modality-specific classification or, more general, querying and is implicitly enabled to generalize to previously unseen datasets in a zero-shot inference setup.

Model

In this section, we describe the key components that make up the proposed model and the way how it handles its input. On a high level, our hybrid architecture combines a ResNet-based CLIP model for visual and textual modalities and an ESResNeXt model for audible modality, as can be seen in Figure 1.

Conceptually, the CLIP model consists of two subnetworks: text and image encoding heads. Both parts of the CLIP model were pre-trained jointly under natural language supervision . Such a training setup enabled the model to generalize the classification ability to image samples that belonged to previously unseen datasets according to the provided labels without any additional fine-tuning.

For the text encoding part, a slightly modified Transformer architecture was used . For the chosen 12-layer model, the input text was represented by a lower-case byte pair encoding with a vocabulary of size $49\,408$ . Due to computational constraints, the maximum sequence length was clipped at 76 .

For the image encoding part of the CLIP model, two different architectures were considered. One was a Vision Transformer (ViT) , whose architecture made it similar to the text-head. Another option was represented by a modified ResNet-50 , whose global average pooling layer was replaced by a QKV-attention layer . As we mentioned in Section 3.1, for the proposed hybrid model we chose the ResNet-based variant of the CLIP model because of its lower computational complexity, in comparison to the ViT-based one.

Given an input batch (text-image pairs) of size $N$, both CLIP-subnetworks produce the corresponding embeddings that are mapped linearly into a multimodal embedding space of size $1\,024$ . In a such setup, CLIP learns to maximize the cosine similarity between matching textual and visual representations, while minimizing it between incorrect ones, which is achieved using symmetric cross entropy loss over similarity measures .

2 ESResNeXt

For the audio encoding part, we decided to apply ESResNeXt model that is based on the ResNeXt-50 architecture and consists of a trainable time-frequency transformation based on complex frequency B-spline wavelets . The chosen model contains moderate number of parameters to learn ($\sim 30$ M), while performing competitive on a large-scale audio dataset, namely AudioSet , and providing state-of-the-art-level classification results on the UrbanSound8K and ESC-50 datasets. Additionally, the ESResNeXt model supports an implicit processing of a multi-channel audio input and provides improved robustness against additive white Gaussian noise and sample rate decrease .

3 Hybrid Model – AudioCLIP

In this work, we introduce an additional – audible – modality into the novel CLIP framework, which is naturally extending the existing model. We consider the newly added modality as equally important as the originally present. Such a modification became possible through the use of the AudioSet dataset that we found suitable for this, as described in Section 4.1.

Thus, the proposed AudioCLIP model incorporates three subnetworks: text-, image- and audio-heads. In addition to the existing text-to-image-similarity loss term, there are two new ones introduced: text-to-audio and image-to-audio. The proposed model is able to process all three modalities simultaneously, as well as any pair of them.

Experimental Setup

In this section, we describe datasets that were used, data augmentation methods we applied, the training process and its corresponding hyper-parameters, finalizing with the performance evaluation methods.

In this work, five image, audio and mixed datasets were used directly and indirectly. Here, we describe these datasets and define their roles in the training and evaluation processes.

For this work, we acquired video frames in addition to audio tracks. Thus, the AudioSet dataset became the glue between the vanilla CLIP framework and our tri-modal extension on top of it. In particular, audio tracks and the respective class labels were used to perform image-to-audio transfer learning for the ESResNeXt model, and then, the extracted frames in addition to audio and class names served as an input for the hybrid AudioCLIP model.

During the training part, ten equally distant frames were extracted from a video recording, and one of them was picked randomly ($\sim\mathcal{U}$) and passed through the AudioCLIP model. In the evaluation phase, the same extraction procedure was performed, with the difference that only a central frame was presented to the model. Performance metrics are reported based on the evaluation set of the AudioSet dataset.

On this dataset, we performed zero-shot inference using the AudioCLIP model trained on AudioSet. Also, the audio encoding head was fine-tuned to the UrbanSound8K dataset in both, standalone and cooperative fashion, and the classification performance in both setups was assessed.

On this dataset, we performed zero-shot inference using the AudioCLIP model trained on AudioSet. Also, the audio encoding head was fine-tuned to the ESC-50 dataset in both, standalone and cooperative fashion, and the classification performance in both setups was assessed.

2 Data Augmentation

In comparison to the composite CLIP dataset (Section 4.1), the audio datasets provide two orders of magnitude less training samples, which makes overfitting an issue, especially for the UrbanSound8K and ESC-50 datasets. To address this challenge, several data augmentation techniques were applied that we describe in this section.

Time Scaling: Simultaneous change of track duration and its pitch is achieved using random scaling along the time axis. This kind of augmentation combines two computationally expensive ones, namely time stretching and pitch shift. Being a faster alternative to the combination of the aforementioned techniques, the time scaling in the range of random factors $[-1.5,1.5],\>\sim\mathcal{U}$ provides a lightweight though powerful method to fight overfitting .

Time Inversion: Inversion of a track along its time axis relates to the random flip of an image, which is an augmentation technique that is widely used in the visual domain. In this work, random time inversion with the probability of $0.5$ was applied to the training samples similarly to .

Random Crop and Padding: Due to the requirement to align track duration before the processing through the model we applied random cropping or padding to the samples that were longer or shorter than the longest track in a non-augmented dataset, respectively. During the evaluation phase, the random operation was replaced by the center one.

3 Training

The entire training process was divided into subsequent steps, which made acquisition of the final AudioCLIP model reliable and assured its high performance. As described in Section 3.1, we took a ResNet-based CLIP text-image-model pre-trained on its own dataset (Section 4.1) and combined it with the ESResNeXt audio-model initialized using ImageNet weights and then pre-trained on the AudioSet dataset .

While the CLIP model was already pre-trained on text-image pairs, we decided to perform an extended AudioSet pre-training of the audio-head first, as it improved performance of the base ESResNeXt model (Table 1), and then to continue training in a tri-modal setting combining it with two other heads. Here, the whole AudioCLIP model was trained jointly on the AudioSet dataset using audio snippets, the corresponding video frames and the assigned textual labels.

Finally, audio-head of the trained AudioCLIP model was fine-tuned on the UrbanSound8K and ESC-50 datasets in a bimodal manner (audio and text) using sound recordings and the corresponding textual labels.

The trained AudioCLIP model and its audio encoding head were evaluated on the ImageNet dataset as well as on the three audio-datasets: AudioSet, UrbanSound8K, and ESC-50.

The initialization of the audio-head’s parameters was split into two steps. First, the ImageNet-initialized ESResNeXt model was trained on the AudioSet dataset in a standalone fashion. Then, the pre-trained audio-head was incorporated into the AudioCLIP model and trained further under the cooperative supervision of the text- and image-heads.

Standalone: The first pre-training step implied the use of the AudioSet dataset as a weight initializer. Here, the ESResNeXt model was trained using the same setup as described in , with the difference in the number of training epochs. In this work, we increased the training time, which turned out into better evaluation performance on the AudioSet dataset and the subsequent downstream tasks, as described in Section 5.1 and independently quantified.

Cooperative: The further pre-training of the audio-head was done jointly with the text- and image-heads. Here, the pre-trained (in a standalone manner) audio-head was modified slightly through the replacement of its classification layer with a randomly initialized one, whose number of output neurons was the same as the size of CLIP’s embedding space.

In this setup, the audio-head was trained as a part of the AudioCLIP model, which made its outputs compatible with the embeddings of the vanilla CLIP model. Parameters of the two other subnetworks, namely text- and image-head, were frozen during the cooperative pre-training of the audio encoding head, thus, these heads served as teachers in a multi-modal knowledge distillation setup.

The performance of the AudioCLIP model trained in such a fashion was assessed and is described in Section 5.

3.2 AudioCLIP Training

The joint training of the audio-head made it compatible with the vanilla CLIP model, however, the distribution of images and textual descriptions in the AudioSet dataset does not follow the one from the CLIP dataset. This could lead to suboptimal performance of the resulting AudioCLIP model on the target dataset as well as on the downstream tasks.

To address this issue, we decided to perform the training of the whole tri-modal model on the AudioSet dataset. Here, all three modality-dedicated heads were tuned together, making the resulting model take into account the distributions of images and textual descriptions (video frames and names of the assigned AudioSet classes, respectively), in addition to the distribution of audio samples. The influence of the whole model training on the network’s performance in comparison to the audio-head-only training is described in Section 5.

3.3 Audio-Head Fine-Tuning

The trained AudioCLIP model provides general-purpose multimodal classification, or more general, querying abilities. However, under some conditions, it is required to acquire a more domain-specific model, which is able to distinguish concepts that differ just slightly.

To address this need, we performed experiments on tuning of the audio encoding head to two target datasets: UrbanSound8K and ESC-50.

Standalone: The ESResNeXt model that served as the audio-head demonstrated strong classification abilities on the chosen downstream tasks . As we performed the AudioSet pre-training step instead of using a pre-trained ESResNeXt model, we fine-tuned it to the UrbanSound8K and ESC-50 datasets as well, in order to assess the change of the classification accuracy.

The fine-tuning step was done the same way as in , which implied the replacement of the classification layer with a randomly initialized one, whose number of outputs was defined by the number of targets in the downstream task. We report the performance of the fine-tuned ESResNeXt model in Section 5.1.1.

Cooperative: During the fine-tuning of the AudioCLIP model to the downstream tasks, only the parameters of the audio-head were being updated, so the text- and image-heads were frozen at this step. In comparison to the AudioSet training, which implied a multi-label setup, the corresponding textual class labels from the UrbanSound8K and ESC-50 datasets were represented by one class per audio sample.

For the fine-tuned AudioCLIP model, we assess the downstream classification performance as well as the querying performance, as described in Section 5.2.

4 Hyper-Parameters

In this work, we trained our model on the AudioSet, UrbanSound8K and ESC-50 datasets. The required hyper-parameters are reported in the current section.

In all training phases, the model parameters were optimized using Stochastic Gradient Descent optimizer with Nesterov’s momentum of $0.9$, weight decay of $5\cdot 10^{-4}$ and batch size of $64$.

The learning rate value decreased exponentially, varying its value $\eta$ and the decrease factor $\gamma$ from $10^{-4}$ and $0.95$, respectively, during the standalone pre-training of the audio-head to $5\cdot 10^{-5}$ and $0.98$ during the fine-tuning of the AudioCLIP model to the downstream tasks.

The number of epochs was set to $30$ for the AudioSet-based training, and to $50$ for the fine-tuning to the downstream tasks.

5 Performance Evaluation

The model performance was assessed based on two tasks: classification and querying. While the evaluation of the first was possible for both, audio-head itself and the full AudioCLIP model, the performance on the latter task was assessed for the multimodal network only.

The evaluation of the classification performance was done using the AudioCLIP model as well as its audio-head, namely ESResNeXt. The latter predicted the class labels directly, as the number of its outputs was equal to the number of targets in the datasets. For the AudioCLIP model, the classification task implied an intermediate step, which included construction of a target from textual labels .

In this work, the performance of the proposed model was evaluated after the training on the AudioSet dataset given audio and/or image as an input. For the UrbanSound8K and ESC-50 datasets, two downstream tasks were evaluated: classification after the training on the target dataset and without the training. The corresponding accuracies are reported in Section 5.1.

5.2 Querying

The multimodal nature and symmetry of AudioCLIP allowed to perform querying of samples represented by another modality. Here, classification can be considered as a sub-task of querying, whose query consists of image and/or audio while the result is represented by text.

In this work, we assessed the querying performance of the trained AudioCLIP model on the ImageNet, AudioSet, UrbanSound8K and ESC-50 datasets. The results include Top-1 Precision/Recall (P@1/R@1) and Mean Average Precision (mAP), and presented in Section 5.2.

Results

The extended pre-training ($30$ epochs instead of $5$) on the AudioSet dataset provided an audio encoding head that had increased performance, in comparison to the original training (from $28.17\,\%$ to $34.14\,\%$, mAP). Such an improvement was also beneficial for the downstream tasks, making the newly trained audio-head to out-perform its base variant on the UrbanSound8K and ESC-50 datasets and achieving accuracy of $89.49\,\%$ and $95.90\,\%$, respectively (Table 1).

1.2 AudioCLIP

Our tri-modal training setup – through the use of video frames – introduced more diversity into audio-head’s target distribution, thus fighting the overfitting issue and further increasing performance in audio classification tasks, in comparison to the audio-only ESResNeXt model. Also, the joint training of all three heads provided an additional performance boost and the ability to use multiple modalities to perform classification, as well as the zero-shot inference capabilities on previously unseen datasets (Table 2).

Partial Training: The training of the audio-head under the supervision of the text- and image-heads already allows to out-perform current state-of-the-art results on the UrbanSound8K and ESC-50 datasets by achieving accuracy of $89.95\,\%$ and $96.65\,\%$, respectively.

Moreover, even the partial training of the AudioCLIP model sets new highest zero-shot classification accuracy on the ESC-50 dataset ($69.40\,\%$, Table 3) as well as out-performs performance of the commonly trained baseline CNN ($64.50\,\%$, Table 3).

Full Training: The joint training of the AudioCLIP model provides further performance improvements in comparison to the partial one. Such a trained AudioCLIP model sets the new state-of-the-art classification accuracy on the UrbanSound8K and ESC-50 datasets ($90.07\,\%$ and $97.15\,\%$, respectively). Also, given the full model training setup, a new zero-shot classification baseline was set for the UrbanSound8K dataset ($68.78\,\%$, Table 3).

2 Querying

The original CLIP model introduced the ability to perform querying using both supported modalities – text and image – in any direction. Given a query (e.g., text), model provided similarity scores for the samples represented by another (visual) modality. Thus, given a dataset and a modality, the set of queries was defined by the unique samples of the chosen modality. In this work, we added the support of audio, enabling the model to query between text, images and audio in any combination. We evaluated the querying performance on the ImageNet, AudioSet, UrbanSound8K and ESC-50 datasets and summarized it in Table 4.

Image by Text: In this setup, all unique sets of class names assigned to the samples from a target dataset were collected and served as textual queries while the results were represented by images (ImageNet, AudioSet). Thus, only the visual samples possessing the same set of labels were considered as relevant results.

For the AudioSet dataset, the full training contributed to the increase of the performance score measured by mAP. However, such a training led to the decrease of the querying performance on the ImageNet dataset, as its distribution is likely different from the AudioSet one.

Audio by Text: Having the same type of query as in the previous setup, here, the result was represented by an audio recording and considered correct if the labels matched the query.

On the AudioSet and UrbanSound8K datasets, the full training increases the querying performance. For the ESC-50 dataset it is not the case, however, the gap is not large and is close to be marginal.

Audio by Image and Vice Versa: For both types of queries – audio by image and image by audio – the full training of the AudioCLIP model was beneficial in terms of querying performance (mAP).

Conclusion

In this work, we extended the CLIP model from textual and visual modalities to audio using an effective sound classification model .

The proposed AudioCLIP model achieves new state-of-the-art classification results on two datasets: UrbanSound8K ($90.07\,\%$) and ESC-50 ($97.15\,\%$). To ease reproducibility, the details on hyper-parameters and implementation as well as weights of the trained models are made available for the communityhttps://github.com/AndreyGuzhov/AudioCLIP.

Additionally, for the zero-shot inference, our model out-performs previous approaches on the ESC-50 dataset with a large gap ($69.40\,\%$) and sets a baseline for the UrbanSound8K dataset ($68.78\,\%$).

We also evaluated the performance of our model on cross-modal querying tasks as well as the influence of the partial and full training on the results in classification and querying tasks.

In the future, we would like to further investigate the performance of the proposed AudioCLIP model on a wider variety of datasets and tasks. Also, changing the backbones of image- and audio-heads to more powerful networks could further improve the model performance.

References