Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection

Introduction

Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers (Roth et al., 2020). As the speaking circumstances are very fluid and change dynamically, ASD has to predict at a fine granularity in time, i.e., at video frame level. This task is an essential frontend for a wide range of multi-modal applications such as audio-visual speech recognition (Afouras et al., 2018c), speech separation (Pan et al., 2020), speaker diarization (Chung et al., 2020b; Chung et al., 2019b) and speaker tracking (Qian et al., 2021a; Qian et al., 2021b).

Among the factors, that humans judge whether a person is speaking, are 1) Does the audio of interest belong to human voice? 2) Are the lips of the person of interest moving? 3) If the above are true, is the voice synchronized with the lip movement? Based on this cognitive finding, there have been deep learning solutions that extract audio and visual features to make binary classification (Tao and Busso, 2017; Ariav and Cohen, 2019; Chung and Zisserman, 2016; Roth et al., 2020). Despite much progress, the existing ASD systems have not fully benefited from two aspects of available information: the temporal dynamics of audio and visual flow, and the interaction between audio and visual signals, that limits the scope of applications, especially for challenging real-world scenarios.

As the short-term audio and visual features represent the salient cues for ASD, most of the existing studies are focused on segment-level information, e.g., a video segment of 200 to 600 ms. However, as illustrated in Figure 1(a), it is hard to judge the speaking activity from a video segment of 200 ms, not to mention the audio-visual synchronization. A longer 2-second video, as displayed in Figure 1(b), would be more evident of the speaking episode. When humans are detecting an active speaker, we typically consider saying an entire sentence that spans over hundreds of video frames for a decision, for example, an audio-visual episode lasting 5 seconds contains 15 words on average (Cutts, 2020; Tauroza and Allison, 1990). A short-term segment of 200 ms doesn’t even cover a complete word. Furthermore, single modality embedding is not reliable in some challenging scenarios. For example, the voice we hear might come from someone else than the target speaker; at the same time, there could be false lip movements, e.g., laughing, eating, and yawning, that are not related to speaking. To summarize, we consider that the inter-modality synchronization, such as speech-lip, speech-face, over the span of an utterance provides more reliable information than short-term segments.

The systems with short-term features extract audio-visual embedding from a fixed-length short segment, e.g., 200 ms (Chung and Zisserman, 2016; Chung, 2019; Zhang et al., 2019), 300 ms (Tao and Busso, 2017), and 440 ms segment (Alcázar et al., 2020; León-Alcázar et al., 2021). By simply increasing the segment size, we are getting the average properties of the segment at the cost of the time resolution of speaking activities. A better way to capture the long-term temporal context is to encode the history of audio or video frame sequence. In this paper, we study an audio-visual ASD framework, denoted as TalkNet. For video signals, the minimum unit is a video frame, i.e., a static image. We study a temporal network to encode the temporal context over multiple video frames. For audio signals, the minimum unit is an audio frame of tens of milliseconds. We study an audio temporal encoder to encode the temporal context over multiple audio frames. In terms of backend classifier, we study an audio-visual cross-attention mechanism to capture inter-modality evidence and a self-attention mechanism to capture long-term speaking evidence.

To the best of our knowledge, this paper is the first study on the use of long-term temporal context, and audio-visual inter-modality interaction for ASD. We make the following contributions.

We propose a feature representation network to capture the long-term temporal context from audio and visual cues;

We propose a backend classifier network that employs audio-visual cross-attention, and self-attention to learn the audio-visual inter-modality interaction;

We propose an effective audio augmentation technique to improve the noise-robustness of the model.

The rest of the paper is organized as follows. In Section 2, we discuss the related work. In Section 3, we formulate the proposed TalkNet framework and present its training process. In Section 4 and Section 5, we report the experiments and their results, respectively. Finally, Section 6 concludes the study.

Related work

This research is built on prior studies on the detection of audio and visual events and the modeling long-term temporal dynamics of audio-visual signals.

There have been prior studies on ASD using audio, video, and the fusion of both. In voice activity detector (VAD), we study how to detect the presence of speech as opposed to other acoustic noises (Sehgal and Kehtarnavaz, 2018; Ding et al., 2020). However, in real-world scenarios, audio signals by distant microphones are inherently ambiguous because of the overlapping speech and the corruption from background noise, which poses challenges to the VAD task. For vision, the facial (Patrona et al., 2016) and upper-body (Shahid et al., 2021; Chakravarty et al., 2015) movements are analyzed to detect if a visible person is speaking. However, the performance is limited due to weak correlation between the body motion and speaking activities. Besides, non-speaking activities, e.g., licking lips, eating food and grinning, may also degrade the ASD performance. Despite these limitations, the audio or visual single modal solutions serve as the foundation for ASD.

Audio-visual processing has seen significant benefits through modality fusion (Minotto et al., 2015; Ephrat et al., 2018). As the speech rhythm and word pronunciation are closely correlated with facial motion, an interesting and promising alternative is to combine both audio and vision information to perform ASD. Exploring audio-visual ASD, one approach is to view it as an assignment task. It is assumed that the detected speech must belong to one of the speakers on the screen (Alcázar et al., 2020; León-Alcázar et al., 2021). However, this assumption does not always hold because there could be cross talk or off-screen speech in practice. Another approach is to perform ASD as a classification task to evaluate the visible face on the screen one-by-one. Some studies (Chung and Zisserman, 2016; Chung, 2019; Afouras et al., 2020) simply concatenate the extracted audio and visual features as the input, and apply a multi-layer perceptron (MLP)-based binary classifier to detect the active speaker at each short video segment, without considering the inter-frame temporal dependency. Others further adopt the backend classifier with temporal structure like recurrent neural network (RNN) (Tao and Busso, 2017, 2019), gated recurrent unit (GRU) (Roth et al., 2020) and long short-term memory (LSTM) (Zhang et al., 2019; Sharma et al., 2020; Ariav and Cohen, 2019), which have achieved preliminary success. Our proposed TalkNet is motivated by this thought.

2. Long-term Temporal Context

As ASD seeks to make a fine-grained decision at audio or video frame level, most of the prior studies employ short-term features and make decisions at split segments of less than 600 ms. While smoothing method can be used to aggregate short-term decisions for for long-term video (Chung, 2019; Alcázar et al., 2020), the potential of long-term features has not been fully explored yet.

It is common that ASD uses individual uni-modal frontend feature extractors to learn the audio and visual embeddings, that is followed by the backend classifier to incorporate audio-visual synchrony. For the uni-modal representation learning, the utterance-level model performs better than the frame-level model in the audio tasks such as audio classification (Pons and Serra, 2019; Fernandes et al., 2021). The recent studies in video object detection also show that it is beneficial to leverage the temporal context at the proposal-level by end-to-end optimization to learn the completed video presentations (Shvets et al., 2019; Wu et al., 2019). As their short-term embeddings encode long-term temporal context, such techniques generally provide improved performance when making short-term decisions. The success in these studies motivates us to consider encoding long-term audio and visual temporal context at the utterance level for ASD task.

On the other hand, audio-visual ASD takes advantage of the cross-modal synchronization information. In audio-visual synchronization studies, by using convolutional neural network (CNN) (Owens and Efros, 2018; Chung et al., 2019a), LSTM (Shalev and Wolf, 2020) or attention model (Cheng et al., 2020), the longer video utterance are used, the more representative features can be extracted, which eventually boost the performance (Chung and Zisserman, 2016; Chung et al., 2019a). These studies demonstrate that long-term temporal context is significantly important to learn the audio-visual relationship in ASD. As ASD aims to learn the modality feature and the audio-visual relationship, we believe it will benefit from long-term temporal context either from intra-modal signals or inter-modal signals.

TalkNet

TalkNet is an end-to-end pipeline that takes the cropped face video and corresponding audio as input, and decide if the person is speaking in each video frame. It consists of a feature representation frontend, and a speaker detection backend classifier, as illustrated in Figure 2. The frontend contains an audio temporal encoder and a video temporal encoder. They encode the frame-based input audio and video signals into the time sequence of audio and video embeddings, that represent temporal context. The backend classifier consists of an inter-modality cross-attention mechanism to dynamically align audio and visual content, and a self-attention mechanism to observe speaking activities from the temporal context at the utterance level.

The visual temporal encoder aims to learn the long-term representation of facial expression dynamics. As illustrated in Figure 3, it consists of the visual frontend and the visual temporal network. We seek to encode the visual stream into a sequence of visual embeddings $F_{v}$ that have the same time resolution.

The visual frontend explores spatial information within each video frame. It consists of a 3D convolutional layer (3D Conv) followed by a ResNet18 block (Afouras et al., 2018a). This frontend encodes the video frame stream into a sequence of frame-based embedding. The visual temporal network consists of a video temporal convolutional block (V-TCN), which has five residual connected rectified linear unit (ReLU), batch normalization (BN) and depth-wise separable convolutional layers (DS Conv1D) (Afouras et al., 2018c), followed by a Conv1D layer to reduce the feature dimension. It aims to represent the temporal content in a long-term visual spatio-temporal structure. For example, for a visual temporal encoder that has a receptive field of 21 video frames, we take a segment of up to 840 ms to encode a video embedding when the video frame rate is 25 frame-per-second (fps).

2. Audio Temporal Encoder

The audio temporal encoder seeks to learn an audio content representation from the temporal dynamics. It is a 2D ResNet34 network with squeeze-and-excitation (SE) module (Hu et al., 2018) introduced in (Chung et al., 2020a). An audio frame is first represented by a vector of Mel-frequency cepstral coefficients (MFCCs). The audio temporal encoder takes the sequence of audio frames as the input, generate the sequence of audio embeddings $F_{a}$ as the output. The ResNet34 are designed with dilated convolutions such that the time resolution of audio embeddings $F_{a}$ matches that of the visual embeddings $F_{v}$ to facilitate subsequent attention mechanism. For example, the audio temporal encoder has a receptive field of 189 audio frames. In other words, we take a segment of 1,890 ms to encode an audio embedding, when the MFCC window step is 10 ms, to capture the long-term temporal context.

3. Audio-visual Cross-Attention

$F_{a}$ and $F_{v}$ are expected to characterize the events that are relevant to speaking activities for audio and visual, respectively. We are motivated by the fact that audio-visual synchronization is an informative cue for speaking activities as well. As audio and visual flow each has its own dynamics, they are not exactly time aligned. The actual audio-visual alignment may depend on the instantaneous phonetic content and the speaking behavior of the speakers. We propose two cross-attention networks along the temporal dimension to dynamically describe such audio-visual interaction.

The core part of the cross-attention network is the attention layer, which is shown in Figure 4 (a). The inputs are the vectors of query ($Q_{a},Q_{v}$), key ($K_{a},K_{v}$), and value ($V_{a},V_{v}$) from audio and visual embeddings, respectively, projected by a linear layer. The outputs are the audio attention feature $F_{a\rightarrow v}$ and visual attention feature $F_{v\rightarrow a}$ as formulated in Eq. (1) and Eq. (2), where $d$ denotes the dimension of $Q$, $K$ and $V$.

As formulated in Eq. (1) and Eq. (2), to learn the interacted new audio feature $F_{a\rightarrow v}$, the attention layer applies $F_{v}$ as the target sequence to generate query, and $F_{a}$ as the source sequence to generate key and value, and to learn $F_{v\rightarrow a}$ vice versa. The attention layer is followed by the feed-forward layer. Residual connection and layer normalization are also applied after these two layers to generate the whole cross-modal attention network. The outputs are concatenated together along the temporal direction.

4. Self-Attention and Classifier

A self-attention network is applied after the cross-attention network to model the audio-visual utterance-level temporal information. As illustrated in Figure 4 (b), this network is similar to the cross-attention network except that now the query ($Q_{av}$), key ($K_{av}$) and value ($V_{av}$) in the attention layer all come from the joint audio-visual feature $F_{av}$. With the self-attention layer, we seek to distinguish the speaking and non-speaking frames.

5. Loss Function

We finally apply a fully connected layer followed by a softmax operation to project the output of the self-attention network to an ASD label sequence. We view ASD as a frame-level classification task. The predicted label sequence is compared with the ground truth label sequence by cross-entropy loss. The loss function is presented in Eq (3), where $s_{i}$ and $y_{i}$ are the predicted and the ground truth ASD labels of $i^{{\text{th}}}$ video frame, $i\in[1,T]$. $T$ refers to the number of video frames.

6. Audio Augmentation with Negative Sampling

The noise-robustness of ASD in the presence of noise and interference speakers remains a challenging topic. One traditional audio augmentation method is to use a large noise dataset (Snyder et al., 2018) to augment the training data by overlaying the noise on top of the original sound track. This method involves the external data source to increase the diversity. However, it is not straightforward to find such acoustic data that matches the video scenes.

To increase the amount of samples, we propose a negative sampling method to offer a simple yet effective solution. In practice, we use one video as the input data during training, and then we randomly select the audio track from another video in the same batch as the noise to perform audio augmentation. Such augmented data effectively have the same label, e.g., active speaker or inactive speaker, as the original sound track. This approach involves the in-domain noise and interference speakers from the training set itself. It does not require data outside the training set for audio augmentation.

Experiments

The AVA-ActiveSpeaker datasethttps://research.google.com/ava/download.html##ava_active_speaker_download is derived from Hollywood movies (Roth et al., 2020). It contains $29,723$, $8,015$ and $21,361$ video utterances in the training, validation and test sets, respectively. The video utterances range from 1 to 10 seconds and are provided as face tracks. We follow the official evaluation tool and report the performance in terms of mean average precision (mAP).

There are several challenges involved in the AVA-ActiveSpeaker dataset. The language is diverse, and the frame per second (fps) of the movies varies. Furthermore, a significant number of videos have blurry images and noisy audio. It also contains many old movies with dubbed dialogues. All these factors make it hard to accurately synchronize the audio-visual signals.

1.2. Columbia Active Speaker Dataset

The Columbia ASD datasethttp://www.jaychakravarty.com/active-speaker-detection/ is a standard benchmark test dataset for ASD (Chakravarty and Tuytelaars, 2016). It contains an 87-minute panel discussion video, with 5 speakers taking turns to speak, in which 2-3 speakers are visible at any given time. We follow the common protocol of this benchmark to use F1 score as the evaluation metric. The Columbia ASD dataset doesn’t provide a common splitting between training and test sets.

1.3. TalkSet

Due to its limited size, the Columbia ASD dataset is usually only used as a test set. Furthermore, the AVA-ActiveSpeaker dataset is labelled with face bounding boxes with a different algorithm, which are incompatible with those of the Columbia ASD dataset. We are motivated by the call for an audio-visual ASD dataset that covers real-world scenarios. This leads to the idea of a new database. We leverage two large-scale audio-visual datasets in the wild, LRS3 (Afouras et al., 2018b) and VoxCeleb2 (Chung et al., 2018), to form a new ASD dataset, named “TalkSet”, that covers all valid ASD conditions.

First, we consider that humans detect active speakers by examining three aspects of a video, 1) On audio signal, is there an active voice? 2) For visual signal, are the lips of someone moving? 3) When there is an active voice and the lips of someone are moving, is the voice synchronized with the lips movement? The above three cues lead to five valid conditions in the real world, which are summarized in Table 1.

We select 90,000 videos with active voice from VoxCeleb2https://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox2.html (Chung et al., 2018). We also collect 60,000 videos without an active voice, at the same time, longer than one second from LRS3https://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrs3.html (Afouras et al., 2018b) using the Kaldi-based voice activity detection system (Povey et al., 2011). In total, we have got 150,000 videos that range from 1 to 6 seconds. The total length of these videos is 151.65 hours, out of which 71.45 hours are speaking and 80.20 hours are non-speaking. We randomly split it into 135,000 videos for training and 15,000 videos for validation. Finally, we adopt the Columbia ASD dataset as the test data.

We understand that both LRS3 and VoxCeleb2 use the S3FD face detection method (Zhang et al., 2017) to provide ground truth face tracking of the speakers. To be consistently, we also apply the same method for face tracking on unknown test videos, including the Columbia ASD dataset.

2. Implementation Details

We build the TalkNet using the PyTorch library with the Adam optimizer. The initial learning rate is $10^{-4}$, and we decrease it by $5\%$ for every epoch. The dimension of MFCC is $13$. All the faces are reshaped into $112\times 112$. We set the dimensions of the audio and visual feature as 128. Both cross-attention and self-attention network contain one transformer layer with eight attention heads.

We randomly flip, rotate and crop the original images to perform visual augmentation. As the Columbia ASD dataset is an open dataset, we apply the additional sources from RIRs data (Ko et al., 2017) and the MUSAN dataset (Snyder et al., 2015) to perform audio augmentation on the TalkSet during training, and evaluate the performance using Sklearn libraryhttps://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html. For the AVA-ActiveSpeaker dataset, we apply the proposed negative sampling technique to add the in-domain noise from the training set itself. Finally, we evaluate the performance on the test set using the official toolhttps://github.com/activitynet/ActivityNet. We also evaluate the performance on the validation set as it comes with the ground truth labels for quick examination.

Results

We now compare the proposed TalkNet with the state-of-the-art systems on both the AVA-ActiveSpeaker and Columbia ASD dataset. First, we summarize the results on the AVA-ActiveSpeaker dataset in Table 2. We observe that TalkNet achieves $92.3\%$ mAP and outperforms the best competitive system, i.e., MAAS-TAN (León-Alcázar et al., 2021), by $3.5\%$ on the validation set. Some studies report their results in terms of Area under the Curve of ROC (AUC) on the same validation set. For ease of comparison, we also report the comparison of AUC results in Table 3. Without surprise, the TalkNet also achieves $3.6\%$ improvement over the best reported AUC, cf., Huang et al. (Huang and Koishida, 2020).

As the ground truth labels of the AVA-ActiveSpeaker test set are not available to the public, we obtain the evaluation results in Table 4 on the test set with the assistance of the organizer. Our $90.8\%$ mAP also outperforms the best prior work by $3.0\%$, cf., Chung et al. (Chung, 2019).

Note that some prior studies (León-Alcázar et al., 2021; Alcázar et al., 2020) applied additional networks to learn the relationship among the cropped face videos. Others (Chung, 2019; Zhang et al., 2019) used the pre-trained model in another large-scale dataset. By contrast, TalkNet only uses the AVA-ActiveSpeaker training set to train the single face videos from scratch without any additional post-processing. We believe that pre-training and other advanced techniques will further improve TalkNet, which is beyond the scope of this paper.

We then evaluate TalkNet on the Columbia Active Speaker Detection dataset. Its performance along with comparison to other existing methods are shown in Table 5. We observe that the F1 score, which is the standard metric in this benchmark, is the maximum for proposed TalkNet, which is $96.2\%$ for the average result that has an improvement over the best existing system by $2.2\%$. For all the five speakers, TalkNet provides the best performance for three of them (Bell, Lieb and Sick). It is noted that Columbia ASD is an open-training dataset, so the methods in Table 5 are trained on different data, so we only claim that our TalkNet is efficient on the Columbia ASD dataset.

2. Ablation Study

We further perform experiments to analyze the contributions of individual techniques deployed in TalkNet.

The prior studies usually use short-term features of 5 to 13 video frames on the AVA-Activespeaker dataset (Chung, 2019; León-Alcázar et al., 2021; Alcázar et al., 2020) for video embedding. We believe that long-term features are more evident of speaking episode. To study the difference between long-term and short-term features, we use a fixed number of $N$ frames instead of the entire video sequence during training and testing, where $N$ is chosen from 5,10,25,50,100 that amounts to 0.2, 0.4, 1, 2 and 4 second.

We report the evaluation results of TalkNet in Table 6, and observe that the system can hardly work with very short video segment, e.g., when $N=5$, as there is not enough temporal context in a 0.2-second segment. As the duration of video increases, mAP improves consistently from 75.2% to 89.4%.

As we increase the duration of videos, there are a fewer number video segments for training. As a result, we don’t observe improvement from 50 frames to 100 frames of video duration. This study confirms our hypothesis that the long-term sequence-level information is a major source of contributions to the improved performance.

2.2. Short-term vs long-term features

To appreciate the contribution of the long-term features, we further compare TalkNet with a prior work (Alcázar et al., 2020), which uses the short-term audio and visual embedding and the relationship between the co-occurring speakers via a two-step training. The first step encodes the low-dimensional representation for video segments of 440 ms, and fuses the audio-visual information, which is similar to TalkNet except that there is neither temporal encoder to increase the receptive fields, and nor attention mechanism.

We first reproduce the system in (Alcázar et al., 2020) to obtain 78.2% mAP for 11 video frames input, which is slightly lower than 79.5% mAP in the original paper due to different batch size setting. Then we extend the segments to 25 video frames and compare the results in Table 7. We observe that the TalkNet obtains a 4.8% improvement from the longer input videos. However, the prior work (Alcázar et al., 2020) does not benefit from the longer segments, with a performance drop of 2.1%. This study suggests that longer video duration doesn’t help without the long audio and visual receptive fields and an adequate attention mechanism.

2.3. Ablation study of TalkNet attention mechanism

To show the contribution of audio-visual relationship and interaction over a long-term video to the ASD task, we conduct the ablation study for the audio-visual attention network in TalkNet. The results are summarized in Table 8. We find that, without cross-attention or self-attention, the performance will drop 0.7% or 1.4% mAP, respectively on the AVA-ActiveSpeaker validation set. When removing the whole audio-visual attention network, the result will decrease to only 90.0% mAP by 2.3%. The results confirm the effectiveness of the cross-attention and self-attention in learning inter-modality cues and long-term audio-visual temporal context.

2.4. Audio augmentation

We report the audio augmentation experiments of TalkNet on the AVA-ActiveSpeaker validation set in Table 9. ‘With neg_sampling’, ‘With noise_aug’ and ‘W/o audio_aug’ stand for our proposed negative sampling method, the traditional audio augmentation method which involves a large noise dataset, and without any audio data augmentation. The proposed TalkNet without any audio data augmentation still outperforms the state-of-the-art. We also observe that there is an obvious difference with and without negative sampling technique. The proposed negative sampling technique outperforms the traditional audio augmentation. These results confirm the efficiency of the negative sampling method, which doesn’t involve external data.

3. Qualitative Analysis

Figure 5 and Figure 6 show some results of TalkNet in the real-world videos with one person and multiple persons on the screen, respectively.

In Figure 7(a), we report the performance for different face sizes on the AVA-ActiveSpeaker validation set. ‘Small’: The face width is smaller than 64 pixels; ‘Middle’: The face width is between 64 to 128 pixels; ‘Large’: The face width is larger than 128 pixels. We observe that the performance decreases as face size gets smaller. It is worth noting that our proposed TalkNet always achieves the best results across all face sizes.

In Figure 7(b), we study the effect of the number of visible faces in a video frame, i.e., 1, 2 or 3 faces, which represent about 90% of all the validation data. From Figure 7(b), we observe that the ASD task becomes more challenging as the number of faces increases. While the performance drops across all methods, the proposed TalkNet is clearly more robust than other competing methods.

Conclusion

In this work, we address and study audio-visual ASD with long-term temporal features. TalkNet utilizes the sentence-level audio-visual videos information as the input to explore the audio-visual relationship and synchronization information. TalkNet outperforms the state-of-the-art results in both two mainstream ASD benchmark by 3.5% and 2.2%.

Acknowledgment

This work was supported by Grant No. A18A2b0046 from the Singapore Government’s Research, Innovation and Enterprise 2020 plan (Advanced Manufacturing and Engineering domain); Grant No. 1922500054 from the Science and Engineering Research Council, Agency of Science, Technology and Research, Singapore, the National Robotics Program.

References