GPU-accelerated Guided Source Separation for Meeting Transcription

Introduction

Automatic speech recognition (ASR) for meetings is characterized by overlapping speech and far-field multi-channel audio . Speaker overlaps, in particular, result in severe degradation in transcription accuracy, both as a result of inaccurate detection of overlapping segments , as well as increased ASR errors on these segments . With the rise of deep neural networks (NNs), there have been several advancements in using NN-based mask estimation methods for speech separation . However, these methods are often limited to fully overlapped synthetic speech, and fail to generalize to real, sparse overlaps that are common in multi-talker meetings . Recently, an alternate formulation of speech separation methods, named continuous speech separation (CSS), targeted specifically for sparse overlaps containing an unknown number of speakers, has been proposed .

Despite growing popularity of supervised methods, beamforming of multi-channel signals using unsupervised mask estimation remains a strong baseline for multi-talker ASR . Among these, the recently proposed guided source separation (GSS) stands out as a particularly effective approach for handling noisy, overlapping speech using diarization information . The method was first proposed for the CHiME-5 challenge, where it provided relative word error rate (WER) improvement of 21.1% on the multi-array track using oracle segmentation . It was later adopted as the challenge baseline for CHiME-6, and used by the winning systems on both oracle and unsegmented tracks .

GSS relies on fundamental ideas from blind source separation (BSS), using spatial mixture models to model the sum of short-time Fourier transform (STFT) bins of multiple speakers . It uses diarization information to (i) estimate the number of mixture components, and (ii) avoid the speaker-frequency permutation problem when processing different frequency bins independently. We will describe the algorithm in detail in Section 2. However, despite its strong performance in the CHiME-5 and CHiME-6 challenges, GSS has seen limited adoption in other multi-talker benchmarks, most notably offline meeting transcription, primarily due to its significant computational cost. For instance, enhancing the CHiME-6 dev set using 80 CPU jobs requires approximately 20 hours with the original GSS implementationhttps://github.com/fgnt/pb_chime5. There have been some efforts to adapt the offline GSS algorithm for real-time enhancement by relying on limited right context , but these are also CPU-bound.

In this paper, we describe our new, publicly-available GPU-accelerated implementation of GSS that aims to remove this computational bottleneck of enhancement. We achieve this primarily by porting all the computations on the GPU, and applying batching at several levels to maximize the GPU memory utilization. Our implementation is inspired by modern deep learning pipelines where background CPU-based workers perform data loading of large tensors, while the data processing is performed by GPUs . We describe our accelerated implementation in detail in Section 3. The resulting 300x speedup allows us to perform ablation experiments using several benchmarks to analyze the importance of factors such as WPE, noise class, context duration, number of BSS iterations, and number of channels, towards GSS performance.

Finally, we provide end-to-end reproducible recipes for meeting transcription of several benchmarks, namely LibriCSS, AMI, and AliMeeting. This includes diarization with and without overlap assignment, GSS-based enhancement, and pretrained models for ASR inference with neural transducers. We believe that our results will provide strong reproducible baselines for all future work on speaker-attributed ASR.

Guided Source Separation

where $K$ is the number of speakers in the recording, and “early” and “late” refer to components of the reverberation. For target-speaker extraction, the objective is to estimate the de-reverberated signal from a desired speaker $k$, i.e., $\widehat{\mathbf{X}}_{t,f,k}$. This estimation is performed in three steps, as described in Fig. 2.

2 Mask estimation using CACGMMs

In the second stage, STFT masks are estimated for each speaker (and noise). The mask estimation technique is based on the “sparsity assumption,” which assumes that only one speaker is active in each time-frequency bin. Using this assumption, the vector in each T-F bin can be assumed to have been generated from a mixture model where each component of the mixture belongs to a different speaker (or noise class). In the case of GSS, each mixture component is a complex angular central Gaussian (CACG), and hence the mixture model is a CACGMM . A CACGMM models sums of unit-normalized complex-valued random variables, and the probability density function at a frequency index $f$ is determined as

where $(\cdot)^{H}$ denotes the Hermitian transpose. Mixture model parameters are usually estimated using the EM algorithm that alternates between estimating the state posteriors (in the E-step) and the parameters of the component model (in the M-step). However, there are two problems in applying EM independently for each frequency bin: (i) the number of sources $K$ is unknown; and (ii) the same mixture component may correspond to different sources in different frequency bin. GSS solves both of these problems by assuming that speaker activities are known for the recording, either through an oracle or a diarization system. Given the speaker activities $a_{t,k}\in\{0,1\}$, we convert the time-invariant mixture weights to time-varying weights as

There may still be a permutation problem between the mixture components for the target speaker and the noise signal, since noise is present throughout the recording. To solve this problem, the GSS algorithm adds a “context window” to each utterance. We run the EM algorithm on the CACGMM until convergence to obtain the final state posteriors $\gamma_{t,f,k}$ as the estimated speaker masks.

3 Mask-based MVDR beamforming

Finally, we compute the spatial covariance matrices for the target signal and background signal as

which are then used to compute the minimum-variance distortionless response (MVDR) filter as

GPU-accelerated Inference

The original GSS implementation is slowed down by the following key factors: (i) All the segments are processed sequentially, so processing time for a recording increases linearly with number of identified segments. (ii) A context window (usually 15s) is used for all segments regardless of the segment duration, resulting in a lot of wasted computation for short segments. (iii) For each segment, the CACGMM-based mask estimation is performed by iterating over all frequency bins (usually 513) sequentially. (iv) All computations (i.e., feature extraction, WPE, mask estimation, beamforming, and iSTFT) are performed on the CPU using NumPy . A workaround for limitation (i) was provided by using MPI-based multi-processing (or Kaldi-style parallelization https://kaldi-asr.org/doc/queue.html) to enhance segments concurrently on a multi-node CPU cluster. Nevertheless, enhancing the CHiME-6 dev set, for instance, may require close to 20 hours (wall clock time) even using 80 CPU jobs (§ 6.4).

We propose to accelerate GSS-based inference by leveraging the power of modern GPU hardware and pipelines inspired from neural network training. First, to address limitation (iv), we use CuPy arrays which speed up array operations significantly using CUDA kernels, compared with regular NumPy-based array operations . Since the most computationally intensive operations in the pipeline (such as CACG probability estimation) involve matrix multiplications (through einsum), GPU-based CUDA kernels are more efficient. However, simply transferring all arrays to CuPy is not sufficient — for example, limitations (i)–(iii) still require sequential processing, which limits GPU utilization. To maximize GPU utilization and improve real-time factor (RTF), we perform the following additional optimizations.

Segment batching. Instead of processing each segment independently, we batch together multiple segments for inference. However, unlike neural network based training pipelines where batching is performed by stacking sequences in parallel, our batches are formed by concatenating segments sequentially along the time ($T$) axis to create “super-segments.” We choose this form of batching because (i) the einsum-based operations are designed to work with 3-D tensors, and (ii) parallel batching of segments with padding would result in wasted memory. Since multiple components of the inference (such as mask estimation and beamforming) compute statistics over the entire segment, we always create super-segments of the same recording with the same target speaker. Furthermore, we only use a single context window for the entire batch (instead of segment-wise context), which further reduces the wasted computations for short segments. This batching technique should work well for the case when optimal reference channels do not vary over the duration of the recording (i.e., when speakers are stationary, which is common for meeting scenarios)We also provide the option for using at most one segment per batch, for the case when speakers are not stationary (§ 6.4)..

CPU-based data-loaders. We ensure that GPU idle time is minimized by off-loading the batch creation process to CPU-based data-loaders (possibly containing multiple workers), similar to deep learning pipelines. Section 4 provides further details about our Lhotse-based data pipeline.

Frequency batching. To address (iii), we modified the CACGMM-based mask estimation to process 3-D tensors $(F,T,M)$ instead of 2-D arrays $(T,M)$. This simple change allows us to process all the frequency bins concurrently in a batch, significantly increasing GPU memory utilization.

Einsum path optimization. As mentioned above, several components in the GSS pipeline are implemented using einsum, and uses an optimal path contraction technique to find the path of minimum floating-point operations through the sequence (often resulting in up to 15x speed-up over a naive computation) . However, the optimal path finding itself is computationally demanding, with a complexity of $\mathcal{O}(N!)$ for $N$ arrays, and since it is performed several times during inference (for example, in each iteration of the CACGMM inference), it overshadows any speed-ups from the actual contracted sum. To remedy this, we cache the optimal computed path in the first iteration and re-use it in subsequent iterations.In practice, since our tensor dimensions often have the same relative order across all batches (i.e., $M$¡$F$¡$T$), we can simply fix the optimal path for all einsum operations. This is because segment batching avoids very short segments that would otherwise result in $T$¡$F$.

Finally, once the enhanced waveform is obtained for the super-segment, we use background worker threads to chunk it into the original segments and save the audios to disk. With all these speed-ups, we were able to enhance a 10-minute LibriCSS recording in 159s (as shown in Fig. 3), of which mask estimation, WPE, and beamformer constituted 59.2%, 24.4%, and 8.5% processing time, respectively. This is equivalent to a real-time factor (RTF) of approximately 0.3. We anticipate that further speed-ups could be obtained using GPUs with larger memory, by using bigger batches.

Implementation Details

From an implementation perspective, we can divide the pipeline into two parts. The data processing part is tasked with efficiently creating segments and corresponding speaker activities, while the inference part performs the actual computations on GPU. We use Lhotse for all data processing, i.e., to store and read recording metadata, to represent speaker activities, and to perform segment batching to create super-segments . For batching we create buckets out of each speaker’s segments on-the-fly, and sample speakers in a round-robin manner (see Lhotse’s DynamicBucketingSampler and RoundRobinSampler), so that metadata from all segments do not need to be stored in memory. The super-segment obtained from the data-loader is converted to a CuPy array in-place, and all subsequent inference is performed on the GPU. Since we use Lhotse’s supervision manifests to store speaker activities, it allows us to use either oracle segments, or read segments from RTTM files (diarization output) with the same data processing pipeline (cf. § 6.1 and 6.2). A typical recipe for enhancing a corpus is below:

Experimental Setup

We performed evaluations on three publicly-available meeting datasets: LibriCSS, AMI, and AliMeeting. LibriCSS consists of multi-channel audio recordings of 8-speaker “simulated conversations” that were created by combining utterances from the LibriSpeech test-clean set . It comprises 10 one-hour long sessions, each of which is made up of six 10-minute “mini sessions” that have different overlap ratios (ranging from 0% to 40%). AMI (Augmented Multi-party Interactions) consists of 100 hours of recorded meetings containing 4 or 5 speakers per session . AliMeeting is a Mandarin-language corpus collected from real meetings, originally designed for ICASSP 2022 M2MeT challenge . Each session consists of a 15 to 30-minute discussion by 2-4 participants. Detailed statistics for all datasets are shown in Table 1.

We used three different mic settings for our experiments: IHM (individual headset microphone), SDM (single distant microphone), and GSS (GSS-enhanced multi-mic). Since LibriCSS does not provide headset recordings, we used the corresponding LibriSpeech utterances concatenated together to simulate IHM. For all datasets, the first channel of the first array was used for the SDM setting. For LibriCSS and AliMeeting, we used all available channels for GSS, whereas for AMI, we used the first of the two arrays.

2 Models

We trained separate transducer-based ASR models for each benchmark. For LibriCSS, we used a pretrained Conformer-transducer trained on LibriSpeech. For AMI and AliMeeting, we trained a Zipformer transducer on a combination of IHM, IHM with simulated reverb, SDM, and GSS-enhanced far-field recordings of the corresponding train set, and the resulting model was used to evaluate all microphone settings. In all cases, we applied three-fold speed perturbation and noise augmentation using MUSAN noises. We used a “stateless” decoder consisting of a convolutional layer with a bi-gram context. The model was trained using a pruned RNN-T loss implemented in k2https://github.com/k2-fsa/{k2,icefall}. For decoding, we used a WFST-based parallel beam search method with beam size 4 . Full training recipes and pretrained models are available on Icefall.

For the non-oracle segmentation experiments in § 6.2, we used a multi-class spectral clustering based diarization system with and without overlap assignment . The system consists of a Pyannote-based speech activity detector fine-tuned on the corresponding train set for AMI and AliMeeting. For embedding extraction, we used a pretrained ResNet101-based network , which was trained on VoxCeleb and CN-Celeb . For these experiments, we report diarization error rates (DER) and concatenated minimum-permutation WER (cpWER) in order to analyze the impact of diarization errors on downstream ASR. We did not use any collars to compute DERs for LibriCSS and AMI, but a collar of 0.25 was used for AliMeeting following the original work. All diarization recipes, generated RTTM files, and inference pipelines for meeting transcription are publicly availablehttps://github.com/desh2608/diarizerhttps://github.com/desh2608/icefall/tree/multi_talker.

Results & Discussion

2 Effect of diarization

For meeting transcription, it may be hard to obtain oracle segmentation, and often a diarization system is used as a pre-processing step for ASR. In Table 3, we investigate the impact of using non-oracle segmentation with GSS-based enhancement. We found that when no enhancement is performed, overlap detection results in little to no cpWER improvement, since the ASR system is unable to handle overlapping segments. This finding corroborates the results of the winning CHiME-6 system , which was able to substantially improve ASR performance on unsegmented recordings using TS-VAD based diarization . Using GSS results in significant improvements, with relative cpWER (or cpCER) reductions of 29.1%, 19.5%, and 19.7% on LibriCSS, AMI, and AliMeeting, respectively.

3 Which factors are most important for GSS?

We performed ablation studies to investigate the effect of several GSS parameters — WPE, noise class, context duration, number of iterations for CACGMM inference, and number of input channels — on the downstream ASR performance, as shown in Fig. 4. WPE was found to be more important for LibriCSS, while using an additional noise class was more important for AMI (Fig. 4). This may be because AMI contains occassional background noise, which is absent in LibriCSS. Increasing the context duration from 5s to 15s resulted in consistent WER gains, but adding further context degraded WER (Fig. 4). This may be because of inclusion of the target speaker segments in the context if it is expanded too far. A similar observation was made earlier for CHiME-5 , where a 15s context resulted in better WER compared to a 2s context .

For both datasets, increasing the number of BSS iterations (for CACGMM inference) beyond 5 did not result in any WER improvements (Fig. 4). Finally, using more input channels was found to be the single most important factor for better WER performance. For example, using seven input channels resulted in relative WER reduction of 50.4% and 21.8% on LibriCSS and AMI, respectively, compared to using two channels. Nevertheless, it follows the law of diminishing returns, as evident by the exponential decay in Fig. 4.

4 Analysis of speed-up

We compared our GSS implementation with the original GSS on the CHiME-6 development set in terms of wall clock time and ASR performance, as shown in Table 4. For ASR inference, we used the publicly available Kaldi recipe and pretrained models from JHU-CLSP’s submission to the CHiME-6 challengehttps://github.com/kaldi-asr/kaldi/blob/master/egs/chime6/s5b_track1https://kaldi-asr.org/models/m12 . We found that our implementation obtained an effective speed-up of 292.2 without any degradation in WER. Since CHiME-6 has non-stationary speakers, we disabled segment batching for this experiment. We can obtain even further speed-ups by enabling this for meeting-like data where speakers are stationary.

Conclusion

We described our GPU-accelerated implementation of GSS-based front-end enhancement for meeting transcription. On the CHiME-6 benchmark, it was found to be 300x faster than the original implementation, thus removing the computational bottleneck associated with this technique. Through experiments conducted on LibriCSS, AMI, and AliMeeting, we showed that GSS-based enhancement can recover up to 80% of the WER difference, in going from close-talk to far-field conditions. We also performed several ablation studies to study the effect of GSS parameters, and showed that using more input channels is the single most important factor for better ASR performance. Our pip-installable package is publicly available at https://github.com/desh2608/gss.

Acknowledgments

This project was partially funded by NSF CCRI Grant No. 2120435, and a fellowship from Amazon via the JHU-Amazon Initiative for Interactive Artificial Intelligence (AI2AI).

References