A Review of Speaker Diarization: Recent Advances with Deep Learning

Introduction

“Diarize” means making a note or keeping an event in a diary. Speaker diarization, like keeping a record of events in such a diary, addresses the question of “who spoke when” by logging speaker-specific salient events on multiparticipant (or multispeaker) audio data. Throughout the diarization process, the audio data would be divided and clustered into groups of speech segments with the same speaker identity/label. As a result, salient events, such as non-speech/speech transition or speaker turn changes, are automatically detected. In general, this process does not require any prior knowledge of the speakers, such as their real identity or the number of participating speakers in the audio data. Thanks to its feature of separating audio streams by these speaker-specific events, speaker diarization can be effectively employed for indexing or analyzing various types of audio data, e.g., audio/video broadcasts from media stations, conversations in conferences, personal videos from online social media or hand-held devices, court proceedings, business meetings, earnings reports in a financial sector, just to name a few.

Traditionally speaker diarization systems consist of multiple, independent sub-modules as presented in Fig. 1. To mitigate any artifacts in acoustic environments, various front-end processing techniques, for example, speech enhancement, dereverberation, speech separation or target speaker extraction, are employed. Voice or speech activity detection (SAD) is then applied to separate speech from non-speech events. Raw speech signals in the selected speech portion are transformed to acoustic features or embedding vectors. In the clustering stage, the transformed speech portions are grouped and labeled by speaker classes and in the post-processing stage, the clustering results are further refined. Each of these sub-modules is optimized individually in general.

During the early years of diarization technology (in the 1990s), the research objective was to benefit automatic speech recognition (ASR) on air traffic control dialogues and broadcast news recordings, by separating each speaker’s speech segments and enabling speaker-adaptive training of acoustic models . In this period some fundamental approaches for measuring the distance between speech segments for speaker change detection and clustering, such as generalized likelihood ratio (GLR) and Bayesian information criterion (BIC) , were developed and quickly became the golden standard. All these efforts collectively laid out paths to consolidate activities across research groups worldwide, leading to several research consortia and challenges in the early 2000s, among which there were the Augmented Multiparty Interaction (AMI) Consortium supported by the European Commission and the RT Evaluation hosted by the National Institute of Standards and Technology (NIST). These organizations, spanning over from a few years to a decade, fostered further advancements on speaker diarization technologies across different data domains from broadcast news and conversational telephone speech (CTS) to meeting conversations . The new approaches resulting from these advancements include, but not limited to, beamforming , information bottleneck clustering (IBC) , variational Bayesian (VB) approaches , joint factor analysis (JFA) .

Speaker specific representation in a total variability space derived from simplified JFA, known as i-vector , found great success in speaker recognition and was quickly adopted by speaker diarization systems as feature representation for short speech segments, segmented in an unsupervised fashion. i-Vector successfully replaced its predecessors such as merely mel-frequency cepstral coefficient (MFCC) or speaker factors (or eigenvoices) to bolster clustering performance in speaker diarization, being combined with principal component analysis (PCA) , variational Bayesian Gaussian mixture model (VB-GMM) , mean shift and probabilistic linear discriminant analysis (PLDA) .

Since the advent of deep learning in the 2010s, there has been a considerable amount of research to take advantage of powerful modeling capabilities of the neural networks for speaker diarization. One representative example is the extraction of the speaker embeddings using neural networks, such as the d-vectors or the x-vectors , which most often are embedding vector representations based on the bottleneck layer output of a deep neural network (DNN) trained for speaker recognition. The shift from i-vector to these neural embeddings contributed to enhanced performance, easier training with more data , and robustness against speaker variability and acoustic conditions. More recently, end-to-end neural diarization (EEND) where individual sub-modules in the traditional speaker diarization systems (c.f., Fig. 1) can be replaced by one neural network gets more attention with promising results . This research direction, although not fully matured yet, could open up unprecedented opportunities to address challenges in the field of speaker diarization, such as, the joint optimization with other speech applications, with overlapping speech, if large-scale data is available for training such powerful neural network-based models.

2 Motivation

Till now, there are two well-rounded overview papers in the area of speaker diarization that survey the development of speaker diarization technology with different focuses. In , various speaker diarization systems and their subtasks in the context of broadcast news and CTS data are reviewed up to till mid 2000s. Thus, the historical progress of speaker diarization technology development in the 1990s and early 2000s are covered. Contrarily, the focus of was put more on speaker diarization for meeting speech and its respective challenges. This paper thus weighs more in the corresponding technologies to mitigate problems from the perspective of meeting environments, where there are usually more participants than broadcast news or CTS data and multi-modal data is frequently available. Since these two papers were published, speaker diarization systems have gone through a lot of notable changes, especially from the leap-frog advancements in deep learning approaches addressing technical challenges across multiple machine learning domains. We believe that this survey work is a valuable contribution to the community to consolidate the recent developments with neural methods and thus facilitate further progress toward a more efficient diarization.

3 Overview and Taxonomy of Speaker Diarization

Attempting to categorize the existing, most-diverse speaker diarization technologies, both in the space of modularized speaker diarization systems before the deep learning era and those based on neural networks of recent years, a proper grouping would be helpful. The main categorization we adopt in this paper is based on two criteria, resulting total of four categories, as shown in Table 1.3. The first criterion is whether the model is trained based on speaker diarization-oriented objective function or not. Any trainable approaches to optimize models in a multispeaker situation and learn relations between speakers are categorized into the “diarization objective” class. The second criterion is whether multiple modules are jointly optimized toward some objective function. If a single sub-module is replaced into a trainable one, such method is categorized into the “Single-module optimization” class. Conversely, joint modeling of segmentation and clustering , joint modeling of speech separation and speaker diarization or fully end-to-end neural diarization system is categorized into the “Joint optimization” class.

Note that our intention of this categorization is to help readers to quickly overview the broad development in the field, and it is not our intention to divide the categories into superior-inferior. Also, while we are aware of many techniques that fall into the category “Non-Diarization Objective” and “Joint Optimization” (e.g., joint front-end and ASR and joint speaker identification and speech separation ), we exclude them in the paper to focus on the review of speaker diarization techniques.

The accuracy of speaker diarization system is measured using diarization error rate (DER) where DER is the sum of three different error types: False alarm (FA) of speech, missed detection of speech and confusion between speaker labels.

To establish a one-to-one mapping between the hypothesis outputs and the reference transcript, Hungarian algorithm is employed. In the 2006 RT evaluation , 0.25 s of “no score” collar (also referred to as “score collar”) is set around every boundary of reference segment to mitigate the effect of inconsistent notation and human errors in reference transcript and this evaluation scheme has been most widely used in speaker diarization studies.

4.2 Jaccard Error Rate

The Jaccard error rate (JER) was first introduced in the DIHARD II evaluation. The goal of JER is to evaluate each speaker with equal weight. Unlike DER which is estimated for the whole utterance altogether, per-speaker error rates are first computed and then averaged to compute JER. Specifically, JER is computed as follows.

4.3 Word-level Diarization Error Rate

While DER is based on the duration of the speaking time of each speaker, word-level DER (WDER) is designed to measure the error that is caused in the lexical (output transcription) side. The motivation of WDER is the discrepancy between DER and the accuracy of the final transcript output since DER relies on the duration of the speaking time that is not always aligned with the word boundaries. The concept of word-breakage ratio was proposed in Silovsky et al. where word-breakage shares similar idea with WDER. Unlike WDER, word-breakage ratio measures the number of speaker-change points occur inside a word boundary. The work in Park and Georgiou suggested the term WDER, evaluating the diarization output with ground-truth transcription. More recently, the joint ASR and speaker diarization system was evaluated in the WDER format in Shafey et al. . Although the way of calculating WDER would differ over the studies, but the underlying mechanism is that the diarization error is calculated by counting the correctly or incorrectly labeled words.

5 Paper Organization

The rest of the paper is organized as follows.

In Section 2, we overview techniques belonging to the “Non-diarization objective” class in the proposed taxonomy, mostly those used in the traditional, modular speaker diarization systems. While there are some overlaps with the counterpart sections of the aforementioned two survey papers in terms of reviewing notable developments in the past, this section would add more latest schemes in the corresponding components of the speaker diarization systems.

In Section 3, we discuss advancements mostly leveraging DNNs trained with the diarization objective where single sub-modules are independently optimized (Subsection 3.1) or jointly optimized (Subsection 3.2) toward fully end-to-end speaker diarization.

In Section 4, we present a perspective of how speaker diarization has been investigated in the context of ASR, reviewing historical interactions between these two domains to peek into the past, present and future of speaker diarization applications.

Section 5 provides information on speaker diarization challenges and corpora to facilitate research activities and anchor technology advances. We also discuss evaluation metrics such as DER, JER and Word-level DER (WDER) in this section.

We share a few examples of how speaker diarization systems are employed in both research and industry practices in Section 6 and conclude this work in Section 7, providing summary and future challenges in speaker diarization.

Modular Speaker Diarization Systems

This section provides an overview of algorithms for speaker diarization belonging to the “Non-diarization objective” class, as shown in Table 1.3. Each subsection in this section corresponds to the explanation of each module in the traditional speaker diarization system, as shown in Figure 1. In addition to the introductory explanation of each module, this section also summarizes the recent techniques within the module.

where $K$ denotes the number of speakers present in the audio signal.

The aim of the front-end techniques described in this section is to estimate the original source signal $\hat{\mathbf{x}}_{i,t}$ given the observation $\mathbf{X}=(\{x_{t,f}\}_{f})_{t}$ for the downstream diarization task,

Although there are numerous speech enhancement, dereverberation, and separation algorithms, e.g., , herein most of the recent techniques used in the DIHARD challenge series , LibriCSS meeting recognition task , and the CHiME-6 challenge track 2 are covered.

Speech enhancement techniques focus mainly on the suppression of the noise component of the noisy speech, which has shown a significant improvement thanks to deep learning. For example, long short-term memory (LSTM)-based speech enhancement is used as a front-end technique in the DIHARD II baseline , i.e.,

where we only consider the single source example (i.e., $K=1$) and omit the source index $i$. This is a regression-based approach by minimizing the objective function,

The log power spectrum or ideal ratio mask is often used as the target domain of the output $\mathbf{s}_{t}$. Also, the speech enhancement used in applies this objective function in each layer based on a progressive manner.

The effectiveness of the speech enhancement techniques can be enhanced via multichannel processing, including minimum variance distortionless response (MVDR) beamforming . demonstrates the significant improvement of the DER from 18.3% to 13.9% in the LibriCSS meeting task based on mask-based MVDR beamforming .

1.2 Dereverberation

Compared with other front-end techniques, the major dereverberation techniques used in various tasks are based on statistical signal processing methods. One of the most widely used techniques is the weighted prediction error (WPE) based dereverberation .

The basic idea of WPE, for the case of single source, i.e., $K=1$, without noise, is to decompose the original signal model Eq. (3) into the early reflection $x_{t,f}^{\text{early}}$ and late reverberation $x_{t,f}^{\text{late}}$ as follows:

where $\Delta$ denotes the number of frames to split the early reflection and late reverberation, and $L$ denotes the filter size.

WPE is widely used as one of the gold standard front-end processing methods, e.g., it is part of the DIHARD and CHiME, which are both the baseline and the top-performing systems . Although the performance improvement of WPE-based dereverberation is not significant, it provides solid performance improvement across almost all tasks. Moreover, WPE is based on linear filtering and since it does not introduce signal distortions, it can be safely combined with downstream front-end and back-end processing steps. Similar to the speech enhancement techniques, WPE-based dereverberation demonstrates additional performance improvements when applied on multichannel signals.

1.3 Speech Separation

Speech separation is a promising family of techniques when the overlapping speech regions are significant. The effectiveness of multichannel speech separation based on beamforming has been widely confirmed . For example, in the CHiME-6 challenge , guided source separation (GSS) based multichannel speech extraction techniques have been used to achieve the top result. On the other hand, single-channel speech separation techniques do not often show any significant effectiveness in realistic multispeaker scenarios, such as the LibriCSS or the CHiME-6 tasks , where speech signals are continuous and contain both overlapping and overlap-free speech regions. The single-channel speech separation systems often produce a redundant non-speech or even a duplicated speech signal for the non-overlap regions, and as such the “leakage” of audio causes many false alarms of speech activity. A leakage filtering method was proposed in tackle the problem, where a significant improvement in the diarization performance was observed after including this processing step in the top-ranked system on the VoxCeleb Speaker Recognition Challenge 2020 .

2 Speech Activity Detection

SAD, also known as voice activity detection (VAD), distinguishes speech from non-speech such as background noise. SAD plays a significant role not only in speaker diarization but also in speaker recognition and speech recognition systems since SAD is a pre-processing step that can create errors that propagate through the whole pipeline. A SAD system consists mostly of two major parts. The first one is a feature extraction front-end, where acoustic features such as zero crossing rate , pitch , signal energy , higher order statistics in the linear predictive coding residual domain or MFCCs are often used. The other part is a classifier, where a model predicts and decides whether the input-frame contains speech or not. A system based on statistical models on spectrum , Gaussian mixture models (GMMs) and on Hidden Markov Models (HMMs) has been traditionally used. After the deep learning approaches gained popularity in the speech signal processing field, numerous DNN-based systems, such as the ones based on MLP , convolutional neural network (CNN) , LSTM , have been also proposed with superior performance to the traditional methods.

The performance of SAD largely affects the overall performance of the speaker diarization system as it can create a significant amount of false positive salient events or miss speech segments . A common practice in speaker diarization tasks is to report DER with “oracle SAD” setup which indicates that the system output is using SAD output that is identical to the ground truth. Conversely, the system output with an actual speech activity detector is referred to as “system SAD” output.

3 Segmentation

In the context of speaker diarization, speech segmentation is a process of breaking the input audio stream into multiple segments to obtain speaker-uniform segments. Therefore, the output unit of the speaker diarization system is determined via a segmentation process. In general, speech segmentation methods for speaker diarization are divided into two major categories: Segmentation by speaker-change point detection and uniform segmentation.

Segmentation by the speaker-change point detection was the gold standard of the earlier speaker diarization systems, where speaker-change points are detected by comparing two hypotheses: $H_{0}$ assumes that both the left and right speech windows are from the same speaker, whereas $H_{1}$ assumes that the two speech windows are from the different speakers. To test these two hypotheses, metric-based approaches were most widely applied. In metric-based approaches, the distribution of the speech feature is assumed to follow a Gaussian distribution $\mathcal{N}(\mu,\Sigma)$ with mean $\mu$ and covariance $\Sigma$. The two hypotheses $H_{0}$ and $H_{1}$ can be then represented as follows:

where $(\mathbf{x}_{i}|i=1,\cdots,N)$ is a sequence of speech features in the interest of the hypothesis testing. A slew of criteria for the metric-based approach were proposed to quantify the likelihood of the two hypotheses. The examples include the Kullback Leibler (KL) distance , Generalized Likelihood Ratio (GLR) and BIC . Among these criteria, BIC has been the most widely used method followed by numerous variants . Thus, in this section, we introduce BIC as a representative example for a metric-based method. If we apply BIC and to the hypotheses described in Eq. (9), a BIC value between two models from two hypotheses is expressed as follows:

where the sample covariance $\Sigma$ is from $\left\{\mathbf{x}_{1},\cdots,\mathbf{x}_{N}\right\}$, $\Sigma_{1}$ is from $\left\{\mathbf{x}_{1},\cdots,\mathbf{x}_{i}\right\}$ and $\Sigma_{2}$ is from $\left\{\mathbf{x}_{i+1},\cdots,\mathbf{x}_{N}\right\}$ and $P$ is the penalty term defined as

where $d$ denotes the dimension of the feature; $N_{1}$ and $N_{2}$ are frame lengths of each window, respectively and $N=N_{1}+N_{2}$. The penalty weight $\lambda$ is generally set to $\lambda=1$. The change point is set when the following equation becomes true:

In general, if speech segmentation is done using the speaker-change point detection method, the length of each segment is not consistent. Therefore, after the advent of the i-vector and DNN-based embeddings , the segmentation based on speaker-change point detection was mostly replaced with uniform segmentation , since varying lengths of the segment created an additional variability into the speaker representation and deteriorated the fidelity of the speaker representations.

In uniform segmentation schemes, the given audio stream input is segmented with a fixed window length and overlap length. Thus, the unit duration of the speaker diarization output stays constant. However, the process of uniform segmentation of the input signals for diarization poses some potential problems because it introduces a trade-off error related to the segment length. The segments created from the uniform segmentation need to be sufficiently short to safely assume that they do not contain multiple speakers. However, at the same time it is important to capture sufficient acoustic information to extract reliable speaker representations.

4 Speaker Representations and Similarity Measure

Speaker representation plays a crucial role for speaker diarization systems to measure the similarity between speech segments. This section will cover such speaker representation and also the similarity measure because they are tightly connected. We first introduce metric-based approaches for similarity measures which were popular from the late 1990s to early 2000s in Section 2.4.1. We then introduce widely used speaker representations for speaker diarization systems that are usually employed together with the uniform segmentation method described in Section 2.4.2 and Section 2.4.3.

From the late 1990s to early 2000s, metric-based approaches were most commonly used for the similarity measurement between speech segments for speaker diarization systems. Methods used for speaker segmentation were also applied to measure the similarity between segments, such as KL distance , GLR , and BIC . As with the case of the segmentation, the BIC-based method, where the similarity between two segments are computed by Eq. (10), was one of the most extensively used metrics due to its effectiveness and ease of implementation. Metric-based approaches are usually employed together with the segmentation approaches based on a speaker-change point detection. The agglomerative hierarchical clustering (AHC) is often applied to obtain the diarization result, which will be detailed in Section 2.5.1.

4.2 Joint Factor Analysis, i-vector and PLDA

Before the advent of speaker representations such as i-vector or x-vector , Gaussian Mixture Model-based Universal Background Model (GMM-UBM) applied to acoustic features demonstrated success in speaker verification tasks. A UBM consists of a large GMM (typically with 512 to 2048 mixtures) trained to represent the speaker-independent distribution of acoustic features. Thus, a GMM-UBM model can be described with the following quantities: mixture weights, mean values and covariance matrix of the mixtures. The log-likelihood ratio between a speaker-adapted GMM and the speaker-independent GMM-UBM is used for speaker verification. Despite the success on modeling the speaker identity, GMM-UBM based speaker verification systems have suffered from intersession variability , which is the variability exhibited by a given speaker from one recording session to another. Such difficulty occurs because the relevance maximum a posteriori (MAP) adaptation step during the speaker enrollment process in the GMM-UBM based speaker verification systems not only captures the speaker-specific characteristics of the speech, but also unwanted channel noise and other nuisances from the acoustic environment.

Joint factor analysis (JFA) was proposed to compensate for the variability issues by separately modeling the inter-speaker variability and the channel or session variability. The JFA approach employs a GMM supervector, which is a concatenated mean of the adapted GMM. For example, suppose a $F$ by 1 speaker-independent GMM mean vector $m_{c}$, where $c$ is the mixture component index and $F$ is the dimension of the feature. Then, a supervector $\mathbf{M}$ has dimension of $CF$ by 1 by concatenating the F-dimensional mean vector for C mixture components. Thus, the supervector $\mathbf{M}$ can be described as follows:

In the JFA approach, the given GMM supervector is decomposed into speaker independent, speaker dependent, channel dependent, and residual components. Thus, the ideal speaker supervector $\mathbf{M}_{J}$ can be decomposed as indicated in Eq. (14), where $\mathbf{m}_{J}$ denotes a speaker independent supervector from the UBM, $\mathbf{V}$ denotes a speaker dependent component matrix, $\mathbf{U}$ denotes a channel dependent component matrix, and $\mathbf{D}$ denotes a speaker-dependent residual component matrix. Along with these component matrices, vector $\mathbf{y}$ is for the speaker factors, vector $\mathbf{x}$ is for the channel factors, and vector $\mathbf{z}$ is for the speaker-specific residual factors. All of these vectors have a prior distribution of $N(0,1)$.

The JFA approach was followed by the study in , in which it was discovered that channel factors in the JFA also contain information regarding the speakers. Thus, Dehak et al. proposed a new method combining the channel and speaker spaces into a combined variability space through a total variability matrix. Thus, the total variability matrix $\mathbf{T}$ models both the channel and the speaker variability, and the latent variable $\mathbf{w}$ weights the column of the matrix $\mathbf{T}$. The variable $\mathbf{w}$ is referred to as the i-vector and is also considered a speaker representation vector. Each speaker and channel in a GMM supervector $\mathbf{M}_{I}$ can be modeled as follows:

where $\mathbf{m}_{I}$ is a speaker-independent and channel-independent supervector that can be taken as a UBM supervector. The process of extracting an i-vector $\mathbf{w}$ for the given recording is formulated as a MAP estimation problem using the Baum–Welch statistics extracted using the UBM, mean supervector $\mathbf{m}_{I}$, and total variability matrix $\mathbf{T}$ trained from the EM algorithm as parameters. The idea behind a speaker representation was greatly popularized through the use of i-vectors, where the speaker representation vector can contain a numerical feature characterizing the vocal tract of each speaker. The i-vector speaker representations have been employed in not only speaker recognition studies but also in numerous speaker diarization studies and have shown a superior performance compared to metric-based methods such as BIC, GLR, and KL, as mentioned in the previous subsection.

Intersession variability in the i-vector approach has been further compensated using backend procedures, such as a linear discriminant analysis (LDA) and within-class covariance normalization (WCCN) , followed by simple cosine similarity scoring. The cosine similarity scoring was later replaced with a probabilistic LDA (PLDA) model in . In the following studies , a method applying a Gaussianization of the i-vectors and thus generating Gaussian assumptions in the PLDA, referred to as G-PLDA or simplified PLDA, was proposed for speaker verification. In general, PLDA employs the following modeling for the given speaker representation $\phi_{ij}$ of the $i$-th speaker in the $j$-th session as indicated below:

Here, $\boldsymbol{\mu}$ is the mean vector, $\mathbf{F}$ is the speaker variability matrix, $\mathbf{G}$ is the channel variability matrix, and $\epsilon_{ij}$ is a residual component. In addition, $\mathbf{h}_{i}$ and $\mathbf{w}_{ij}$ are latent variables specific for the speaker and session, respectively. In G-PLDA, both latent variables, $\mathbf{h}_{i}$ and $\mathbf{w}_{ij}$, are assumed to follow a standard Gaussian prior. During the training process of the PLDA, $\boldsymbol{\mu}$, $\mathbf{\Sigma}$, $\mathbf{F}$, and $\mathbf{G}$ are estimated using the expectation maximization (EM) algorithm. Based on the estimated statistics, two hypotheses are tested: hypothesis $H_{0}$ for a case in which two samples are from the same speaker, and hypothesis $H_{1}$ for when two samples are from different speakers. Under the hypothesis $H_{0}$, the given speaker representations $\phi_{1}$ and $\phi_{2}$ are modeled as follows with a common latent variable $\mathbf{h}_{12}$:

On the other hand, under the hypothesis $H_{1}$, $\phi_{1}$ and $\phi_{2}$ are modeled as follows with separate latent variable $\mathbf{h}_{1}$ and $\mathbf{h}_{2}$:

In G-PLDA, it is assumed that $\phi$ is generated from a Gaussian distribution, which results in the following conditional density function .

Using Eq. (16)-(41), the log likelihood ratio can be described as follows:

The log likelihood ratio $s\left(\phi_{1},\phi_{2}\right)$ in the above equation was originally used for speaker verification to choose a hypothesis between $H_{0}$ and $H_{1}$ by checking whether $s\left(\phi_{1},\phi_{2}\right)$ is positive or negative. The PLDA for speaker representations also employed in speaker diarization and the log likelihood $s\left(\phi_{1},\phi_{2}\right)$ is used to check the similarity between clusters. Further details regarding the clustering approach using PLDA is described in Section 2.5.1.

4.3 Neural Network Based Speaker Representations

Speaker representations for speaker diarization have also been heavily affected by the rise of deep learning approaches. The idea behind DNN-based representation learning was first introduced for face recognition tasks . As the fundamental idea of a neural network-based representation, we can use the deep neural network architecture to map the input signal source (an image or an audio clip) to a dense vector containing floating-point numbers. This is achieved by taking the values from a layer in the neural network model after forward-propagating the input signal to the layer that we take the values from. The mapping process from the input signal to the speaker embedding is based on the nonlinear modeling capability of multiple layers in the DNNs. In so doing, the training process of the DNNs allows the neural networks to learn the mapping without specifying any components or factors, which is in contrast to traditional factor analysis models based on decomposable components. In this sense, the components in JFA are more explainable than the parameters trained in DNN models trained for speaker embedding extraction. In addition, DNN-based speaker representation learning does not involve predefined probabilistic models (e.g., GMM-UBM) for the input acoustic features. In relation to this, DNN-based speaker representation achieves an improved efficiency during the inference phase because the solution used by factor-analysis based methods involves a computationally intensive matrix inversion operation , whereas DNN-based embedding extractors involve fewer demanding operations, such as multiple linear transformations, with non-linear function computations for obtaining the speaker representation vector. Thus, the representation learning process has become more straight-forward and the inference speed has been improved compared to the traditional factor-analysis based methods. Among many of the neural-network based speaker representations, d-vector remains one of the most prominent speaker representation extraction frameworks. The stacked filter bank features, which include context frames as an input feature, are employed, and multiple fully connected layers are trained with cross entropy loss. Speaker representation vectors, also referred to as d-vectors, are obtained from the last fully connected layer, as indicated in Fig. 3. The d-vector scheme appears in numerous speaker diarization papers, e.g., in .

DNN-based speaker representations are even more improved when using an x-vector , which demonstrates a superior performance, winning the NIST speaker recognition challenge 2018 and the first DIHARD challenge . Fig. 3 shows the structure of an x-vector extractor. The time-delay architecture and statistics pooling layer differentiate the x-vector architecture from that of a d-vector. The statistics pooling layer aggregates the frame-level outputs from the previous layer and computes its mean and standard deviation, passing them on to the following layer. Thus, it can allow the extraction of x-vectors from a variable length input. This is advantageous not only for speaker verification but also for speaker diarization because speaker diarization systems are bound to process segments that are shorter than the predetermined uniform segment length when the segment should be truncated at the end of an utterance.

5 Clustering

A clustering algorithm is applied to make clusters of the speech segments based on the speaker representation and similarity measure explained in the previous section. Here, we introduce the most commonly used clustering methods for speaker diarization.

AHC is a clustering method that has been constantly employed in many speaker diarization systems with different distance metrics such as BIC , KL and PLDA . AHC is an iterative process of merging the existing clusters until the clustering process meets a criterion. The AHC process starts with the calculation of the similarity between N singleton clusters. At each step, a pair of clusters that has the highest similarity is merged. The iterative merging process of AHC is illustrated in a dendrogram, which is presented in Fig. 4.

One of the most important aspects of AHC is the stopping criterion. For the speaker diarization task, the AHC process can be stopped using either a similarity threshold or a target number of clusters. Ideally, if PLDA is used as a distance metric, the AHC process should be stopped at $s\left(\phi_{1},\phi_{2}\right)=0$ in Eq.(43). However, the stopping metric is adjusted to obtain an accurate number of clusters based on a development set. Conversely, if the number of speakers is known or estimated using other methods, the AHC process can be stopped when the clusters created by the AHC process reaches the pre-determined number of speakers $k$.

5.2 Spectral Clustering

Spectral clustering is a widely used clustering approach for speaker diarization. While there are many variations, spectral clustering involves the following steps.

Affinity matrix calculation: There are many ways to generate affinity matrix $\mathbf{A}$ depending on the way the affinity value is processed. The raw affinity value $d$ is processed by kernels such as $\exp\left(-d^{2}/\sigma^{2}\right)$, where $\sigma$ is a scaling parameter. On the other hand, the raw affinity value $d$ could also be masked by zeroing the values below a threshold to only keep the prominent values.

Laplacian matrix calculation : The graph Laplacian can be calculated in two ways: normalized and unnormalized. The degree matrix $\mathbf{D}$ contains diagonal elements $d_{i}=\sum_{j=1}^{n}a_{ij}$ where $a_{ij}$ is the element of the $i$-th row and $j$-th column in an affinity matrix $\mathbf{A}$.

Eigen decomposition: The graph Laplacian matrix $\mathbf{L}$ is decomposed into the eigenvector matrix $\mathbf{X}$ and the diagonal matrix that contains eigenvalues. Thus, $\mathbf{L}=\mathbf{X}\mathbf{\Lambda}\mathbf{X}^{\top}$.

Re-normalization (optional): the rows of $\mathbf{X}$ is normalized so that $y_{ij}=x_{ij}/\left(\sum_{j}x_{ij}^{2}\right)^{1/2}$ where $x_{ij}$ and $y_{ij}$ are the elements of the $i$-th row and $j$-th column in matrices $\mathbf{X}$ and $\mathbf{Y}$, respectively.

Speaker counting: Speaker number is estimated by finding the maximum eigengap .

Among many variations of spectral clustering algorithm, the Ng-Jordan-Weiss (NJW) algorithm is often employed for the speaker diarization task with variation in the kernel for the calculation of the affinity values . Unlike the AHC approach, spectral clustering is mostly used with cosine distance . In addition, the LSTM based similarity measurement with spectral clustering also exhibited competitive performance. Depending on the datasets, the spectral clustering approach with cosine distance measurement outperforms AHC with PLDA while using the same speaker representation for both clustering methods.

5.3 Other Clustering Algorithms

The k-means algorithm is often employed in studies on speaker diarization due to its simplicity and ease of implementation. However, the k-means algorithm generally underperforms the well-known clustering algorithms such as spectral clustering and AHC. In addition, there are a few speaker diarization studies employed the mean-shift clustering algorithm, which assigns the given data points to the clusters iteratively by finding the modes in a non-parametric distribution. The Mean-shift clustering algorithm was employed in the speaker diarization task with KL distance in , i-vector and cosine distance in , and i-vector and PLDA in .

6 Post-processing

Resegmentation is a process of refining the speaker boundary that is roughly estimated using the clustering procedure. In , the Viterbi resegmentation method based on the Baum-Welch algorithm was introduced. In this method, the estimation of Gaussian mixture model corresponding to each speaker and Viterbi-algorithm-based resgmentation using the estimated speaker GMM are alternately applied.

A method for representing the diarization process based on the variational Bayeian hidden Markov model (VB-HMM) was proposed, and was shown to be superior to Viterbi resegmentation . The VB-HMM-based diarization can be seen as a joint optimization of segmentation and clustering, which will be separately introduced in Section 2.7.

6.2 System Fusion

As another direction of post processing, there have been a series of studies on the fusion method of multiple diarization results to improve the diarization accuracy. While it is widely known that the system combination generally yields better result for various systems (e.g., speech recognition or speaker recognition ), the combination of multiple diarization hypotheses poses several unique problems. First, speaker labeling is not standardized among different diarization systems. Second, the estimated number of speakers may differ among different diarization systems. Finally, the estimated time boundaries may also be different among multiple diarization systems. System combination methods for speaker diarization systems need to handle these problems during the fusion process of multiple hypotheses.

In , a method for selecting the best diarization result among many diarization systems was proposed. In this method, a whole sequence of diarization result for a recording from each diarization system is treated as one object to be clustered. AHC is applied to the set of diarization results, in which the distance of two clusters is measured using the symmetric DER between the diarization results belonging to the two clusters. The iterative merging process of AHC is executed until the number of clusters becomes two. Finally, in the bigger cluster among the two final clusters according to the number of elements in each cluster, the diarization result that has the smallest distance to all other diarization results is selected as the final result. In , two diarization systems are combined by finding the matching between two speaker clusters, and then performing resegementation based on the matching result.

More recently, the diarization output voting error reduction (DOVER) method was proposed to combine multiple diarization results based on the voting scheme. In the DOVER method, the speaker labels among different diarization systems are aligned one by one to minimize the DER between the hypotheses (processes 2 and 3 of Fig. 6). After aligning all hypotheses, each system votes its speaker label to each segmented region (each system may have different weights for voting), and the speaker label that gains the highest voting weight is selected for each segmented region (the process 4 of Fig. 6). In case multiple speaker labels get the same voting weight, a heuristic approach is employed to break the ties (such as selecting the result from the first system) is used.

The DOVER method has an implicit assumption that there is no overlapping speech, i.e., at most only one speaker is assigned for each time index. To combine the diarization hypotheses with overlapping speakers, two methods were recently proposed. In , the authors proposed the modified DOVER method, in which the speaker labels in different diarization results are first aligned with a root hypothesis, and the speech activity of each speaker is estimated based on the weighted voting score for each speaker for each small segment. Raj et al. proposed a method called DOVER-Lap, in which the speakers of multiple hypotheses are aligned via a weighted k-partite graph matching, and the number of speakers $K$ for each small segment is estimated based on the weighted average of multiple systems to select the top-$K$ voted speaker labels. Both the modified DOVER and DOVER-Lap showed DER improvement for the speaker diarization result with speaker overlaps.

7 Joint Optimization of Segmentation and Clustering

This subsection introduces a VB-HMM-based diarization technique, which can be regarded as a joint optimization of segmentation and clustering, and thus cannot be well categorized in Section 2.1–2.6. The VB-HMM framework was proposed as an extension of the VB-based speaker clustering by introducing HMM to constrain the speaker transitions. In the VB-HMM framework , the speech feature $\mathbf{X}=(\mathbf{x}_{t}|t=1,...,T)$ is assumed to be generated from HMM where each HMM state corresponds to one of $K$ possible speakers. Suppose that we have $M$ HMM states, $M$-dimensional variable $\mathbf{Z}=(\mathbf{z}_{t}|t=1,...,T)$ is introduced where $k$-th element of $\mathbf{z}_{t}$ is 1 if the $k$-th speaker is speaking at the time index $t$, and 0 otherwise. At the same time, the distribution of $\mathbf{x}_{t}$ is modeled based on a hidden variable $\mathbf{Y}=\{\mathbf{y}_{k}|k=1,...,K\}$, where $\mathbf{y}_{k}$ denotes a low dimensional vector for the $k$-th speaker. Given these notations, the joint probability of $\mathbf{X}$, $\mathbf{Y}$, and $\mathbf{Z}$ is decomposed as follows:

where $P(\mathbf{X}|\mathbf{Z},\mathbf{Y})$ is the emission probability modeled by GMM whose mean vector is represented by $\mathbf{Y}$, $P(\mathbf{Z})$ is the transition probability of the HMM, and $P(\mathbf{Y})$ is the prior distribution of $\mathbf{Y}$. Because $\mathbf{Z}$ represents the trajectory of speakers, the diarization problem can be expressed as the inference problem of $\mathbf{Z}$ that maximizes the posterior distribution $P(\mathbf{Z}|\mathbf{X})=\int P(\mathbf{Z},\mathbf{Y}|\mathbf{X})d\mathbf{Y}$. Since it is intractable to directly solve this problem, the VB method is used to estimate the model parameters that approximate $P(\mathbf{Z},\mathbf{Y}|\mathbf{X})$.

Recently, a simplified version of VB-HMM that works on the x-vector, known as VBx, was proposed . In VBx, $P(X|Z,Y)$ is calculated using the x-vector based on the PLDA model. While the original VB-HMM works on the granularity of the frame-level feature, VBx works on the granularity of the x-vector, and thus can be seen as a clustering method that jointly models speaker turn and speaker duration.

The VB-HMM diarization was originally designed as a standalone diarization framework. However, it requires parameter initialization to start the VB estimation, and the parameters are usually initialized based on the result of another speaker clustering. In that context, VB-HMM is widely employed as the final step of speaker diarization (e.g., ). For example in , AHC was first performed to under-cluster the x-vector, and VBx was then applied to obtain a better cluster given the AHC-based result as the initial parameter. Finally, VB-HMM was further applied to refine the boundary obtained by VBx.

‡ EEND itself is a joint model of SAD, segmentation, speaker embedding, and clustering. ⋆ It can also be seen that a speech separation module is implicitly embedded in the model to cope with speaker overlaps.

Recent Advances in Speaker Diarization Using Deep Learning

This section introduces various recent efforts toward deep learning-based speaker diarization techniques. The methods that incorporate deep learning into a single component of speaker diarization, such as clustering or post-processing, are introduced in Section 3.1. The methods that unify several components of speaker diarization into a single neural network are introduced in Section 3.2. For the overview of speaker diarization techniques using deep learning, refer to Table 2.7. It should be noted that there are some works that take additional input of speaker profiles. These methods may not be categorized as a diarization technique in a traditional definition. Nevertheless, we introduce them as they are optimized in a multispeaker situation to learn the relations between speakers and hence categorized as “Trained Based on the Diarization Objective” in Table 1.3.

Enhancing the clustering procedure based on the deep learning is an active research area and several methods have been proposed for speaker diarization. This section will cover the representative works in such a direction.

An approach based on the graph neural network (GNN) was proposed in . As shown in Fig. 7, this method aims at purifying the similarity matrix used in the spectral clustering (Section 2.5.2). Assuming a sequence of speaker embeddings $\{\mathbf{e}_{1},...\mathbf{e}_{N}\}$, where $N$ is the sequence length, the input $\mathbf{x}_{i}^{0}$ to the GNN is $\{\mathbf{x}_{i}^{0}=\mathbf{e}_{i}|i=1,\ldots,N\}$. The output $\mathbf{x}_{i}^{(p)}$ of the $p$-th layer of the GNN is now:

where $\mathbf{L}$ is a normalized affinity matrix added by self-connection, $\mathbf{W}$ is a trainable weight matrix for the $p$-th layer, and $\sigma(\cdot)$ is a nonlinear function. The GNN is trained by minimizing the distance between the reference and the estimated affinity matrices. The distance is calculated using a combination of histogram loss and nuclear norm . The GNN-based speaker diarization method was evaluated on the CALLHOME dataset and an in-house meeting dataset, and significantly outperformed any of the conventional clustering methods.

Besides that, different approaches have been proposed to generate the affinity matrix. In , a self-attention-based neural network model was introduced to directly generate a similarity matrix from a sequence of speaker embeddings. In , several affinity matrices with different temporal resolutions were fused into a single affinity matrix based on a neural network.

A different approach aiming at improving clustering was proposed in , called deep embedded clustering (DEC). The goal of DEC was to transform the input features (herein referred to as speaker embeddings) making them more separable in a given number of clusters/speakers. In order to make cluster differentiable, each embedding is provided with a probability of “belonging” to each of the available speaker clusters, i.e. $q_{i,j}$ can be interpreted as the probability of assigning sample $i$ to cluster $j$ (i.e., a soft assignment):

where $z_{i}$ is the bottleneck feature, $a$ is the degree of freedom of the Student’s $t-$distribution, $\mu_{i}$ is the centroid of $i$-th cluster and $f_{i}$ is the soft cluster frequency with $f_{i}=\sum q_{ij}$. The clusters are iteratively refined based on the target distribution according to the bottleneck features estimated using an autoencoder.

The initial version of DEC had some problems, and refined algorithm called improved DEC (IDEC) was later proposed with better accuracy on speaker diarization . Firstly, there was a potential risk that the neural network is converged to a trivial solution to generates corrupted embeddings. To avoid this risk, Guo et al. proposed to explicitly preserve the local structure of the data by adding a reconstruction loss between the output of the autoencoder and the input feature. Dimitriadis further addressed the issue by introducing the loss function to enforce the distribution of speaker turns being uniform across all speakers, i.e., all speakers contribute equally to the session. This assumption is not always valid for real recordings but it constrains the solution space enough to avoid the empty clusters without affecting the overall performance. Finally, Dimitriadis also proposed an additional loss term that penalizes the distance from the centroid $\mu_{i}$, bringing the behavior of the algorithm closer to k-means.

Overall, the loss function of the IDEC consists of four loss terms, i.e., $L_{c}$, the clustering error term that is originally proposed in DEC; $L_{r}$, the reconstruction error term ; $L_{u}$ is the uniform “speaker airtime” distribution loss ; and $L_{MSE}$, the loss to measure the distance of the bottleneck features from the centroids ,

where $\alpha,\beta,\gamma,\text{ and }\delta$ are the weight on the loss functions that is fine-tuned on some held-out data.

1.2 Learning the Distance Estimator

In this section a novel approach using a trainable distance function is presented. The basic idea is based on the relational recurrent neural networks (RRNNs). RRNNs were introduced by to address “relational information learning” problems. Such models learn relations between a sequence of input features like the notion of “closer” or “further”, e.g, two points in space are closer than a third one, etc. Speaker diarization can be seen as part of this class of problems, since the final decision depends on the distance between speech segments and speaker profiles or centroids.

There are several issues that potentially limits the accuracy of speaker diarization systems. Firstly, as mentioned in Section 2.3, the duration of segments when extracting speaker embeddings poses a trade-off between the time resolution and the robustness of the extracted speaker representations. Secondly, speaker embedding extractors are not explicitly trained to provide optimal representations for speaker diarization, despite the fact these invariant, discriminative representations are used to separate thousands of speakers . Thirdly, the distance metric is often based on a heuristic approach and/or dependent on certain assumptions that do not necessarily hold, e.g., assuming Gaussianity in the case of PLDA . Finally, the audio chunks are treated independently and any temporal context is simply ignored in conventional clustering methods as described in Section 2.5. These issues can be attributed to the distance metric function, and most of them can be addressed with RRNNs, where a data-driven, memory-based approach bridges the performance gap between the heuristic and the trainable distance estimation approaches.

In this context, an approach for learning the relationship between the speaker cluster centroids (or speaker profiles) and the embeddings is proposed in (Fig. 8). In this work, the diarization process is considered to be a classification task on an already segmented audio, as in Section 2.3, either uniformly or based on estimated speaker-change salient points . The speaker embeddings $x_{j}$ for each segment, which are assumed to be speaker-homogeneous, are extracted and then compared with all the available speaker profiles or speaker centroids. The most suitable speaker label is assigned to each segment by minimizing the distance-based loss function, i.e., the relationship between embeddings and profiles. As discussed in , the RRNN-based distance estimation exhibits consistent improvements in its performance when compared with the more traditional distance estimation approaches such as the cosine distance or the PLDA-based distance. Note that although the task in is speaker identification, an extension to the speaker diarization is rather straightforward when the speaker profiles are pre-estimated, either as centroids using any of the traditional clustering algorithm (Section 2.5 and 3.1.1), or using prior knowledge.

1.3 Post Processing Based on Deep Learning

Because TS-VAD requires the i-vectors of speakers, pre-processing to obtain the i-vectors is necessary. The procedure proposed in is as follows:

Estimate i-vectors for each speaker given the diarization result.

Apply TS-VAD given the estimated i-vectors.

Refine i-vectors given the TS-VAD result.

TS-VAD was proposed as a part of the winning system of CHiME-6 Challenge , and showed a significantly better DER compared with the conventional clustering based approach . However, it has a drawback, i.e., the maximum number of speakers that the model can handle is limited by the dimension of the output vector.

As a different approach, Horiguchi et al. proposed the application of the EEND model (detailed in Section 3.2.4) to refine the result of a clustering-based speaker diarization . A clustering-based speaker diarization method can handle a large number of speakers but unable to handle overlapped speech. Conversly, EEND has opposite characteristics. To complementarily use the two methods, The authors in first applied a conventional clustering method. Then, the two-speaker EEND model was iteratively applied for each pair of detected speakers to refine the time boundary of overlapped regions.

2 Joint Optimization for Speaker Diarization

To model the distribution of the speaker change, UIS-RNN then introduces a latent variable $\mathbf{Z}=(z_{t}\in\{0,1\}|t=2,\ldots,T)$, where $z_{t}$ becomes 1 if the speaker indices at time $t-1$ and $t$ are different, and 0 otherwise. The joint probability including $\mathbf{Z}$ is then decomposed as follows.

Finally, the term $P(\mathbf{x}_{t},y_{t},z_{t}|\mathbf{x}_{1:t-1},y_{1:t-1},z_{1:t-1})$ is further decomposed into three components.

Here, $P(\mathbf{x}_{t}|\mathbf{x}_{1:t-1},y_{1:t})$ denotes the sequence generation probability, and modeled by gated recurrent unit (GRU)-based recurrent neural networks. $P(y_{t}|z_{t},y_{1:t-1})$ denotes the speaker assignment probability and modeled by a distance-dependent Chinese restaurant process , which can model the distribution of unbounded number of speakers. Finally, $P(z_{t}|z_{1:t-1})$ represents the speaker change probability and is modeled by the Bernoulli distribution. Since all models are represented by trainable ones, UIS-RNN can be trained in a supervised fashion by finding the parameters that maximize $\log P(\mathbf{X},\mathbf{Y},\mathbf{Z})$ over training data. The inference can be conducted by finding $\mathbf{Y}$ that maximizes $\log P(\mathbf{X},\mathbf{Y})$ given $\mathbf{X}$ based on the beam search in an online fashion. While UIS-RNN works in an online fashion, it demonstrated better DER than that of the offline system based on spectral clustering.

2.2 Joint Segmentation, Embedding Extraction, and Re-segmentation

A speaker diarization method based on the region proposal networks (RPN) was proposed to jointly perform segmentation, speaker embedding extraction, and resegmentation . The RPN was originally proposed to detect multiple objects from a two-dimensional image , and one-dimensional variant along with the time-axis is used for speaker diarization.

As can be seen from Fig. 10 (a), the STFT features with a size of time and frequency bin is first converted to the feature map with a size of time, frequency and channels using CNNs. Then, other three types of neural networks are applied on various sizes of sliding windows (named “anchor”) along with the time axis. For each anchor, the three neural networks perform SAD, speaker embedding extraction, and region refinement, respectively. Here, SAD is the task to estimate the probability of speech activity for the anchor region. Speaker embedding extraction is the task to generate an embedding to represent the speaker characteristics of the audio corresponding to the anchor region. Finally, region refinement is the task to estimate the difference between the shape (i.e. duration and center position) of the anchor and that of the corresponding reference region.

The inference procedure by RPN is presented in Fig. 10 (b). RPN is first applied to list the anchors with speech activity probability higher than the pre-determined threshold. The anchors are then clustered using a conventional clustering method (e.g., k-means) based on the speaker embeddings estimated for each anchor. Finally, highly overlapped anchors after region refinement are removed, a method known as the non-maximum suppression.

The RPN-based speaker diarization system has the advantage of handling overlapped speech with possibly any number of speakers. Also, it is much simpler than the conventional speaker diarization system. It was shown in multiple datasets that this system achieved significantly better DER than the conventional clustering-based speaker diarization system .

2.3 Joint Speech Separation and Diarization

There are also recent researches on the joint modeling of speech separation and speaker diarization. Kounades-Bastian et al. proposed the incorporation of a speech activity model into speech separation based on the spatial covariance model with non-negative matrix factorization. They derived the EM algorithm to estimate separated speech and the speech activity of each speaker from the multichannel overlapped speech. While their method jointly performs speaker diarization and speech separation, it is based on a statistical modeling, and the estimation was conducted solely based on the observation, i.e. without any model training.

($\mathbf{e}_{b-1,i}$ is set to $\mathbf{0}$ if it was not calculated previously) ii. $\mathbf{R}_{b,i}=\max(\mathbf{R}_{b,i-1}-\mathbf{M}_{b,i},\mathbf{0})$ iii. If $\frac{1}{TF}\sum_{t,f}\mathbf{R}_{b,i}(t,f)<threshold$, stop iteration.

A separated speech for speaker $i$ at audio block $b$ can be obtained by $\mathbf{M}_{b,i}\odot\mathbf{X}_{b}$ where $\odot$ is the element-wise multiplication. The speaker embedding $\mathbf{e}_{b,i}$ is used to keep track of the speaker of adjacent blocks. Thanks to the iterative approach, this neural network can cope with the variable number of speakers while jointly performing speech separation and speaker diarization. The online RSAN was evaluated by using real meeting dataset with up to six speakers, and showed better results than the clustering-based method .

2.4 Fully End-to-end Neural Diarization

EEND has multiple advantages. First, EEND can handle overlapping speech in a sound way. Second, the network is directly optimized toward the maximization of diarization accuracy, by which we can expect a high accuracy. Third, it can be retrained by a real data (i.e. not synthetic data) just by feeding a reference diarization label while it is often not straitforward for the prior works. However, EEND also has several limitations. First, the model architecture limits the maximum number of speakers that the model can cope with. Second, EEND consists of BLSTM or self-attention based neural networks, making it difficult to do online processing. Third, it was empirically suggested that EEND tends to overfit to the distribution of the training data .

To cope with an unbounded number of speakers, several extensions of EEND have been investigated. Horiguchi et al. proposed an extension of EEND with the encoder-decoder-based attractor (EDA) (Fig. 13). This method applies an LSTM-based encoder-decoder on the output of EEND to generate multiple attractors. Attractors are generated until the attractor existing probability becomes less than the threshold. Then, each attractor is multiplied by the embeddings generated from EEND to calculate the speech activity for each speaker. EEND-EDA was evaluated on CALLHOME (two to six speakers) and DIHARD 2 (one to nine speakers) dataset and showed better performance than the clustering-based baseline system.

On the other hand, Fujita et al. proposed another approach to output the speech activity one after another by using a conditional speaker chain rule. In this method, a neural network is trained to produce a posterior probability $P(\mathbf{y}_{k}|\mathbf{y}_{1},\ldots,\mathbf{y}_{k-1},\mathbf{X})$, where $\mathbf{y}_{k}=(y_{t,k}\in\{0,1\}|t=1,\ldots,T)$ is the speech activity for the $k$-th speaker. Then, the joint speech activity probability of all speakers can be estimated from the following speaker-wise conditional chain rule as:

During inference, the neural network is repeatedly applied until the speech activity $y_{k}$ for the last estimated speaker approaches zero. Kinoshita et al. proposed a different approach that combines EEND and speaker clustering. In their method, a neural network is trained to generate speaker embeddings and the speech activity probability. Speaker clustering constrained by the estimated speech activity by EEND is applied to align the estimated speakers among the different processing blocks.

There are also a few recent trials to extend the EEND for online processing. Xue et al. proposed a method using a speaker tracing buffer to better align the speaker labels of adjacent processing blocks. Han et al. proposed a block online version of EEND-EDA by carrying the hidden state of the LSTM-encoder to generate the attractors block by block.

Speaker Diarization in the Context of ASR

From a conventional perspective, speaker diarization is considered a pre-processing step for ASR. In the traditional system structures for speaker diarization, presented in Fig. 1, speech inputs are processed sequentially across the diarization components without considering the ASR performance, which is usually measured using the word error rate (WER). WER is the number of misrecognized words (substitution error, insertion error, and deletion error) divided by the number of reference words. One issue is that the tight boundaries of speech segments as the outcomes of speaker diarization have a high chance of causing unexpected word truncation or deletion errors in ASR decoding. In this section we discuss how the speaker diarization systems have been developed in the context of ASR, not only resulting in better WER by preventing speaker diarization from affecting the ASR performance, but also benefiting from ASR artifacts to enhance diarization performance. More recently, there have been a few pioneering proposals made for the joint modeling of speaker diarization and ASR, which will also be introduced in this section.

The lexical information from the ASR output has been employed for the speaker diarization system in a few different ways. First, the earliest approach was the RT03 evaluation which used word boundary information for the purpose of segmentation. In , a general ASR system for broadcast news data was built, in which the basic components are segmentation, speaker clustering, speaker adaptation and system combination after ASR decoding from the two sub-systems with the different adaptation methods. The authors used the word boundary information from the ASR system for speech segmentation, and compared it with the BIC-based speech segmentation. While the performance gain by the ASR-based segmentation was insignificant, this was the first attempt to take advantage of ASR output to enhance the diarization performance. In addition, the ASR result was used to refine SAD in IBM’s submission for RT07 evaluation. The system that appeared in incorporates word alignments from the speaker independent ASR module and refines the SAD result to reduce false alarms so that the speaker diarization system can have better clustering quality. The segmentation system in also takes advantage of word alignments from ASR. The authors in focused on the word-breakage problem, in which the words from the ASR output are truncated by segmentation results since the segmentation results and the decoded word sequences are not aligned. Therefore, word-breakage ratio was proposed to measure the rate of change points detected inside intervals corresponding to words. The DER and word-breakage ratio were used to measure the influence of the word truncation problem. While the aforementioned early works of speaker diarization systems that leverage the ASR output focus on the word alignment information to refine the SAD or segmentation result, the speaker diarization system in created a dictionary for the phrases commonly appearing in broadcast news. The phrases in this dictionary provide the identity of who is speaking, who will speak and who spoke in the broadcast news scenario. For example, “This is [name]” indicates who was the speaker of the broadcast news section. Although the early studies on speaker diarization did not fully leverage the lexical information to drastically improve the DER, the idea of integrating the information from ASR output has been adopted by many studies to refine or improve the speaker diarization output.

2 Using lexical information from ASR

The more recent speaker diarization systems that take advantage of the ASR transcript have employed a DNN model to capture the linguistic pattern in the given ASR output to enhance the speaker diarization result. The authors in proposed a way of using the linguistic information for the speaker diarization task where participants have distinct roles that are known to the speaker diarization system. Fig. 14 shows the diagram of the speaker diarization system discussed in . In this system, a neural text-based speaker change detector and a text-based role recognizer are employed. By using both linguistic and acoustic information, DER was significantly improved compared with the acoustic only system.

Lexical information from the ASR output was also used for speaker segmentation by employing a sequence-to-sequence model that outputs speaker turn tokens. Based on the estimated speaker turn, the input utterance is segmented accordingly. The experimental results in indicate that using both acoustic and lexical information can be exploited and an extra advantage can be obtained owing to the word boundaries we get from the ASR output.

The authors of presented follow-up research within the above thread. Unlike the system in , the lexical information from the ASR module was integrated with the speech segment clustering process by employing an integrated adjacency matrix. The adjacency matrix is obtained from the max operation between the acoustic information created from affinities among audio segments and lexical information matrix created by segmenting the word sequence into word chunks that are likely to be spoken by the same speaker. Fig. 15 presents a diagram that explains how lexical information is integrated in an affinity matrix with acoustic information. The integrated adjacency matrix leads to an improved speaker diarization performance for the CALLHOME American English dataset.

3 Joint ASR and Speaker Diarization with Deep Learning

Motivated by the recent success of deep learning and end-to-end modeling, several models have been proposed to jointly perform ASR and speaker diarization. As with the previous section, the ASR results contain a strong cue to improve speaker diarization. On the other hand, speaker diarization results can be used to improve the accuracy of ASR, for example, by adapting the ASR model toward each estimated speaker. Joint modeling can leverage such inter-dependency to improve both ASR and speaker diarization. In the evaluation, a WER metric that counts word hypotheses with speaker-attribution errors as misrecognized words, such as speaker-attributed WER or concatenated minimum-permutation WER (cpWER) , is often used. ASR-specific metrics (e.g., speaker-agnostic WER) or diarization-specific metrics (e.g., DER mentioned in Section 1.4.1) are also used complementarily.

The first approach is the introduction of a speaker tag in the transcription of end-to-end ASR models (Fig. 16). Shafey et al. proposed to insert a speaker role tag (e.g., $\langle$doctor$\rangle$ and $\langle$patient$\rangle$) into the output of a recurrent neural network-transducer (RNN-T)-based ASR system. This method was evaluated by using doctor-patient conversation, and a significant reduction in WDER was reported with a marginal degradation of WER. Similarly, Mao et al. proposed the insertion of a speaker identity tag into the output of an attention-based encoder-decoder ASR system, and showed an improvement of DER especially when the oracle utterance boundaries were not given. The works by Shafey et al. and Mao et al. showed that the insertion of speaker tags is a simple and promising way to jointly perform ASR and speaker diarization. On the other hand, the speaker roles or speaker identity tags need to be determined and fixed during training. Thus, it is difficult to cope with an arbitrary number of speakers using this approach.

where the Viterbi approximation is applied to obtain the final equation. This maximization problem is further decomposed into two iterative problems as follows:

where $i$ is the iteration index of the procedure. In , Eq. (57) is modeled by the target speaker ASR and Eq. (58) is modeled by the overlap-aware speaker embedding estimation. This method obtains a speaker-attributed WER similar to that of the target-speaker ASR with oracle speaker embeddings for two-speaker conversation data of the Corpus of Spontaneous Japanese . On the other hand, it requires an iterative application of the target-speaker ASR and a speaker embedding extraction scheme, which make it challenging to apply the method in online mode.

As a third line of approaches, end-to-end speaker-attributed ASR (SA-ASR) model was recently proposed to jointly perform speaker counting, multi-talker ASR, and speaker identification . Contrary to the first two approaches, the end-to-end SA-ASR model takes the additional input of speaker profiles and identifies the index of speaker profiles based on the attention mechanism (Fig. 18). Thanks to the attention mechanism for the speaker identification and multi-talker ASR capability based on serialized output training , there is no limitation in the number of speakers that the model can cope with. In case relevant speaker profiles are supplied in the inference, the end-to-end SA-ASR model can automatically transcribe the utterance while identifying the speaker of each utterance based on the supplied speaker profiles. On the other hand, in case of the relevant speaker profiles cannot be used prior to the inference, the end-to-end SA-ASR model can still be applied using dummy profiles, and the speaker clustering on the internal speaker embeddings of the end-to-end SA-ASR model (“speaker query” in Fig. 18) is used to diarize the speaker . The end-to-end SA-ASR model was evaluated by using the LibriCSS dataset , and exhibited significantly better cpWER than the combination of multitalker ASR and speaker diarization .

Diarization Evaluation Series and Datasets

This section describes the evaluation series and the commonly used datasets for speaker diarization evaluations. The summary of the most commonly used datasets that include English is shown in Table 5.

† Most of the contents are English while there are few non-English contents.

NIST SRE 2000 (Disk-8), often referred to as the CALLHOME dataset, is the most widely used dataset for speaker diarization in recent papers. This dataset contains 500 sessions of multilingual telephonic speech. Each session has two to seven speakers with two dominant speakers in each conversation.

AMI Corpus The AMI database includes 100 h of meeting recordings from multiple sites in 171 meeting sessions. The AMI database provides an audio source recorded using lapel microphones that are separately recorded and amplified for each speaker. Another audio source is recorded using omnidirectional microphone arrays mounted on the table while meeting. The AMI database is a suitable dataset for the evaluation of speaker diarization systems integrated with the ASR module since AMI provides forced alignment data that contains word and phoneme level timings along with the transcript and speaker label. Each meeting session has three to five speakers. • ICSI Meeting Corpus The ICSI meeting corpus contains 75 meeting corpus with four meeting types. The ICSI meeting corpus provides word level timing along with the transcript and speaker label. The audio source is recorded using close-talking individual microphone and six tabletop microphones to provide speaker-specific channel and multichannel recording. Each meeting has 3 to 10 participants.

CHiME-5/6 challenge and its dataset The CHiME-5 challenge and CHiME-6 challenge were designed as series of ASR competitions for the daily conversation of multiple speakers. The dataset was provided at the CHiME-5 challenge, and it contains 50 h of multiparty real conversations in the everyday home environment. It contains speaker labels, segmentation, and corresponding transcriptions. The audio source is recorded using six four-channel microphone arrays located in the kitchen and dining/living rooms in a house and also binaural microphones worn by participants. The number of participants is fixed at four. While the oracle diarization results were allowed to be used for the ASR task in the CHiME-5 challenge, CHiME-6 challenge track 2 requires the result of both ASR and diarization. The primary evaluation metric for such a track was cpWER, which counts both the speaker-attributed errors and word recognition errors in the WER calculation. DER and JER were also evaluated as secondary metrics without “score collar” and with overlapped regions. The CHiME-5/6 corpus was also used as one track in the DIHARD 2 challenge.

VoxSRC Challenge and VoxConverse corpus The VoxCeleb Speaker Recognition Challenge (VoxSRC) is the recent evaluation series for speaker recognition systems . The goal of VoxSRC is to test how well the current technology can cope with the speech “in the wild”. This evaluation series initially started with a pure speaker verification task , and the diarization task was added as track 4 at the latest evaluation at the VoxCeleb Speaker Recognition Challenge 2020 (VoxSRC-20) . The VoxConverse dataset was used for the speaker diarization task with DER as the primary metric, and JER as the secondary metric. The VoxConverse dataset contains 74 h of human conversation extracted from YouTube videos. The dataset is divided into development set (20.3 h, 216 recordings), and test set (53.5 h, 310 recordings). The number of speakers in each recording has a wide range of variety from 1 speaker to 21 speakers. The audio includes various types of noises such as background music, laughter, etc. It also contains a significant portion of overlapping speech from 0% to 30.1% depending on the recording. While the dataset contains the visual information as well as audio, as of June 2021, only the audio of the development set was released under a Creative Commons Attribution 4.0 International License for research purposes. The audio of the evaluation set was used as a blind test set.

LibriCSS The LibriCSS corpus contains 10 h of multichannel recordings and was designed for the research of speech separation, speech recognition, and speaker diarization. It was created by playing the audio in the LibriSpeech corpus in a real meeting room, and recorded using a 7-channel microphone array. It consists of 10 sessions, each of which is further decomposed to six 10-min mini-sessions. Each mini-session was made by audio of eight speakers and designed to have different overlap ratios from 0% to 40%. To facilitate the research, the baseline system for speech separation and ASR and the baseline system that integrates speech separation, speaker diarization and ASR have been developed and released.

DIHARD Challenge and its dataset DIHARD evaluation focuses on the performance gap of state-of-the-art diarization systems on challenging domains. The first DIHARD challenge, DIHARD 1, started with track 1 (oracle SAD) and track 2 (system SAD). The evaluation data was a collection of various corpus. It includes very challenging datasets such as clinical interviews, web videos, and speech in the wild (e.g., recordings in restaurants), as well as relatively less challenging datasets, such as CTS and audio books to diversify the domains. DIHARD 2 additionally included multichannel speaker diarization task in track 3 (oracle SAD) and track 4 (system SAD) using the recordings from the CHiME-5 corpus . In the latest DIHARD challenge, DIHARD 3, the CTS dataset was added, whereas multichannel tracks 3 and 4 were excluded. The DIHARD challenge employs DER and JER for the evaluation metric without “score collar” and with overlap region.

Rich Transcription Evaluation Series The RT Evaluation is the pioneering evaluation series of initiating deeper investigation on speaker diarization in relation to ASR. The main purpose of this effort was to create ASR technologies that would produce transcriptions with descriptive metadata, like who said when, where speaker diarization plays in. Thus, the main tasks in the evaluation were ASR and speaker diarization. The domains of the data of interest were broadcast news, CTS and meeting recordings with multiple participants. Throughout the period of 2002-2009, the RT evaluation series promoted and gauged advances in speaker diarization and ASR technology. The evaluations among this period are named as RT evaluations (RT-02, RT-03S, RT-03F, and RT-05F) and RT Meeting Recognition (RT-06S, RT-07S and RT-09). These evaluations and their datasets include speaker diarization evaluation as a part of automatic metadata extraction (MDE).

There are also several corpora that have been used for the diarization research but not covered in the list above. The Corpus of Spontaneous Japanese contains about 12 h of two-speaker dialogue recorded using headset microphones. AISHELL-4 is a relatively new Mandarin Chinese dataset containing 118 h of four to eight speakers in a conference scenario. It is recorded by 8-ch circular microphone array as well as headset microphones for each participant. The ESTER-1 and ESTER-2 evaluation campaign datasets are a set of French recordings designed for three task category: Segmentation (S), Transcription (T) and Information Extraction (E). In the ESTER-1 and ESTER-2 evaluation campaign, speaker diarization was evaluated as one of the core tasks among other tasks including speaker tracking, sound event tracking, and transcriptions. The datasets for ESTER-1 and ESTER-2 include 100 h and 150 h of manually transcribed French radio broadcast news, respectively. ETAPE is also a French speech processing evaluation dataset that contains 36 h of TV and radio shows with both prepared and spontaneous speech. Unlike the ESTER evaluation series, ETAPE targets cross-show speaker diarization.

Applications

The goal of meeting transcription is to automatically generate speaker-attributed transcripts during real-life meetings based on their audio and optional video recordings. Accurate meeting transcription is among the processing steps in a pipeline for several tasks, such as, summarization and topic extraction. Similarly, the same transcription system can be used in other domains such as healthcare .

Although this task was introduced by NIST in the RT evaluation series back in 2003 , the initial systems had very poor performance, and consequently commercialization of the technology was not possible. However, recent advances in the areas of speech recognition , far-field speech processing , speaker ID and diarization , have greatly improved the speaker-attributed transcription accuracy, enabling such commercialization. Bimodal processing combining cameras with microphone arrays has further improved the overall performance .

The variety of application scenarios, customer needs, and business scope, different constraints may be imposed on meeting transcription systems. For example, it is most often required to provide the resulting transcriptions in low latency, making the diarization and recognition even more challenging. However, the architecture of the transcription system can substantially improve the overall performance, e.g., by using microphone arrays of known geometry as the input device. Also, in the case where the expected meeting attendees are known beforehand, the transcription system can further improve speaker attribution, all while providing the exact name of the speaker, instead of randomly generated discrete speaker labels.

Two different scenarios in this space are presented: first, a fixed-geometry microphone array combined with a fish-eye camera system. Second, an ad-hoc geometry microphone array system without a camera. In both scenarios, a “non-binding” list of participants and their corresponding speaker profiles are considered to be known. In particular, the transcription system has access to the invitees’ names and profiles; however, the actual attendees may not accurately match those invited. As such, there is an option to include “unannounced” participants. In addition, some of the invitees may not have profiles. In both scenarios, there is a constraint of low-latency transcriptions, where initial results need to be shown with low latency. The finalized results can be updated later in an offline mode. Some of the technical challenges to overcome are :

Although ASR on overlapping speech is one of the main challenges in meeting transcription, limited progress has been made over the years. Numerous multichannel speech separation methods have been proposed based on independent component analysis (ICA) or spatial clustering , but their application to a meeting setup had limited success. In addition, neural network-based separation methods such as permutation invariant training (PIT) or deep clustering (DC) cannot adequately address reverberation and background noise .

Flexible framework: It is desirable that the transcription system is capable of processing all the available information, such as the multichannel audio and visual cues. The system needs to process a dynamically changing number of audio channels without loss of performance. As such, the architecture needs to be modular enough to encompass the different settings.

The speaker-attributed ASR of natural meetings requires online or streaming ASR, audio pre-processing such as dereverberation, and accurate diarization and speaker identification. These multiple processing steps are usually optimized separately and thus, the overall pipeline is most frequently inefficient.

The use of multiple, unsynchronized audio streams, e.g., audio capturing using mobile devices, adds complexity to the meeting setup and processing. In return, we gain a potentially better spatial coverage since the devices are usually distributed around the room and near the speakers. As part of the application scenario, the meeting participants bring their personal devices, which can be repurposed to improve the overall quality of meeting transcription quality. On the other hand, while there are several pioneering studies , it is unclear what the best strategies are for consolidating multiple asynchronous audio streams and to what extent they work for natural meetings in online and offline setups.

Based on these considerations, an architecture of a meeting transcription system with asynchronous distant microphones has been proposed in . In this work, various fusion strategies have been investigated: from early fusion beamforming of the audio signals, to mid-fusion combination of senones per channel, to late fusion combination of the diarization and ASR results . The resulting system performance was benchmarked on real-world meeting recordings against fixed geometry systems. As aforementioned, the requirement of speaker-attributed transcriptions with low latency was also adhered to. In addition to the end-to-end system analysis, the paper proposed the idea of “leave-one-out beamforming” in the asynchronous multi-microphone setup, enriching the “diversity” of the resulting signals, as proposed in . Finally, it is described how an online, incremental version of recognizer output voting error reduction (ROVER) can process both the ASR and diarization outputs, enhancing the overall speaker-attributed ASR performance.

2 Conversational Interaction Analysis and Behavioral Modeling

Speech and spoken language are central to conversational interactions. They carry crucial information about a speaker’s intent, emotions, identity, age and other individual and interpersonal traits and state variables including health state. Computational advances are increasingly allowing access to such rich information . For example, knowing how much, and how, a child speaks in an interaction contains critical information about the child’s developmental state, and offers clues to clinicians in diagnosing disorders such as autism . Such analyses are made possible by capturing and processing the audio recordings of the interactions, which often involve two or more people. An important foundational step is the identification and association of the speech portions belonging to specific individuals involved in the conversation. The technologies providing these capabilities are SAD and speaker diarization. Speech portions segmented with speaker-specific information provided by speaker diarization, by itself without any explicit lexical transcription, can offer important information to domain experts who can take advantage of speaker diarization results for quantitative turn-taking analysis.

A domain that is the most relevant in such analyses of spoken conversational interactions relates to behavioral signal processing (BSP) , which refers to the technology and algorithms for modeling and understanding human communicative, affective and social behaviors. For example, these may include analyzing how positive or negative a person is, how empathic an individual is toward another, what the behavior patterns reveal about the relationship status, and the health condition of an individual . BSP involves addressing all the complexities of spontaneous interactions in conversations with additional challenges involved in handling and understanding emotional, social and interpersonal behavioral dynamics revealed through verbal and nonverbal cues of the interaction participants. Therefore, the knowledge of speaker specific vocal information plays a significant role in BSP, requiring highly accurate speaker diarization performance. For example, the speaker diarization module is employed as a pre-processing module for analyzing psychotherapy mechanisms and quality and suicide risk assessment .

Another popular application of speaker diarization for conversation interaction analysis is the medical doctor-patient interactions. In the system described in , the nature of memory problems of a patient is detected from the conversations between neurologists and patients. The speech and language features extracted from the ASR transcripts combined with the speaker diarization results are used to predict the type of disorder. An automated assistant system for medical domain transcription is proposed in , which includes the speaker diarization module, ASR module and natural language generation module. The automated assistant module accepts the audio clip and outputs grammatically correct sentences describing the topic of the conversation, subject and subject’s symptom.

3 Audio Indexing

Content-based audio indexing is a well known application domain for speaker diarization. It can provide metainformation such as the content or data type of a given audio data to make information retrieval efficient since search query by machines would be limited by such metadata. The more diverse information was available, the better efficiency could be achieved in retrieving audio contents from a database.

One useful piece of information for the audio indexing would be ASR transcripts to understand the content of speech portions in the audio data. Speaker diarization can augment those transcripts in terms of “who spoke when”, which was the main purpose of the RT evaluation series , as discussed in Sections 4.1 and 5.3. The aggregated spoken utterances from speakers by a speaker diarization system also enable per-speaker summary or keyword list-up, which can be used for other query values to retrieve relevant contents from the database. In , we can get a view of how speaker diarization outputs can be linked for information searching in consumer-facing applications.

4 Conversational AI

Thanks to the advances of ASR technology, the applications of ASR have evolved from simple voice command recognition systems to conversational AI systems. Conversational AI systems, as opposed to voice command recognition systems, have features that are lacking in voice command recognition systems. The fundamental idea of conversational AI is to build a machine that humans can talk to and interact with. In this sense, focusing on an interested speaker in a multiparty setting is one of the most important features of conversational AI. Moreover, speaker diarization becomes an essential feature for conversational AI. For example, conversational AI equipped in a car can pay attention to a specific speaker that is demanding a piece of information from the navigation system by applying speaker diarization along with ASR.

Smart speakers and voice assistants are the most popular products in which speaker diarization plays a significant role for conversational AI. Since the response time and online processing are the crucial factors in real-life settings, the demand for end-to-end speaker diarization systems integrated into the ASR pipeline is growing. The performance of incremental (online) ASR and speaker diarization of the commercial ASR services are evaluated and compared in . It is expected that the real-time and low latency aspect of speaker diarization will be more emphasized in the speaker diarization systems in the future since the performance of online diarization and online ASR still have much room for improvement.

Challenges and the Future of Speaker Diarization

This paper has provided a comprehensive overview of speaker diarization techniques, highlighting the recent development of deep learning-based diarization approaches. In the early days, a speaker diarization system was developed as a pipeline of sub-modules including front-end processing, SAD, segmentation, speaker embedding extraction, clustering, and post-processing, leading to a standalone system without much connection to other components in a given speech application. With the emergence of the deep learning technology, more and more advancements have been made for speaker diarization, from a method that replaces a single module into a deep-learning-based method, to a fully EEND. Furthermore, as the speech recognition technology becomes more accessible, a trend to tightly integrate speaker diarization into the ASR systems has emerged, such as benefiting from the ASR output to improve the accuracy of speaker diarization. Recently, joint modeling for speaker diarization and speech recognition is investigated in an attempt to enhance the overall performance of speaker diarization. Thanks to these great achievements, speaker diarization systems have already been used in many applications, including meeting transcription, conversational interaction analysis, audio indexing, and conversational AI systems.

As we have seen, tremendous progress has been made for speaker diarization systems. Nevertheless, there is still much room for improvement. As the final remark, we conclude this paper by listing the remaining challenges for speaker diarization toward future research and development.

Most speaker diarization methods assume that an entire recording can be observed to execute speaker diarization. However, numerous applications such as meeting transcription systems or smart agents require very short latency for assigning the speaker. While there have been several attempts to make online speaker diarization system both for clustering-based systems (e.g., ) and neural network-based diarization systems (e.g., ), it still remains as a challenging problem.

A model that is trained on a data in a specific domain often works poorly on data in another domain. For example, it is experimentally known that the EEND model tends to overfit to the distribution of the speaker overlaps of the training data . Such a domain mismatch issue is universal for any training-based method. Given the growing interest for trainable speaker diarization systems, it will become more important to evaluate the ability for handling the variety of inputs. The international evaluation efforts for speaker diarization such as the DIHARD challenge or VoxSRC also have great importance in this direction.

Overlap of multitalker speech is the inevitable nature of conversation. For example, an average of 12% to 15% of speaker overlap was observed in meeting recordings , and it could even increase in daily conversations . Nevertheless, many traditional speaker diarization systems, especially clustering-based systems, have only focused on non-overlapping regions and even the overlapping regions are excluded in the evaluation metric . While the topic has been studied for long years (e.g. early works ), there is a growing interest for handling the speaker overlaps toward better speaker diarization, including the application of speech separation , post-processing , and joint modeling of speech separation and speaker diarization .

Many applications require ASR results along with speaker diarization results. In the modular combination of speaker diarization and ASR, some systems locate a speaker diarization system before ASR while some systems locate a speaker diarization system after ASR . Both types of systems showed a strong performance for a specific task, and determining the best kind of system architecture for the speaker diarization and ASR tasks is still an open problem . Furthermore, there is another line of research to jointly perform speaker diarization and ASR , which was introduced in Section 4. The joint modeling approach could leverage the inter-dependency between speaker diarization and ASR to better perform both tasks. However, it has not yet been fully investigated whether such joint frameworks perform better than the well-tuned modular systems. Overall, the integration of speaker diarization and ASR is one of the hottest topics that is still being investigated by many researchers.

Visual information contains a strong clue for the identification of speakers. For example, the video captured by a fish-eye camera was used to improve the accuracy of speaker diarization in a meeting transcription task . The visual information was also used to significantly improve the accuracy of speaker diarization. for speaker diarization on YouTube video . While these studies showed the effectiveness of visual information, the audiovisual speaker diarization has yet been rarely investigated compared with audio-only speaker diarization, and there will be many rooms for improvement.

References