Multimodal Local-Global Ranking Fusion for Emotion Recognition

Introduction

Emotion recognition is a core research area at the intersection of artificial intelligence and human communication analysis. It has immense applications towards robotics (Liu et al., 2017; Alonso-Martin et al., 2013), dialog systems (Pittermann et al., 2010, 2009), intelligent tutoring systems (Vail et al., 2016; Malekzadeh et al., 2015; Petrovica et al., 2017), and healthcare diagnosis (Frantzidis et al., 2010). Emotion recognition is multimodal in nature as humans utilize multiple communicative modalities in a structured fashion to convey emotions (Busso et al., 2008). Two of these important modalities are acoustic and visual. In the acoustic modality, humans use prosody and various vocal expressions. In the visual modality, humans utilize facial expressions, hand gestures, and body language. Each modality is crucial when analyzing human emotions, making emotion recognition a challenging domain of artificial intelligence.

Some emotional expressions are almost universal and can be recognized directly from a video segment. For example, an open mouth with raised eyebrows and a loud voice is likely to be associated with surprise. These can be seen as person-independent behaviors and absolute emotions can be directly inferred from these behaviors (left panel of Figure 1). However, emotions are also expressed in a person-dependent fashion with idiosyncratic behaviors. In these cases, it may not be possible to directly estimate absolute emotion intensities. Instead, it would be easier to compare two video segments of the same person and judge whether there was a relative change in emotion intensities (right panel of Figure 1). For example, a person could have naturally furrowed eyebrows and we should not always interpret this as a display of anger, but rather compare all nonverbal behaviors between two video segments to determine the relative changes in his displayed anger. From a psychological approach, research has also highlighted the advantages of using ordinal (relative) representations of human signals (Miller, 1956; Stewart et al., 2005; Elliott et al., ).

In this paper, we introduce the Multimodal Local-Global Ranking Fusion (MLRF) model which performs emotion recognition by integrating both direct prediction and relative prediction approaches. This is performed by dividing the emotion recognition task into three easier multimodal subtasks (Figure 2). The first subtask is the multimodal local ranking task. Given two short segments randomly selected from an entire video, the model is tasked with determining if there was an increase or decrease in the displayed emotion intensity. This task is often simpler than the direct emotion recognition problem since the model only needs to compare relative emotion ranks rather than compute the absolute intensities. The second subtask is the global ranking task, which uses the previous results of local rankings to infer relative global emotion ranks using a Bayesian skill rating algorithm (Baltrušaitis et al., 2017; Herbrich et al., 2007). The third subtask involves direct-relative fusion of direct emotion predictions from observed multimodal behaviors with relative emotion ranks estimated from local-global ranking. This integration of direct and relative emotion predictions allows MLRF to model both person-independent and person-dependent behaviors for complete emotion recognition. We show that MLRF is suitable for multimodal tasks by performing experiments on an audio-visual emotion recognition benchmark. The proposed MLRF approach displays excellent performance over the baselines, improving over other algorithms for multimodal fusion.

Related Work

Previous approaches in multimodal emotion recognition can be categorized as follows:

Non-temporal Models: These approaches simplify the temporal aspect by averaging information through time (Abburi et al., 2016). Fusion is performed by concatenating the multimodal inputs. However, these methods tend to overfit without discovering generalizable speaker-independent and speaker-dependent features (Xu et al., 2013). More complex fusion methods learn separate models for each modality and combine the outputs (Snoek et al., 2005). However, simple decision voting is unable to discover the complex multimodal combinations involved in speaker-dependent features (Liang et al., 2018).

Temporal Models: Long Short-term Memory Networks (LSTMs) (Hochreiter and Schmidhuber, 1997; Graves et al., 2013; Schuster and Paliwal, 1997) have been extended for multimodal settings (Rajagopalan et al., 2016) and with binary gating mechanisms to remove noisy modalities (Chen et al., 2017). More advanced models use memory mechanisms (Zadeh et al., 2018a), low-rank approximations to tensor products (Liu et al., 2018), multiple attention stages (Liang et al., 2018) or assignments (Zadeh et al., 2018b) or generative-discriminative objectives to learn factorized (Tsai et al., 2018) or joint multimodal representations (Pham et al., 2018). To our knowledge, our approach is the first to approach multimodal fusion with a neural local-global ranking fusion approach. The strength of our approach lies in approaching both speaker-independent and speaker-dependent features via direct and relative emotion predictions respectively. Algorithmically, our divide-and-conquer insight simplifies the emotion recognition task into three easier multimodal subtasks and allows us to incorporate probabilistic structure as compared to entirely neural approaches.

Our work is also related to ranking algorithms. Bayesian ranking algorithms have been used in ranking the skills of players in Chess (Elo, 1978; Veček et al., 2014) and online games (Herbrich et al., 2007). Recently, ranking methods have been applied for facial expression intensity estimation (Baltrušaitis et al., 2017). To the best of our knowledge, we are the first to integrate relative measures from local-global ranking with direct predictions for emotion recognition. We also apply our approach to a multimodal setting where temporal information is primordial and there exist complex interactions between the acoustic and visual modalities.

Multimodal Local-Global Ranking Fusion

The Multimodal Local-Global Ranking Fusion (MLRF) model (Figure 2) aims to integrate both direct and relative emotion prediction approaches to model person-independent and person-dependent behaviors. This is achieved by subdividing the emotion recognition task into three easier subtasks: (1) multimodal local ranking, (2) global ranking, and (3) direct-relative fusion. Relative emotion rankings are inferred from the first two sub-tasks. The third subtask performs direct emotion predictions while at the same time integrating relative emotion rankings for final emotion recognition.

2. Multimodal Local Ranking

This defines a binary classification problem over two short segments. This task is simpler than the direct emotion recognition problem since the local ranking model only needs to compare the differences in emotion intensities rather than compute the exact emotion intensities themselves.

To solve the multimodal local ranking problem, we define an estimator $f_{local}$ with parameters $\Theta_{local}$:

Solving this problem involves multimodal fusion since $\mathbf{x}_{j_{w}}$ and $\mathbf{x}_{k_{w}}$ represent data from the input modalities $\mathcal{M}$. In order to solve for parameters $\Theta_{local}$, we minimize the empirical measure of the categorical cross-entropy between the target local ranks ${r}_{j,k}$ and our estimated local ranks $\hat{r}_{j,k}$:

In practice, we parametrize $f_{local}$ using an LSTM (Hochreiter and Schmidhuber, 1997) which takes as input the differences of multimodal tensors:

where $\bigoplus$ denotes tensor concatenation of all unimodal feature vectors $\mathbf{x}^{m}$. $\mathbf{x}_{local}$ is the input sequence to the LSTM that performs multimodal local ranking. A neural network classification layer on the final LSTM output $\mathbf{h}_{T}$ is used to estimate local ranks $\hat{r}_{j,k}$.

In contrast to (Baltrušaitis et al., 2017) who developed a model for images only, our problem involves multimodal video segments. In our models, local comparisons of emotion intensities are performed over a time window $w$. Emotion intensities often require more than still frames, especially when including the acoustic modality. Furthermore, human communicative behaviors can be asynchronous and a longer time window is required to track changes in both visual and acoustic behaviors displayed by the person.

3. Global Ranking

The second task is the global ranking task, which uses the previous results of local rankings to infer global emotion ranks using a Bayesian skill rating algorithm (Herbrich et al., 2007). The algorithm will infer global emotion ranks $e_{t}$ at each time step $t$ of the multimodal video. These ranks are initially sampled from a prior distribution $\mathcal{N}(\mu_{t},\sigma_{t}^{2})$. The algorithm models global emotion ranks $e_{t}$ as hidden variables that are not directly observed from the data. What we do observe are local rankings ${r}_{j,k}$ between two time segments $j$ and $k$. By our definition of local ranks, the conditional probabilities of local ranks $p({r}_{j,k}=1|e_{j},e_{k})$ will be equal to $p(e_{j}>e_{k})$, the probability of rank $e_{j}$ being larger than $e_{k}$. Given estimated ranks $\hat{r}_{j,k}$, we estimate global relative emotion ranks $\hat{e}_{t}$ using a ranking algorithm which involves message passing over factor graph models. A detailed treatment is presented in (Herbrich et al., 2007; Baltrušaitis et al., 2017). After all iterations of global ranking, we obtain estimated global emotion ranks $\hat{e}_{t}$ for all video segments.

4. Direct-Relative Fusion

The third task involves direct-relative fusion. The global emotion ranks $\hat{\mathbf{e}}=\langle\hat{e}_{1},\cdots,\hat{e}_{T}\rangle$ are incorporated with the raw multimodal inputs ${\mathbf{x}^{m}}$ to estimate final emotion intensities. This allows us to perform direct estimation of the absolute emotion intensities from multimodal data while at the same time integrating relative emotion ranks from local-global rankings.

The integration of direct and relative predictions is performed by learning a function $f_{fusion}$ with parameters $\Theta_{fusion}$:

where $\hat{\mathbf{y}}$ are the predicted emotion labels. We solve for $\Theta_{fusion}$ by minimizing an empirical measure of the loss between $\hat{\mathbf{y}}$ and $\mathbf{y}$:

where $\bigoplus$ denotes tensor concatenation. Using an LSTM on $\mathbf{x}_{fusion}$ allows us to capture the temporal dependencies across multimodal time series data and label emotion intensities. A time-distributed neural network regression layer is used on the LSTM outputs $\mathbf{h}_{t}$ to estimate final emotion intensities $\hat{y}_{t}$. The global emotion ranks $\hat{\mathbf{e}}$ can be incorporated with multimodal data $\mathbf{x}^{m}$ using any multimodal fusion method. As a result, our approach represents a generalizable framework to integrate relative emotion intensities into a variety of multimodal fusion models.

Experiments

We use the AVEC16 dataset (RECOLA) (Ringeval et al., 2013) for audio-visual emotion recognition. AVEC16 consists of 9 training videos, 8 are used to optimize parameters (our training set) and 1 is held out to tune hyperparameters (our validation set). The 9 validation videos are used as our test set to compare each method (test videos are not publicly released). We use the provided appearance and geometric visual features and acoustic features. Each video has 7501 time steps after alignment between the modalities and is labeled continuously for arousal and valence at every time step. The metric used is the concordance correlation coefficient (CCC) (I-Kuei Lin, 1989).

2. Baseline Models

We compare to the following: EF-LSTM (Early Fusion LSTM) uses a single LSTM (Hochreiter and Schmidhuber, 1997) on concatenated multimodal inputs. We also implement the EF-SLSTM (stacked) (Graves et al., 2013), EF-BLSTM (bidirectional) (Schuster and Paliwal, 1997) and EF-SBLSTM (stacked bidirectional) versions and report the best result. GF-LSTM (Gated Fusion LSTM) (Yu et al., 2017) extends the EF-LSTM by assigning an LSTM to each modality and combining the final LSTM outputs with a gated attention fusion layer for final prediction. MV-LSTM (Multi-View LSTM) (Rajagopalan et al., 2016) allocates regions inside a LSTM to different modalities using parameters $\alpha$ and $\beta$. We experiment with the view-specific ($0<\alpha\leq 1,\beta=0$), coupled ($\alpha=0,0<\beta\leq 1$), hybrid ($0<\alpha<1,0<\beta<1$) and fully connected ($\alpha=1,\beta=1$) topologies as well. Our model is indicated by MLRF-$\mathbf{k}$ where $k$ is the number of local comparison pairs. We set the default window size for local ranking as $w=200$.

3. Results on Emotion Recognition

MLRF achieves better results on arousal and valence regression as compared to the baselines (Table 1). Our results show that incorporating local-global ranking estimates into simple models (EF-LSTM in our experiments) prove more effective than engineering complex neural architectures such as the Gated-LSTM (Yu et al., 2017) and the MV-LSTM (Rajagopalan et al., 2016). Furthermore, although the videos contain more than 7500 time steps, simply sampling 500-1000 local ranking pairs per video significantly improved final performance. As a result, incorporating relative emotion intensities via MLRF is an effective method without adding significant computational complexity.

4. Discussion

Effect of Number of Local Comparison Pairs: Table 1 shows that performance increases as the number of sampled local comparison pairs increases. More observations of local ranks $\hat{r}_{j,k}$ improves our estimates of global emotion ranks $\hat{e}_{t}$, which in turn provide better relative emotion intensities for emotion recognition.

Effect of Window Size: From Table 2, we observe that increasing the window size $w$ for multimodal local ranking is important. This supports the fact that human communicative behaviors are asynchronous and a longer time window is required to track changes in visual and acoustic behaviors displayed by the speaker.

Effect of Direct and Relative Approaches: We found that fusing direct emotion predictions from observed multimodal behaviors with relative emotion predictions from local-global ranking estimates was crucial (Table 3). Therefore, integrating both direct person-independent and relative person-dependent approaches is important for emotion recognition.

Conclusion

This paper approached multimodal emotion recognition from both direct person-independent and relative person-dependent perspectives. Our proposed Multimodal Local-Global Ranking Fusion (MLRF) model integrates direct and relative predictions by dividing emotion recognition into three easier subtasks: multimodal local ranking, global ranking and direct-relative fusion. Our experiments showed that MLRF displays excellent performance on multimodal tasks. Therefore, incorporating direct emotion predictions from multimodal behaviors and relative emotion ranks from local-global rankings is a promising direction for multimodal machine learning.

Acknowledgements

This material is based upon work partially supported by Samsung. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of Samsung, and no official endorsement should be inferred.

References