Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning

Introduction

Multimodal sentiment analysis is an emerging field at the intersection of natural language processing, computer vision, and speech processing. Sentiment analysis aims to find the attitude of a speaker or writer towards a document, topic or an event (Pang et al. 2008). Sentiment can be expressed by the spoken words, the emotional tone of the delivery and the accompanying facial expressions. As a result, it is helpful to combine visual, language, and acoustic modalities for sentiment prediction (Perez-Rosas et al. 2013). To combine cues from different modalities, previous work mainly focused on holistic video-level feature fusion. This was done with simple features (such as bag-of-words and average smile intensity) calculated over an entire video as representations of verbal, visual and acoustic features (Wang et al. 2016; Zadeh et al. 2016b). These simplistic fusion approaches mostly ignore the structure of speech by focusing on simple statistics from videos and combining modalities at an abstract level.

The cornerstone of our approach is capturing the full structure of speech using a time-dependent recurrent approach that is able to properly perform modality fusion at every timestep. This understanding of speech structure is important due to two major reasons: 1) There are local interactions between modalities, such as how loud a word is being uttered which has roots in language and acoustic modalities or whether or not a word was accompanied by a smile which has roots in language and vision. Considering local interaction helps in dealing with commonly researched problems in natural language processing such as ambiguity, sarcasm or limited context by providing more information from the visual and acoustic modalities. Consider the word “crazy”; this word can have a positive sentiment if accompanied by a smile or can have a negative sentiment if accompanied by a frown. At the same time the word “great” accompanied by a frown implies sarcastic speech. Also, in limited context, inference of sentiment intensity is difficult. For example the word “good” accompanied by neutral nonverbal behavior could mean that the utterance is positive; but the same word accompanied with big smile could mean highly positive sentiment. 2) There are global interactions between modalities mostly established based on temporal relations between modalities. Examples include a delayed laughter due to a speaker’s utterance of words or a delayed smile because of a speech pause. Each of the modalities also have their own intramodal interactions (such as how different gestures happen over the utterance), which can be characterized as the global structure of speech.

To properly model structure of speech, two key questions need to be answered: “what modality to look for at each moment in time?” and “what moments in speech are important in the communication?”. To address the first question, a model should be able to “gate” the useful modalities at each moment in time. If a modality does not contain useful information or the modality is too noisy that negatively affects the performance of the model, the model should be able to shut off the modality and perform inference based on information present in the other modalities. To address the second question, a model should be able to divert it’s “attention” to key moments of communication such as when a polarized word is uttered or when a smile happens. In this paper we introduce the Gated Multimodal Embedding LSTM with Temporal Attention model, which explicitly accounts for these two key questions by using a gated mechanism for multimodal fusion at each time step and a Temporal Attention Layer for sentiment prediction. Our model is able to explore the structure of speech through a stateful recurrent mechanism and perform fusion at word level between different modalities. This gives our model the ability to account for local (by word level fusion of multimodal information) and global (by stateful multimodal memory) interactions between modalities.

The remainder of the paper is as follows. In Section 2, we review the related work in multimodal sentiment analysis. In Section 3, we give formal definition of our problems and present our approaches in detail. In Section 4, we describe the CMU-MOSI dataset, experimental methodology and baseline models. The results on CMU-MOSI dataset are presented in Section 5. A detailed analysis of our model’s components is provided in Section 6. Section 7 concludes the paper.

Related Work

Deep learning approaches have been became extremely popular in the past few years (Young et al. 2017). The field of multimodal machine learning specifically has gotten unprecedented momentum (Baltrušaitis et al. 2017). Multimodal models have been used for sentiment analysis (Zadeh et al. 2016b; Poria et al. 2017; Zadeh et al. 2016a); medical purposes, such as detection of suicidal risk, PTSD and depression (Scherer et al. 2016; Venek et al. 2016; Yu et al. 2013; Valstar et al. 2016); emotion recognition (Poria et al. 2017); image captioning and media description (You et al. 2016; Donahue et al. 2015); question answering (Antol et al. 2015); and multimodal translation (Specia et al. 2016). Our work is specifically connected to the following areas:

Sentiment analysis on written text modality has been well-studied (Pang et al. 2008) with models that predict sentiment from language. Early works used the bag of words or n-gram representations (Yang and Cardie 2012) to derive sentiment from individual words. Other approaches focused on opinionated words (Taboada et al. 2011; Poria et al. 2012; Poria et al. 2016), and some applied more complicated structures such as trees (Socher et al. 2013) and graphs (Poria et al. 2014). These structures aimed to derive the sentiment of sentence based on the sentiment of individual words and their compositions. More recent works used dependency-based semantic analysis (Agarwal et al. 2015), distributional representations for sentiment (Iyyer et al. 2015) and a convolutional architecture for the semantic modeling of sentences (Kalchbrenner et al. 2014). However, we are primarily working on spoken text rather than written text, which gives us the opportunity to integrate additional audio and visual modalities. These modalities are helpful by providing additional information, but sometimes may be noisy.

Multimodal sentiment analysis integrates verbal and nonverbal behavior to predict user sentiment. Though various multimodal datasets with sentiment labels exist (Wöllmer et al. 2013; Morency et al. 2011; Pérez-Rosas et al. 2013), the CMU-MOSI dataset (Zadeh et al. 2016b) is the first dataset with opinion level sentiment labels.

Recent multimodal sentiment analysis approaches focus on deep neural networks, including Convolutional Neural Networks (CNN) with multiple-kernel learning (Poria et al. 2015a) and Select-Additive Learning (SAL) CNN (Wang et al. 2016) which learns generalizable features across speakers. (Behnaz Nojavanasghari 2016) uses an unimodal deep neural network for three modalities (language, acoustic and visual) and explores the effectiveness of early fusion and late fusion. The features of the three modalities are similar to our work, but to fuse the multimodal information their work simply uses an concatenation approach at video level while we propose and justify the use of more advanced methods for multimodal fusion.

Besides sentiment, other speakers’ attributes such as persuasion, passion and confidence could also be analyzed by similar methods (Park et al. 2014; Majumder et al. 2017). (Chatterjee et al. 2015) proposes an ensemble classification approach that combines two different perspectives on classifying multimodal data: the first perspective assumes inter-modality conditional independence, while the second perspective explicitly captures the correlation between modalities is and recognized by a clustering based kernel similarity approach. These methods can also be applied to multimodal sentiment analysis.

Our gated controller for visual and acoustic modality is inspired by the controller used by Zoph and Le (Zoph and Le 2016), where they use a Recurrent Neural Network (RNN) controller to determine the structure of a CNN.

Compared to previous work, our method has two major advantages. To the best of our knowledge, our model is the first to use word level modality fusion, which means that we align each word to corresponding video frames and audio segments. Secondly, we are also the first to propose an attention layer and a input gate controller trained by reinforcement learning to approach the problem of noisy modalities.

Proposed Approaches

In this section, we give a detailed description of our proposed approach which will be divided into 2 modules: the Gated Multimodal Embedding Layer and the LSTM with Temporal Attention model. The Gated Multimodal Embedding Layer performs selective multimodal fusion at each time step (word level) using input modality gates, and the LSTM with Temporal Attention performs sentiment prediction with attention to each time step. Together, these modules combine to form the Gated Multimodal Embedding LSTM with Temporal Attention (GME-LSTM(A)). The section ends with the training details for the GME-LSTM(A).

One problem of previous models is that of noisy modalities during multimodal fusion since the textual modality may be negatively affected by the visual and audio modalities. As a result, useful textual features may be ignored because the corresponding visual or acoustic feature is noisy and important information may be lost.

Our solution is to introduce an on/off input gate controller that determines if the acoustic or visual modality at each time step should contribute to the overall prediction. The reason why we apply the input gate controller on acoustic/visual features while always letting textual features in is that the language modality is much more reliable for multimodal sentiment analysis than visual or acoustic. Also visual and acoustic modalities can be noisy or unreliable since audio visual feature extraction is done automatically using methods that add additional noise.

Mathematically, we formalize this with inputs $\mathbf{x}^{a}_{t}$ and $\mathbf{x}^{v}_{t}$ representing the audio and visual inputs at time step $t$ respectively. We have a controller $C^{a}$, with weights $\theta_{a}$, for determining the on/off of audio modalities, and $C^{v}$, with weights $\theta_{v}$, for determining the on/off of visual modalities. These controllers are implemented as deep neural networks $C^{a}(\ \cdot\ ;\theta_{a})$ and $C^{v}(\ \cdot\ ;\theta_{v})$ that take in $\mathbf{x}^{a}_{t}$ and $\mathbf{x}^{v}_{t}$ as input and outputs a binary label $c^{a}_{t}$ and $c^{v}_{t}$ (0/1) respectively. The binary output of these controllers mimics the act of accepting or rejecting a modality based on its noise level. The new inputs $\mathbf{x}^{\prime a}_{t}$ and $\mathbf{x}^{\prime v}_{t}$ to the network are:

We concatenate features from three different modalities: visual, audio and text to form the inputs $\mathbf{x}_{t}$ to the word level LSTM with Temporal Attention, described in the next section. By extracting features with alignment at the word level, we exploit the temporal correlation among different modalities $\mathbf{x}_{t}^{w},\mathbf{x}^{\prime a}_{t},\mathbf{x}^{\prime v}_{t}$, but at the same time the model is less affected by the impact of noisy modalities during multimodal fusion.

2. LSTM with Temporal Attention

The second component is a sentiment prediction model that captures the temporal interactions on the multimodal embedding layer. This component learns the global interactions between modalities for sentiment prediction. To do so, we use an LSTM with Temporal Attention (LSTM(A)). The Gated Multimodal Embedding $\mathbf{x}_{t}$ is passed as input to the LSTM at each time step $t$. A LSTM (Hochreiter and Schmidhuber 1997) with a forget gate (Gers et al. 2000) is used to learn global temporal information on multimodal input data $\mathbf{X}_{t}$:

where $\mathbf{i}$, $\mathbf{f}$ and $\mathbf{o}$ are the input, forget and output gates of the LSTM, $\mathbf{c}$ is the LSTM memory cell and $\mathbf{h}$ is the LSTM output, $\mathbf{W}$ and $\mathbf{U}$ linearly transform $\mathbf{X}_{t}$ and $\mathbf{h}_{t-1}$ respectively into the LSTM gate space and the $\odot$ operator denotes the Hadamard product (entry-wise product).

We use an attention model similar to (Wang 2016) to selectively combine temporal information from the input modalities by adaptively predicting the most important time steps towards sentiment of a video clip. We expect relevant information to sentiment to have high attention weights. For example if a person is “crying” or “laughing”, this information is relevant to the sentiment of the opinion and should have higher importance than a neutral word such as “movie”. This attention mechanism also allows the modalities to act as complimentary information. In cases where language is not helpful, the model can adaptively focus on the presence of sentiment related non-verbal behaviors such as facial gestures and tone of voice.

Mathematically, we add a soft attention layer $\mathbf{\alpha}$ on top of the sequence of LSTM hidden outputs. $\mathbf{\alpha}$ is obtained by multiplying the hidden layer at each time step $t$ with a shared vector $\mathbf{w}$ and passing the sequence through a softmax function.

The attention units $\alpha$ are used to weight the importance of each time step’s hidden layer to final sentiment prediction. Suppose $\mathbf{H}$ represents the matrix of all hidden units of the LSTM $[\mathbf{h}_{1};...;\mathbf{h}_{T}]$. Then the final sentiment prediction $\hat{y}$ is obtained by:

where function $Q$ represents a dense layer with a non-linear activation. We select Mean Absolute Error (MAE) as the loss function. Though Mean Square Error (MSE) is a more popular choice for loss function, MAE is a common criteria for sentiment analysis (Zadeh et al. 2016a).

Figure 1 shows the full structure of the GME-LSTM(A) model.

3. Training Details for GME-LSTM(A)

To train the GME-LSTM(A), we need to know how output decisions of the controller affects the performance of our LSTM(A) model. Given the weights of the gate controller and input data $\mathbf{x}^{a}_{1:T}$ and $\mathbf{x}^{v}_{1:T}$, the controller decides whether we should reject an input or not. The rejected inputs are replaced with $\mathbf{0}$, while the accepted inputs are not changed. In this way we obtain the new inputs $\mathbf{x}^{\prime a}_{1:T}$ and $\mathbf{x}^{\prime v}_{1:T}$. After we train the LSTM(A) with the new inputs $(\mathbf{x}^{w}_{1:T},\mathbf{x}^{\prime a}_{1:T},\mathbf{x}^{\prime v}_{1:T})$, we get a MAE loss, $\mathcal{L}$, on the validation set. Here $\mathcal{L}$ can be seen an indicator of how well our controller affects the performance of the model. Note that that lower MAE implies better performance, so we use $e^{-\mathcal{L}}$ as the reward signal to train the controllers.

Take the visual controller $C^{v}$ as an example: we are maximizing the expected reward, represented by $J(\theta^{v})$:

where $T$ is the total number of time steps in the dataset. The sentiment prediction MAE $\mathcal{L}$ in the reward signal is non-convex and non-differentiable with respect to the parameters of the GME since changes in the outputs of the GME change the MAE $\mathcal{L}$ in a discrete manner. Straightforward gradient descent methods will not explore all the possible regions of the function. This form of problem has been studied in reinforcement learning where policy gradient methods balance exploration and optimization by randomly sampling many possible outputs of the GME controller before optimizing for best performance. Specifically, the REINFORCE algorithm (Williams 1992) is used to iteratively update $\theta^{v}$:

An empirical approximation of the above quantity is to sample the outputs of the controller (Zoph and Le 2016):

where $n$ is the number of different inputs datasets $(\mathbf{x}^{w}_{1:T},\mathbf{x}^{\prime a}_{1:T},\mathbf{x}^{\prime v}_{1:T})$ that the controller samples, and $\mathcal{L}_{k}$ is the MAE on the validation dataset after the model is trained on $k$th inputs set.

In order to reduce variance of this estimation, we employ a baseline function $b$ (Zoph and Le 2016):

where $b$ is an exponential moving average of the previous MAEs on the validation set.

If we take the visual input gate controller as an example, the detailed training algorithm for the visual input gate controller is shown in Algorithm 1. The acoustic input gate is trained in the same manner.

Experimental Methodology

In this section, we describe the experimental methodology including the dataset, data splits for training, validation and testing, the input features and their preprocessing, the experimental details and finally review the baseline models that we compare our results to.

We test on the Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis (CMU-MOSI) dataset (Zadeh et al. 2016a), which is a collection of online videos in which a speaker is expressing his or her opinions towards a movie. Each video is split into multiple clips, and each clip contains one opinion expressed by one or more sentences. A clip has one sentiment label $y\in[-3,+3]$ which is a continuous value representing speaker’s sentiment towards a certain aspect of the movie. Figure 2 depicts a snapshot from the CMU-MOSI dataset.

The CMU-MOSI dataset consists of 93 videos / 2199 labeled clips and training is performed on the labeled clips. Each video in the CMU-MOSI dataset is from a different speaker. We use the train and test sets defined in (Wang et al. 2016) which trains on 52 videos/1284 clips (52 distinct speakers), validates on 10 videos/229 clips (10 distinct speakers) and tests on 31 videos/686 clips (31 distinct speakers). There is no speaker dependent contamination in our experiments so our model is generalizable and learns speaker-independent features.

2. Input Features

We use text, video, and audio as input modalities for our task. For text inputs, we use pre-trained word embeddings (glove.840B.300d) (Pennington et al. 2014) to convert the transcripts of videos in the CMU-MOSI dataset into word vectors. This is a 300 dimensional word embedding trained on 840 billion tokens from the common crawl dataset. For audio inputs, we use COVAREP (Degottex et al. 2014) to extract acoustic features including 12 Mel-frequency cepstral coefficients (MFCCs), pitch tracking and voiced/unvoiced segmenting features, glottal source parameters, peak slope parameters and maxima dispersion quotients. For video inputs, we use Facet (iMotions 2017) and OpenFace (Baltrušaitis et al. 2016; Zadeh et al. 2017) to extract a set of features including facial action units, facial landmarks, head pose, gaze tracking and HOG features (Zhu et al. 2006).

3. Implementation Details

Before training, we select the best 20 features from Facet and 5 from COVAREP using univariate linear regression tests. The selected Facet and COVAREP features are linearly normalized by the maximum absolute value in the training set.

For the LSTM(A) model, we set the number of hidden units of the LSTM as 64. The maximum sequence length of the LSTM, $T$, is 115. There are 50 units in the ReLU fully connected layer. The model is trained using ADAM (Kingma and Ba 2014) with learning rate 0.0005 and MAE (mean absolute error) as the loss function.

For the GME-LSTM(A) model, the visual and audio controllers are each implemented as a neural network with one hidden layer of 32 units and sigmoid activation. The number of samples $n$ generated from the controller at each training step is 5. Each sampled LSTM(A) model is trained using ADAM (Kingma and Ba 2014) with learning rate 0.0005 and MAE (mean absolute error) as the loss function. The input gate controller is then trained using ADAM (Kingma and Ba 2014) with learning rate 0.0001.

Experimental Results

We compare the performance of our methods to the following state-of-the-art multimodal sentiment analysis models:

SAL-CNN (Selective Additive Learning CNN) (Wang et al. 2016) is a multimodal sentiment analysis model that attempts to prevent identity-dependent information from being learned so as to improve generalization based only on accurate indicators of sentiment.

SVM-MD (Support Vector Machine Multimodal Dictionary) (Zadeh et al. 2016b) is a SVM trained for classification or regression on multimodal concatenated features for each video clip.

C-MKL (Convolutional Multiple Kernel Learning) (Poria et al. 2015b) is a multimodal sentiment analysis model which uses a CNN for textual feature extraction and multiple kernel learning for prediction.

RF (Random Forest) is a baseline intended for comparison to a non neural network approach.

Random is a baseline which always predicts a random sentiment intensity between $$ (Zadeh et al. 2016b). This is designed as a lower bound to compare model performance.

Human performance was recorded when humans are asked to predict the sentiment score of each opinion segment (Zadeh et al. 2016b). This acts as a future target for machine learning methods.

Since sentiment analysis based on language has been well-studied, we also compare our methods with following text-based models:

RNTN (Recursive Neural Tensor Network) (Socher et al. 2013) is a well-known sentiment analysis method that leverages the sentiment of words and their dependency structure.

DAN (Deep Average Network) (Iyyer et al. 2015) is a simple but efficient sentiment analysis model that uses information only from distributional representation of the words.

D-CNN (DynamicCNN) (Kalchbrenner et al. 2014) is among the state of the art models in text-based sentiment analysis which uses a convolutional architecture adopted for the semantic modeling of sentences.

Finally, any model with “text” appended denotes the model trained only on the textual modality of the CMU-MOSI video clips.

2. Results

In this section, we summarize the results on multimodal sentiment analysis. In Table 1, we compare our proposed approaches with previous state-of-the-art multimodal as well as language-based baseline models for sentiment analysis (described in Section 5.1).

The multimodal section of Table 1 shows the performance of our two proposed approaches compared to other multimodal baseline methods. The model we proposed, GME-LSTM(A) as well as the version without gated controller LSTM(A), both outperform multimodal and single modality sentiment analysis models. The GME-LSTM(A) model gives the best result achieved across all models, improving upon the state of the art by 4.08% in binary classification accuracy and 13.2% in MAE. Since GME-LSTM(A) is able to attend both in time, using soft attention as well as in input modality, using the Gated Multimodal Embedding Layer, it is not a surprise that this model outperforms all others.

The language section of Table 1 shows that LSTM(A) on a single modality, language, obtains slightly worse performance than some language-based methods. This is because these methods use more complicated language models such as dependency-based parse tree. However, by combining cues from audio and video with careful multimodal fusion, GME-LSTM(A) immediately outperforms all language-based and multimodal baseline models. This jump in performance shows that good temporal attention and multimodal fusion is key: our model benefit from the addition of input modalities more so than other models did.

Discussion

In this section, we analyze the usefulness of our model’s different components, demonstrating that the Temporal Attention Layer and the Gated Multimodal Embedding over input modalities are both crucial towards multimodal fusion and sentiment prediction.

Language is most important in predicting sentiment. In both the LSTM model (Table 2) and the LSTM(A) model (Table 3), using only the text modality provides a better sentiment prediction than using unimodal audio and visual modalities.

Acoustic and visual modality are noisy. When we provide additional modalities to the LSTM model without attention (Table 2), the performance does not improve significantly. Using all three modalities actually leads to slightly worse performance in F-score and MAE as compared to using fewer input modalities. This allows us to deduce that the audio and video features are probably noisy and may hurt the model’s performance if multimodal fusion is not carefully performed.

Temporal Attention improves sentiment prediction. On the other hand, when we use the the LSTM(A) model, Table 3 shows that adding more modalities improves sentiment regression and classification. The LSTM(A) (Table 3) consistently outperforms the LSTM (Table 2) across all modality combinations. We hypothesize that by using temporal attention, the model will assign the largest attention weights to time steps where all 3 modalities give strong, consistent sentiment predictions and abandon noisy frames altogether. As a result, temporal attention improves sentiment prediction despite the presence of noisy acoustic and visual modalities.

Successful cases of the LSTM(A) model. To obtain a better insight into our model’s performance, we provide some successful cases to demonstrate the contribution of the Temporal Attention Layer. By further studying the attention weights $\alpha$ in the LSTM(A), we can find which words/time steps the model focuses on. The following are examples of successful cases when we look at the textual modality alone. Each line represents a single transcript and the bold word indicates the word which the model assigns the highest attention weight to.

I thought it was fun. And she really enjoyed the film. But a lot of the footage was kind of unnecessary.

The highlighted words are all words that are good indicators of positive or negative sentiment.

The LSTM(A) model combines word meanings with audio and visual indicators. Figure 3 and Figure 4 are examples where the LSTM(A) model is successful when we use all 3 modalities. In these examples, the LSTM(A) model is able to leverage the word level alignment of audio and visual modalities to overcome the ambiguity in the corresponding aligned word. The LSTM(A) model is able to determine overall video sentiment to a greater accuracy as compared to the LSTM model without attention.

Word level fusion enables fine grained multimodal analysis. We see that the model is indeed capturing the meaning of words and implicitly classifying them based on their sentiment: positive, negative or neutral. For neutral words, the model correctly looks at the aligned visual and audio modalities to make a prediction. Therefore, the model is learning the indicators of sentiment from facial gestures and tone of voice as well. This is a benefit of word level fusion since we can examine exactly what the model is learning at a finer resolution.

2. Gated Multimodal Embedding Analysis

Gated Multimodal Embedding helps multimodal fusion. The LSTM(A) model is still susceptible to noisy modalities. Table 4 shows that the GME-LSTM(A) model outperforms the LSTM(A) model on all metrics, indicating that there is value in attending in modalities using the Gated Multimodal Embedding .

GME-LSTM(A) model correctly selects helpful modalities. To obtain a better insight into the effect of the Gated Multimodal Embedding Layer, a successful example is shown in Figure 5, where the input gate controller for the visual modality correctly identifies frames where obvious facial expressions are displayed, and rejects those with a blank expression.

GME-LSTM(A) model correctly rejects noisy modalities. We now revisit a failure case of the LSTM(A) model, where the speaker is covering her mouth during the word that gives best sentiment prediction, “cute” (Figure 6). The LSTM(A) model is focusing on an uninformative time step and makes a poor sentiment prediction. In other words, the model may be confused if the added visual and audio modalities are uninformative or noisy. We found that the Gated Multimodal Embedding correctly rejects the noisy visual input at the time step of “cute” and the GME-LSTM(A) model gives a sentiment prediction closer to the ground truth (Figure 6). This is a good example where the GME-LSTM(A) model directly tackles the problem that motivated its development: the issue of noisy modalities that hurts performance when multimodal fusion is not carefully performed. Specifically, the GME-LSTM(A) model was able to learn that the visual modality was mismatched with the textual modality, further recognizing that the visual modality was noisy while the corresponding word was a good indicator of positive speaker sentiment.

Conclusion

In this paper we proposed Gated Multimodal Embedding LSTM with Temporal Attention model for multimodal sentiment analysis. Our approach is the first of it’s kind to perform multimodal fusion at word level. Furthermore to build a model that is suitable for the complex structure of speech, we introduce selective word-level fusion between modalities using gating mechanism trained using reinforcement learning. We use attention model to divert the focus of our model to important moments in speech. The stateful nature of our model allows for long interactions to be captured between different modalities. We show state of the art performance in MOSI dataset and we bring qualitative analysis of how our model is able to deal with various challenges of understanding communication dynamics.

References