Efficient Low-rank Multimodal Fusion with Modality-Specific Factors

Introduction

Multimodal research has shown great progress in a variety of tasks as an emerging research field of artificial intelligence. Tasks such as speech recognition Yuhas et al. (1989), emotion recognition, De Silva et al. (1997), Chen et al. (1998), Wöllmer et al. (2013), sentiment analysis, Morency et al. (2011) as well as speaker trait analysis and media description Park et al. (2014a) have seen a great boost in performance with developments in multimodal research.

However, a core research challenge yet to be solved in this domain is multimodal fusion. The goal of fusion is to combine multiple modalities to leverage the complementarity of heterogeneous data and provide more robust predictions. In this regard, an important challenge has been on scaling up fusion to multiple modalities while maintaining reasonable model complexity. Some of the recent attempts Fukui et al. (2016), Zadeh et al. (2017) at multimodal fusion investigate the use of tensors for multimodal representation and show significant improvement in performance. Unfortunately, they are often constrained by the exponential increase of cost in computation and memory introduced by using tensor representations. This heavily restricts the applicability of these models, especially when we have more than two views of modalities in the dataset.

In this paper, we propose the Low-rank Multimodal Fusion, a method leveraging low-rank weight tensors to make multimodal fusion efficient without compromising on performance. The overall architecture is shown in Figure 1. We evaluated our approach with experiments on three multimodal tasks using public datasets and compare its performance with state-of-the-art models. We also study how different low-rank settings impact the performance of our model and show that our model performs robustly within a wide range of rank settings. Finally, we perform an analysis of the impact of our method on the number of parameters and run-time with comparison to other fusion methods. Through theoretical analysis, we show that our model can scale linearly in the number of modalities, and our experiments also show a corresponding speedup in training when compared with other tensor-based models.

The main contributions of our paper are as follows:

We propose the Low-rank Multimodal Fusion method for multimodal fusion that can scale linearly in the number of modalities.

We show that our model compares to state-of-the-art models in performance on three multimodal tasks evaluated on public datasets.

We show that our model is computationally efficient and has fewer parameters in comparison to previous tensor-based methods.

Related Work

Multimodal fusion enables us to leverage complementary information present in multimodal data, thus discovering the dependency of information on multiple modalities. Previous studies have shown that more effective fusion methods translate to better performance in models, and there’s been a wide range of fusion methods.

Early fusion is a technique that uses feature concatenation as the method of fusion of different views. Several works that use this method of fusion Poria et al. (2016) , Wang et al. (2016) use input-level feature concatenation and use the concatenated features as input, sometimes even removing the temporal dependency present in the modalities Morency et al. (2011). The drawback of this class of method is that although it achieves fusion at an early stage, intra-modal interactions are potentially suppressed, thus losing out on the context and temporal dependencies within each modality.

On the other hand, late fusion builds separate models for each modality and then integrates the outputs together using a method such as majority voting or weighted averaging Wortwein and Scherer (2017), Nojavanasghari et al. (2016). Since separate models are built for each modality, inter-modal interactions are usually not modeled effectively.

Given these shortcomings, more recent work focuses on intermediate approaches that model both intra- and inter-modal dynamics. Fukui et al. (2016) proposes to use Compact Bilinear Pooling over the outer product of visual and linguistic representations to exploit the interactions between vision and language for visual question answering. Similar to the idea of exploiting interactions, Zadeh et al. (2017) proposes Tensor Fusion Network, which computes the outer product between unimodal representations from three different modalities to compute a tensor representation. These methods exploit tensor representations to model inter-modality interactions and have shown a great success. However, such methods suffer from exponentially increasing computational complexity, as the outer product over multiple modalities results in extremely high dimensional tensor representations.

For unimodal data, the method of low-rank tensor approximation has been used in a variety of applications to implement more efficient tensor operations. Razenshteyn et al. (2016) proposes a modified weighted version of low-rank approximation, and Koch and Lubich (2010) applies the method towards temporally dependent data to obtain low-rank approximations. As for applications, Lei et al. (2014) proposes a low-rank tensor technique for dependency parsing while Wang and Ahuja (2008) uses the method of low-rank approximation applied directly on multidimensional image data (Datum-as-is representation) to enhance computer vision applications. Hu et al. (2017) proposes a low-rank tensor-based fusion framework to improve the face recognition performance using the fusion of facial attribute information. However, none of these previous work aims to apply low-rank tensor techniques for multimodal fusion.

Our Low-rank Multimodal Fusion method provides a much more efficient method to compute tensor-based multimodal representations with much fewer parameters and computational complexity. The efficiency and performance of our approach are evaluated on different downstream tasks, namely sentiment analysis, speaker-trait recognition and emotion recognition.

Low-rank Multimodal Fusion

In this section, we start by formulating the problem of multimodal fusion and introducing fusion methods based on tensor representations. Tensors are powerful in their expressiveness but do not scale well to a large number of modalities. Our proposed model decomposes the weights into low-rank factors, which reduces the number of parameters in the model. This decomposition can be performed efficiently by exploiting the parallel decomposition of low-rank weight tensor and input tensor to compute tensor-based fusion. Our method is able to scale linearly with the number of modalities.

In this paper, we formulate multimodal fusion as a multilinear function $f:V_{1}\times V_{2}\times...\times V_{M}\rightarrow H$ where $V_{1},V_{2},...,V_{M}$ are the vector spaces of input modalities and $H$ is the output vector space. Given a set of vector representations, $\{z_{m}\}_{m=1}^{M}$ which are encoding unimodal information of the $M$ different modalities, the goal of multimodal fusion is to integrate the unimodal representations into one compact multimodal representation for downstream tasks.

Tensor representation is one successful approach for multimodal fusion. It first requires a transformation of the input representations into a high-dimensional tensor and then mapping it back to a lower-dimensional output vector space. Previous works have shown that this method is more effective than simple concatenation or pooling in terms of capturing multimodal interactions Zadeh et al. (2017), Fukui et al. (2016). Tensors are usually created by taking the outer product over the input modalities. In addition, in order to be able to model the interactions between any subset of modalities using one tensor, Zadeh et al. (2017) proposed a simple extension to append $1$s to the unimodal representations before taking the outer product. The input tensor $\mathcal{Z}$ formed by the unimodal representation is computed by:

where \operatorname*{\raisebox{-1.0pt}{\scalebox{1.44}{\bigotimes}}}_{m=1}^{M} denotes the tensor outer product over a set of vectors indexed by $m$, and $z_{m}$ is the input representation with appended $1$s.

One of the main drawbacks of tensor fusion is that we have to explicitly create the high-dimensional tensor $\mathcal{Z}$. The dimensionality of $\mathcal{Z}$ will increase exponentially with the number of modalities as $\prod_{m=1}^{M}d_{m}$. The number of parameters to learn in the weight tensor $\mathcal{W}$ will also increase exponentially. This not only introduces a lot of computation but also exposes the model to risks of overfitting.

2 Low-rank Multimodal Fusion with Modality-Specific Factors

As a solution to the problems of tensor-based fusion, we propose Low-rank Multimodal Fusion (LMF). LMF parameterizes $g(\cdot)$ from Equation 2 with a set of modality-specific low-rank factors that can be used to recover a low-rank weight tensor, in contrast to the full tensor $\mathcal{W}$. Moreover, we show that by decomposing the weight into a set of low-rank factors, we can exploit the fact that the tensor $\mathcal{Z}$ actually decomposes into $\{z_{m}\}_{m=1}^{M}$, which allows us to directly compute the output $h$ without explicitly tensorizing the unimodal representations. LMF reduces the number of parameters as well as the computation complexity involved in tensorization from being exponential in $M$ to linear.

The minimal $R$ that makes the decomposition valid is called the rank of the tensor. The vector sets $\{\{w^{(i)}_{m,k}\}_{m=1}^{M}\}_{i=1}^{R}$ are called the rank $R$ decomposition factors of the original tensor.

In LMF, we start with a fixed rank $r$, and parameterize the model with $r$ decomposition factors $\{\{w^{(i)}_{m,k}\}_{m=1}^{M}\}_{i=1}^{r},k=1,...,d_{h}$ that can be used to reconstruct a low-rank version of these $\widetilde{\mathcal{W}}_{k}$.

We can regroup and concatenate these vectors into $M$ modality-specific low-rank factors. Let $\mathbf{w}_{m}^{(i)}=[w_{m,1}^{(i)},w_{m,2}^{(i)},...,w_{m,d_{h}}^{(i)}]$, then for modality $m$, $\{\mathbf{w}_{m}^{(i)}\}_{i=1}^{r}$ is its corresponding low-rank factors. And we can recover a low-rank weight tensor by:

Nevertheless, by introducing the low-rank factors, we now have to compute the reconstruction of \mathcal{W}=\sum_{i=1}^{r}\operatorname*{\raisebox{-1.0pt}{\scalebox{1.44}{\bigotimes}}}_{m=1}^{M}\mathbf{w}^{(i)}_{m} for the forward computation. Yet this introduces even more computation.

2.2 Efficient Low-rank Fusion Exploiting Parallel Decomposition

In this section, we will introduce an efficient procedure for computing $h$, exploiting the fact that tensor $\mathcal{Z}$ naturally decomposes into the original input $\{z_{m}\}_{m=1}^{M}$, which is parallel to the modality-specific low-rank factors. In fact, that is the main reason why we want to decompose the weight tensor into $M$ modality-specific factors.

Using the fact that \mathcal{Z}=\operatorname*{\raisebox{-1.0pt}{\scalebox{1.44}{\bigotimes}}}_{m=1}^{M}z_{m}, we can simplify equation 5:

where \operatorname*{\raisebox{-3.5pt}{\scalebox{1.44}{\Lambda}}}_{m=1}^{M} denotes the element-wise product over a sequence of tensors: \operatorname*{\raisebox{-3.5pt}{\scalebox{1.44}{\Lambda}}}_{t=1}^{3}x_{t}=x_{1}\circ x_{2}\circ x_{3}.

An illustration of the trimodal case of equation 3.2.2 is shown in Figure 1. We can also derive equation 3.2.2 for a bimodal case to clarify what it does:

An important aspect of this simplification is that it exploits the parallel decomposition of both $\mathcal{Z}$ and $\mathcal{W}$, so that we can compute $h$ without actually creating the tensor $\mathcal{Z}$ from the input representations $z_{m}$. In addition, different modalities are decoupled in the simplified computation of $h$, which allows for easy generalization of our approach to an arbitrary number of modalities. Adding a new modality can be simply done by adding another set of modality-specific factors and extend Equation 3.2.2. Last but not least, Equation 3.2.2 consists of fully differentiable operations, which enables the parameters $\{\mathbf{w}_{m}^{(i)}\}_{i=1}^{r}\,m=1,...,M$ to be learned end-to-end via back-propagation.

Using Equation 3.2.2, we can compute $h$ directly from input unimodal representations and their modal-specific decomposition factors, avoiding the weight-lifting of computing the large input tensor $\mathcal{Z}$ and $\mathcal{W}$, as well as the $r$ linear transformation. Instead, the input tensor and subsequent linear projection are computed implicitly together in Equation 3.2.2, and this is far more efficient than the original method described in Section 3.1. Indeed, LMF reduces the computation complexity of tensorization and fusion from $O(d_{y}\prod_{m=1}^{M}d_{m})$ to $O(d_{y}\times r\times\sum_{m=1}^{M}d_{m})$.

In practice, we use a slightly different form of Equation 3.2.2, where we concatenate the low-rank factors into $M$ order-3 tensors and swap the order in which we do the element-wise product and summation:

and now the summation is done along the first dimension of the bracketed matrix. $[\cdot]_{i,:}$ indicates the $i$-th slice of a matrix. In this way, we can parameterize the model with $M$ order-3 tensors, instead of parameterizing with sets of vectors.

Experimental Methodology

We compare LMF with previous state-of-the-art baselines, and we use the Tensor Fusion Networks (TFN) Zadeh et al. (2017) as a baseline for tensor-based approaches, which has the most similar structure with us except that it explicitly forms the large multi-dimensional tensor for fusion across different modalities.

We design our experiments to better understand the characteristics of LMF. Our goal is to answer the following four research questions:

(1) Impact of Multimodal Low-rank Fusion: Direct comparison between our proposed LMF model and the previous TFN model.

(2) Comparison with the State-of-the-art: We evaluate the performance of LMF and state-of-the-art baselines on three different tasks and datasets.

(3) Complexity Analysis: We study the modal complexity of LMF and compare it with the TFN model.

(4) Rank Settings: We explore performance of LMF with different rank settings.

The results of these experiments are presented in Section 5.

We perform our experiments on the following multimodal datasets, CMU-MOSI Zadeh et al. (2016a), POM Park et al. (2014b), and IEMOCAP Busso et al. (2008) for sentiment analysis, speaker traits recognition, and emotion recognition task, where the goal is to identify speakers emotions based on the speakers’ verbal and nonverbal behaviors.

CMU-MOSI The CMU-MOSI dataset is a collection of 93 opinion videos from YouTube movie reviews. Each video consists of multiple opinion segments and each segment is annotated with the sentiment in the range , where -3 indicates highly negative and 3 indicates highly positive.

POM The POM dataset is composed of 903 movie review videos. Each video is annotated with the following speaker traits: confident, passionate, voice pleasant, dominant, credible, vivid, expertise, entertaining, reserved, trusting, relaxed, outgoing, thorough, nervous, persuasive and humorous.

IEMOCAP The IEMOCAP dataset is a collection of 151 videos of recorded dialogues, with 2 speakers per session for a total of 302 videos across the dataset. Each segment is annotated for the presence of 9 emotions (angry, excited, fear, sad, surprised, frustrated, happy, disappointed and neutral).

To evaluate model generalization, all datasets are split into training, validation, and test sets such that the splits are speaker independent, i.e., no identical speakers from the training set are present in the test sets. Table 1 illustrates the data splits for all datasets in detail.

2 Features

Each dataset consists of three modalities, namely language, visual, and acoustic modalities. To reach the same time alignment across modalities, we perform word alignment using P2FA Yuan and Liberman (2008) which allows us to align the three modalities at the word granularity. We calculate the visual and acoustic features by taking the average of their feature values over the word time interval Chen et al. (2017).

Language We use pre-trained 300-dimensional Glove word embeddings Pennington et al. (2014) to encode a sequence of transcribed words into a sequence of word vectors.

Visual The library Facetgoo.gl/1rh1JN is used to extract a set of visual features for each frame (sampled at 30Hz) including 20 facial action units, 68 facial landmarks, head pose, gaze tracking and HOG features Zhu et al. (2006).

Acoustic We use COVAREP acoustic analysis framework Degottex et al. (2014) to extract a set of low-level acoustic features, including 12 Mel frequency cepstral coefficients (MFCCs), pitch, voiced/unvoiced segmentation, glottal source, peak slope, and maxima dispersion quotient features.

3 Model Architecture

In order to compare our fusion method with previous work, we adopt a simple and straightforward model architecture The source code of our model is available on Github at https://github.com/Justin1904/Low-rank-Multimodal-Fusion for extracting unimodal representations. Since we have three modalities for each dataset, we simply designed three unimodal sub-embedding networks, denoted as $f_{a},f_{v},f_{l}$, to extract unimodal representations $z_{a},z_{v},z_{l}$ from unimodal input features $x_{a},x_{v},x_{l}$. For acoustic and visual modality, the sub-embedding network is a simple 2-layer feed-forward neural network, and for language modality, we used an LSTM Hochreiter and Schmidhuber (1997) to extract representations. The model architecture is illustrated in Figure 1.

4 Baseline Models

We compare the performance of LMF to the following baselines and state-of-the-art models in multimodal sentiment analysis, speaker trait recognition, and emotion recognition.

Support Vector Machines Support Vector Machines (SVM) Cortes and Vapnik (1995) is a widely used non-neural classifier. This baseline is trained on the concatenated multimodal features for classification or regression task Pérez-Rosas et al. (2013), Park et al. (2014a), Zadeh et al. (2016b).

Deep Fusion The Deep Fusion model (DF) Nojavanasghari et al. (2016) trains one deep neural model for each modality and then combine the output of each modality network with a joint neural network.

Tensor Fusion Network The Tensor Fusion Network (TFN) Zadeh et al. (2017) explicitly models view-specific and cross-view dynamics by creating a multi-dimensional tensor that captures unimodal, bimodal and trimodal interactions across three modalities.

Memory Fusion Network The Memory Fusion Network (MFN) Zadeh et al. (2018a) accounts for view-specific and cross-view interactions and continuously models them through time with a special attention mechanism and summarized through time with a Multi-view Gated Memory.

Bidirectional Contextual LSTM The Bidirectional Contextual LSTM (BC-LSTM) Zadeh et al. (2017), Fukui et al. (2016) performs context-dependent fusion of multimodal data.

Multi-View LSTM The Multi-View LSTM (MV-LSTM) Rajagopalan et al. (2016) aims to capture both modality-specific and cross-modality interactions from multiple modalities by partitioning the memory cell and the gates corresponding to multiple modalities.

Multi-attention Recurrent Network The Multi-attention Recurrent Network (MARN) Zadeh et al. (2018b) explicitly models interactions between modalities through time using a neural component called the Multi-attention Block (MAB) and storing them in the hybrid memory called the Long-short Term Hybrid Memory (LSTHM).

5 Evaluation Metrics

Multiple evaluation tasks are performed during our evaluation: multi-class classification and regression. The multi-class classification task is applied to all three multimodal datasets, and the regression task is applied to the CMU-MOSI and the POM dataset. For binary classification and multiclass classification, we report F1 score and accuracy Acc$-k$ where k denotes the number of classes. Specifically, Acc$-2$ stands for the binary classification. For regression, we report Mean Absolute Error (MAE) and Pearson correlation (Corr). Higher values denote better performance for all metrics except for MAE.

Results and Discussion

In this section, we present and discuss the results from the experiments designed to study the research questions introduced in section 4.

In this experiment, we compare our model directly with the TFN model since it has the most similar structure to our model, except that TFN explicitly forms the multimodal tensor fusion. The comparison reported in the last two rows of Table 2 demonstrates that our model significantly outperforms TFN across all datasets and metrics. This competitive performance of LMF compared to TFN emphasizes the advantage of Low-rank Multimodal Fusion.

2 Comparison with the State-of-the-art

We compare our model with the baselines and state-of-the-art models for sentiment analysis, speaker traits recognition and emotion recognition. Results are shown in Table 2. LMF is able to achieve competitive and consistent results across all datasets.

On the multimodal sentiment regression task, LMF outperforms the previous state-of-the-art model on MAE and Corr. Note the multiclass accuracy is calculated by mapping the range of continuous sentiment values into a set of intervals that are used as discrete classes.

On the multimodal speaker traits Recognition task, we report the average evaluation score over 16 speaker traits and shows that our model achieves the state-of-the-art performance over all three evaluation metrics on the POM dataset.

On the multimodal emotion recognition task, our model achieves better results compared to the state-of-the-art models across all emotions on the F1 score. F1-emotion in the evaluation metrics indicates the F1 score for a certain emotion class.

3 Complexity Analysis

Theoretically, the model complexity of our fusion method is $O(d_{y}\times r\times\sum_{m=1}^{M}d_{m})$ compared to $O(d_{y}\prod_{m=1}^{M}d_{m})$ of TFN from Section 3.1. In practice, we calculate the total number of parameters used in each model, where we choose $M=3$, $d_{1}=32$, $d_{2}=32$, $d_{3}=64$, $r=4$, $d_{y}=1$. Under this hyper-parameter setting, our model contains about 1.1e6 parameters while TFN contains about 12.5e6 parameters, which is nearly 11 times more. Note that, the number of parameters above counts not only the parameters in the multimodal fusion stage but also the parameters in the subnetworks.

Furthermore, we evaluate the computational complexity of LMF by measuring the training and testing speeds between LMF and TFN. Table 3 illustrates the impact of Low-rank Multimodal Fusion on the training and testing speeds compared with TFN model. Here we set rank to be 4 since it can generally achieve fairly competent performance.

Based on these results, performing a low-rank multimodal fusion with modality-specific low-rank factors significantly reduces the amount of time needed for training and testing the model. On an NVIDIA Quadro K4200 GPU, LMF trains with an average frequency of 1134.82 IPS (data point inferences per second) while the TFN model trains at an average of 340.74 IPS.

4 Rank Settings

To evaluate the impact of different rank settings for our LMF model, we measure the change in performance on the CMU-MOSI dataset while varying the number of rank. The results are presented in Figure 4. We observed that as the rank increases, the training results become more and more unstable and that using a very low rank is enough to achieve fairly competent performance.

Conclusion

In this paper, we introduce a Low-rank Multimodal Fusion method that performs multimodal fusion with modality-specific low-rank factors. LMF scales linearly in the number of modalities. LMF achieves competitive results across different multimodal tasks. Furthermore, LMF demonstrates a significant decrease in computational complexity from exponential to linear time. In practice, LMF effectively improves the training and testing efficiency compared to TFN which performs multimodal fusion with tensor representations.

Future work on similar topics could explore the applications of using low-rank tensors for attention models over tensor representations, as they can be even more memory and computationally intensive.

Acknowledgements

This material is based upon work partially supported by the National Science Foundation (Award # 1833355) and Oculus VR. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of National Science Foundation or Oculus VR, and no official endorsement should be inferred.

References