Multimodal Transformer for Unaligned Multimodal Language Sequences

Introduction

Human language possesses not only spoken words but also nonverbal behaviors from vision (facial attributes) and acoustic (tone of voice) modalities Gibson et al. (1994). This rich information provides us the benefit of understanding human behaviors and intents (Manning et al., 2014). Nevertheless, the heterogeneities across modalities often increase the difficulty of analyzing human language. For example, the receptors for audio and vision streams may vary with variable receiving frequency, and hence we may not obtain optimal mapping between them. A frowning face may relate to a pessimistically word spoken in the past. That is to say, multimodal language sequences often exhibit “unaligned” nature and require inferring long term dependencies across modalities, which raises a question on performing efficient multimodal fusion.

For evaluation, we perform a comprehensive set of experiments on three human multimodal language benchmarks: CMU-MOSI Zadeh et al. (2016), CMU-MOSEI Zadeh et al. (2018b), and IEMOCAP Busso et al. (2008). Our experiments show that MulT achieves the state-of-the-art (SOTA) results in not only the commonly evaluated word-aligned setting but also the more challenging unaligned scenario, outperforming prior approaches by a margin of 5%-15% on most of the metrics. In addition, empirical qualitative analysis further suggests that the crossmodal attention used by MulT is capable of capturing correlated signals across asynchronous modalities.

Related Works

Prior work for analyzing human multimodal language lies in the domain of inferring representations from multimodal sequences spanning language, vision, and acoustic modalities. Unlike learning multimodal representations from static domains such as image and textual attributes Ngiam et al. (2011); Srivastava and Salakhutdinov (2012), human language contains time-series and thus requires fusing time-varying signals Liang et al. (2018); Tsai et al. (2019). Earlier work used early fusion approach to concatenate input features from different modalities Lazaridou et al. (2015); Ngiam et al. (2011) and showed improved performance as compared to learning from a single modality. More recently, more advanced models were proposed to learn representations of human multimodal language. For example, Gu et al. (2018) used hierarchical attention strategies to learn multimodal representations, Wang et al. (2019) adjusted the word representations using accompanying non-verbal behaviors, Pham et al. (2019) learned robust multimodal representations using a cyclic translation objective, and Dumpala et al. (2019) explored cross-modal autoencoders for audio-visual alignment. These previous approaches relied on the assumption that multimodal language sequences are already aligned in the resolution of words and considered only short-term multimodal interactions. In contrast, our proposed method requires no alignment assumption and defines crossmodal interactions at the scale of the entire sequences.

Transformer Network.

Transformer network Vaswani et al. (2017) was first introduced for neural machine translation (NMT) tasks, where the encoder and decoder side each leverages a self-attention Parikh et al. (2016); Lin et al. (2017); Vaswani et al. (2017) transformer. After each layer of the self-attention, the encoder and decoder are connected by an additional decoder sublayer where the decoder attends to each element of the source text for each element of the target text. We refer the reader to Vaswani et al. (2017) for a more detailed explanation of the model. In addition to NMT, transformer networks have also been successfully applied to other tasks, including language modeling (Dai et al., 2018; Baevski and Auli, 2019), semantic role labeling (Strubell et al., 2018), word sense disambiguation (Tang et al., 2018), learning sentence representations (Devlin et al., 2018), and video activity recognition (Wang et al., 2018).

This paper absorbs a strong inspiration from the NMT transformer to extend to a multimodal setting. Whereas the NMT transformer focuses on unidirectional translation from source to target texts, human multimodal language time-series are neither as well-represented nor discrete as word embeddings, with sequences of each modality having vastly different frequencies. Therefore, we propose not to explicitly translate from one modality to the others (which could be extremely challenging), but to latently adapt elements across modalities via the attention. Our model (MulT) therefore has no encoder-decoder structure, but it is built up from multiple stacks of pairwise and bidirectional crossmodal attention blocks that directly attend to low-level features (while removing the self-attention). Empirically, we show that our proposed approach improves beyond standard transformer on various human multimodal language tasks.

Proposed Method

In this section, we describe our proposed Multimodal Transformer (MulT) (Figure 2) for modeling unaligned multimodal language sequences. At the high level, MulT merges multimodal time-series via a feed-forward fusion process from multiple directional pairwise crossmodal transformers. Specifically, each crossmodal transformer (introduced in Section 3.2) serves to repeatedly reinforce a target modality with the low-level features from another source modality by learning the attention across the two modalities’ features. A MulT architecture hence models all pairs of modalities with such crossmodal transformers, followed by sequence models (e.g., self-attention transformer) that predicts using the fused features.

The core of our proposed model is crossmodal attention module, which we first introduce in Section 3.1. Then, in Section 3.2 and 3.3, we present in details the various ingredients of the MulT architecture (see Figure 2) and discuss the difference between crossmodal attention and classical multimodal alignment.

Following prior works on transformers (Vaswani et al., 2017; Chen et al., 2018; Devlin et al., 2018; Dai et al., 2018), we add a residual connection to the crossmodal attention computation. Then, another positionwise feed-forward sublayer is injected to complete a crossmodal attention block (see Figure 3(b)). Each crossmodal attention block adapts directly from the low-level feature sequence (i.e., $Z_{\beta}^{}$ in Figure 3(b)) and does not rely on self-attention, which makes it different from the NMT encoder-decoder architecture (Vaswani et al., 2017; Shaw et al., 2018) (i.e., taking intermediate-level features). We argue that performing adaptation from low-level feature benefits our model to preserve the low-level information for each modality. We leave the empirical study for adapting from intermediate-level features (i.e., $Z_{\beta}^{[i-1]}$) in Ablation Study in Section 4.3.

2 Overall Architecture

To ensure that each element of the input sequences has sufficient awareness of its neighborhood elements, we pass the input sequences through a 1D temporal convolutional layer:

where $k_{\{L,V,A\}}$ are the sizes of the convolutional kernels for modalities $\{L,V,A\}$, and $d$ is a common dimension. The convolved sequences are expected to contain the local structure of the sequence, which is important since the sequences are collected at different sampling rates. Moreover, since the temporal convolutions project the features of different modalities to the same dimension $d$, the dot-products are admittable in the crossmodal attention module.

Positional Embedding.

To enable the sequences to carry temporal information, following Vaswani et al. (2017), we augment positional embedding (PE) to $\hat{X}_{\{L,V,A\}}$:

Crossmodal Transformers.

Based on the crossmodal attention blocks, we design the crossmodal transformer that enables one modality for receiving information from another modality. In the following, we use the example for passing vision ($V$) information to language ($L$), which is denoted by “$V\rightarrow L$”. We fix all the dimensions ($d_{\{\alpha,\beta,k,v\}}$) for each crossmodal attention block as $d$.

Each crossmodal transformer consists of $D$ layers of crossmodal attention blocks (see Figure 3(b)). Formally, a crossmodal transformer computes feed-forwardly for $i=1,\dots,D$ layers:

where $f_{\theta}$ is a positionwise feed-forward sublayer parametrized by $\theta$, and $\text{CM}_{V\rightarrow L}^{[i],\text{mul}}$ means a multi-head (see Vaswani et al. (2017) for more details) version of $\text{CM}_{V\rightarrow L}$ at layer $i$ (note: $d$ should be divisible by the number of heads). LN means layer normalization (Ba et al., 2016).

Self-Attention Transformers and Prediction.

3 Discussion about Attention & Alignment

When modeling unaligned multimodal language sequences, MulT relies on crossmodal attention blocks to merge signals across modalities. While the multimodal sequences were (manually) aligned to the same length in prior works before training (Zadeh et al., 2018b; Liang et al., 2018; Tsai et al., 2019; Pham et al., 2019; Wang et al., 2019), we note that MulT looks at the non-alignment issue through a completely different lens. Specifically, for MulT, the correlations between elements of multiple modalities are purely based on attention. In other words, MulT does not handle modality non-alignment by (simply) aligning them; instead, the crossmodal attention encourages the model to directly attend to elements in other modalities where strong signals or relevant information is present. As a result, MulT can capture long-range crossmodal contingencies in a way that conventional alignment could not easily reveal. Classical crossmodal alignment, on the other hand, can be expressed as a special (step diagonal) crossmodal attention matrix (i.e., monotonic attention Yu et al. (2016)). We illustrate their differences in Figure 4.

Experiments

In this section, we empirically evaluate the Multimodal Transformer (MulT) on three datasets that are frequently used to benchmark human multimodal affection recognition in prior works (Pham et al., 2019; Tsai et al., 2019; Liang et al., 2018). Our goal is to compare MulT with prior competitive approaches on both word-aligned (by word, which almost all prior works employ) and unaligned (which is more challenging, and which MulT is generically designed for) multimodal language sequences.

Each task consists of a word-aligned (processed in the same way as in prior works) and an unaligned version. For both versions, the multimodal features are extracted from the textual (GloVe word embeddings Pennington et al. (2014)), visual (Facet iMotions (2017)), and acoustic (COVAREP Degottex et al. (2014)) data modalities. A more detailed introduction to the features is included in Appendix D.

For the word-aligned version, following Zadeh et al. (2018a); Tsai et al. (2019); Pham et al. (2019), we first use P2FA Yuan and Liberman (2008) to obtain the aligned timesteps (segmented w.r.t. words) for audio and vision streams, and we then perform averaging on the audio and vision features within these time ranges. All sequences in the word-aligned case have length 50. The process remains the same across all the datasets. On the other hand, for the unaligned version, we keep the original audio and visual features as extracted, without any word-segmented alignment or manual subsampling. As a result, the lengths of each modality vary significantly, where audio and vision sequences may contain up to $>1,000$ time steps. We elaborate on the three tasks below.

CMU-MOSI Zadeh et al. (2016) is a human multimodal sentiment analysis dataset consisting of 2,199 short monologue video clips (each lasting the duration of a sentence). Acoustic and visual features of CMU-MOSI are extracted at a sampling rate of $12.5$ and $15$ Hz, respectively (while textual data are segmented per word and expressed as discrete word embeddings). Meanwhile, CMU-MOSEI Zadeh et al. (2018b) is a sentiment and emotion analysis dataset made up of 23,454 movie review video clips taken from YouTube (about $10\times$ the size of CMU-MOSI). The unaligned CMU-MOSEI sequences are extracted at a sampling rate of $20$ Hz for acoustic and $15$ Hz for vision signals.

IEMOCAP.

IEMOCAP Busso et al. (2008) consists of 10K videos for human emotion analysis. As suggested by Wang et al. (2019), 4 emotions (happy, sad, angry and neutral) were selected for emotion recognition. Unlike CMU-MOSI and CMU-MOSEI, this is a multilabel task (e.g., a person can be sad and angry simultaneously). Its multimodal streams consider fixed sampling rate on audio ($12.5$ Hz) and vision ($15$ Hz) signals. We follow Poria et al. (2017); Wang et al. (2019); Tsai et al. (2019) to report the binary classification accuracy and the F1 score of the predictions.

2 Baselines

We choose Early Fusion LSTM (EF-LSTM) and Late Fusion LSTM (LF-LSTM) as baseline models, as well as Recurrent Attended Variation Embedding Network (RAVEN) Wang et al. (2019) and Multimodal Cyclic Translation Network (MCTN) Pham et al. (2019), that achieved SOTA results on various word-aligned human multimodal language tasks. To compare the models comprehensively, we adapt the connectionist temporal classification (CTC) Graves et al. (2006) method to the prior approaches (e.g., EF-LSTM, MCTN, RAVEN) that cannot be applied directly to the unaligned setting. Specifically, these models train to optimize the CTC alignment objective and the human multimodal objective simultaneously. We leave more detailed treatment of the CTC module to Appendix B. For fair comparisons, we control the number of parameters of all models to be approximately the same. The hyperparameters are reported in Appendix C. All experiments are conducted on 1 GTX-1080Ti GPU. The code for our model and experiments can be found in https://github.com/yaohungt/Multimodal-Transformer

3 Quantitative Analysis

We first evaluate MulT on the word-aligned sequences— the “home turf” of prior approaches modeling human multimodal language Sheikh et al. (2018); Tsai et al. (2019); Pham et al. (2019); Wang et al. (2019). The upper part of the Table 1, 2, and 3 show the results of MulT and baseline approaches on the word-aligned task. With similar model sizes (around 200K parameters), MulT outperforms the other competitive approaches on different metrics on all tasks, with the exception of the “sad” class results on IEMOCAP.

Unaligned Experiments.

Next, we evaluate MulT on the same set of datasets in the unaligned setting. Note that MulT can be directly applied to unaligned multimodal stream, while the baseline models (except for LF-LSTM) require the need of additional alignment module (e.g., CTC module).

The results are shown in the bottom part of Table 1, 2, and 3. On the three benchmark datasets, MulT improves upon the prior methods (some with CTC) by 10%-15% on most attributes. Empirically, we find that MulT converges faster to better results at training when compared to other competitive approaches (see Figure 5). In addition, while we note that in general there is a performance drop on all models when we shift from the word-aligned to unaligned multimodal time-series, the impact MulT takes is much smaller than the other approaches. We hypothesize such performance drop occurs because the asynchronous (and much longer) data streams introduce more difficulty in recognizing important features and computing the appropriate attention.

Ablation Study.

To further study the influence of the individual components in MulT, we perform comprehensive ablation analysis using the unaligned version of CMU-MOSEI. The results are shown in Table 4.

First, we consider the performance for only using unimodal transformers (i.e., language, audio or vision only). We find that the language transformer outperforms the other two by a large margin. For example, for the Acc${}_{2}^{h}$ metric, the model improves from $65.6$ to $77.4$ when comparing audio only to language only unimodal transformer. This fact aligns with the observations in prior work Pham et al. (2019), where the authors found that a good language network could already achieve good performance at inference time.

Finally, we study the importance of individual crossmodal transformers according to the target modalities (i.e., using $[V,A\rightarrow L]$, $[L,A\rightarrow V]$, or $[L,V\rightarrow A]$ network). As shown in Table 4, we find crossmodal attention modules consistently improve over the late- and early-fusion transformer models in most metrics on unaligned CMU-MOSEI. In particular, among the three crossmodal transformers, the one where language($L$) is the target modality works best. We also additionally study the effect of adapting intermediate-level instead of the low-level features from source modality in crossmodal attention blocks (similar to the NMT encoder-decoder architecture but without self-attention; see Section 3.1). While MulT leveraging intermediate-level features still outperform models in other ablative settings, we empirically find adapting from low-level features works best. The ablations suggest that crossmodal attention concretely benefits MulT with better representation learning.

4 Qualitative Analysis

To understand how crossmodal attention works while modeling unaligned multimodal data, we empirically inspect what kind of signals MulT picks up by visualizing the attention activations. Figure 6 shows an example of a section of the crossmodal attention matrix on layer 3 of the $V\rightarrow L$ network of MulT (the original matrix has dimension $T_{L}\times T_{V}$; the figure shows the attention corresponding to approximately a 6-sec short window of that matrix). We find that crossmodal attention has learned to attend to meaningful signals across the two modalities. For example, stronger attention is given to the intersection of words that tend to suggest emotions (e.g., “movie”, “disappointing”) and drastic facial expression changes in the video (start and end of the above vision sequence). This observation advocates one of the aforementioned advantage of MulT over conventional alignment (see Section 3.3): crossmodal attention enables MulT to directly capture potentially long-range signals, including those off-diagonals on the attention matrix.

Discussion

In the paper, we propose Multimodal Transformer (MulT) for analyzing human multimodal language. At the heart of MulT is the crossmodal attention mechanism, which provides a latent crossmodal adaptation that fuses multimodal information by directly attending to low-level features in other modalities. Whereas prior approaches focused primarily on the aligned multimodal streams, MulT serves as a strong baseline capable of capturing long-range contingencies, regardless of the alignment assumption. Empirically, we show that MulT exhibits the best performance when compared to prior methods.

We believe the results of MulT on unaligned human multimodal language sequences suggest many exciting possibilities for its future applications (e.g., Visual Question Answering tasks, where the input signals is a mixture of static and time-evolving signals). We hope the emergence of MulT could encourage further explorations on tasks where alignment used to be considered necessary, but where crossmodal attention might be an equally (if not more) competitive alternative.

Acknowledgements

This work was supported in part by DARPA HR00111990016, AFRL FA8750-18-C-0014, NSF IIS1763562 #1750439 #1722822, Apple, Google focused award, and Samsung. We would also like to acknowledge NVIDIA’s GPU support.

References

Appendix A Positional Embedding

A purely attention-based transformer network is order-invariant. In other words, permuting the order of an input sequence does not change transformer’s behavior or alter its output. One solution to address this weakness is by embedding the positional information into the hidden units (Vaswani et al., 2017).

for $i=1,\dots,T$ and $j=0,\lfloor\frac{d}{2}\rfloor$. Therefore, each feature dimension (i.e., column) of PE are positional values that exhibit a sinusoidal pattern. Once computed, the positional embedding is added directly to the sequence so that $X+\text{PE}$ encodes the elements’ position information at every time step.

Appendix B Connectionist Temporal Classification

Connectionist Temporal Classification (CTC) (Graves et al., 2006) was first proposed for unsupervised Speech to Text alignment. Particularly, CTC is often combined with the output of recurrent neural network, which enables the model to train end-to-end and simultaneously infer speech-text alignment without supervision. For the ease of explanation, suppose the CTC module now are aiming at aligning an audio signal sequence [$a_{1},a_{2},a_{3},a_{4},a_{5},a_{6}$] with length $6$ to a textual sequence “I am really really happy” with length $5$. In this example, we refer to audio as the source and texts as target signal, noting that the sequence lengths may be different between the source to target; we also see that the output sequence may have repetitive element (i.e., “really”). The CTC Graves et al. (2006) module we use comprises two components: alignment predictor and the CTC loss.

First, the alignment predictor is often chosen as a recurrent networks such as LSTM, which performs on the source sequence then outputs the possibility of being the unique words in the target sequence as well as a empty word (i.e., x). In our example, for each individual audio signal, the alignment predictor provides a vector of length $5$ regarding the probability being aligned to [x, ‘I’, ‘am’, ‘really’, ‘happy’].

Next, the CTC loss considers the negative log-likelihood loss from only the proper alignment for the alignment predictor outputs. The proper alignment, in our example, can be results such as

[x, ‘I’, ‘am’, ‘really’, ‘really’, ‘happy’];

[‘I’, ‘am’, x, ‘really’, ‘really’, ‘happy’];

[‘I’, ‘am’, ‘really’, ‘really’, ‘really’, ‘happy’];

[‘I’, ‘I’, ‘am’, ‘really’, ‘really’, ‘happy’]

In the meantime, some examples of the suboptimal/failure cases would be

[x, x, ‘am’, ‘really’, ‘really’, ‘happy’];

[‘I’, ‘am’, ‘I’, ‘really’, ‘really’, ‘happy’];

When the CTC loss is minimized, it implies the source signals are properly aligned to target signals.

To sum up, in the experiments that adopting the CTC module, we train the alignment predictor while minimizing the CTC loss. Then, excluding the probability of blank words, we multiply the probability outputs from the alignment predictor to source signals. The source signal is hence resulting in a pseudo-aligned target singal. In our example, the audio signal is then transforming to a audio signal [$a_{1}^{\prime},a_{2}^{\prime},a_{3}^{\prime},a_{4}^{\prime},a_{5}^{\prime}$] with sequence length $5$, which is pseudo-aligned to [’I’, ’am’, ’really’, ’really’, ’happy’].

Appendix C Hyperparameters

Table 5 shows the settings of the various MulTs that we train on human multimodal language tasks. As previously mentioned, the models are contained at roughly the same sizes as in prior works for the purpose of fair comparison. For hyperparameters such as the dropout rate and number of heads in crossmodal attention module, we perform a basic grid search. We decay the learning rate by a factor of 10 when the validation performance plateaus.

Appendix D Features

The features for multimodal datasets are extracted as follows:

Language. We convert video transcripts into pre-trained Glove word embeddings (glove.840B.300d) (Pennington et al., 2014). The embedding is a $300$ dimensional vector.

Vision. We use Facet (iMotions, 2017) to indicate $35$ facial action units, which records facial muscle movement (Ekman et al., 1980; Ekman, 1992) for representing per-frame basic and advanced emotions.

Audio. We use COVAREP (Degottex et al., 2014) for extracting low level acoustic features. The feature includes 12 Mel-frequency cepstral coefficients (MFCCs), pitch tracking and voiced/unvoiced segmenting features, glottal source parameters, peak slope parameters and maxima dispersion quotients. Dimension of the feature is $74$.