Exploring Interpretable LSTM Neural Networks over Multi-Variable Data

Introduction

Recently, recurrent neural networks (RNNs), especially long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) and gated recurrent units (GRU) (Cho et al., 2014), have been proven to be powerful sequence modeling tools in various tasks e.g. language modelling, machine translation, health informatics, time series, and speech (Ke et al., 2018; Lin et al., 2017; Guo et al., 2016; Lipton et al., 2015; Sutskever et al., 2014; Bahdanau et al., 2014). In this paper, we focus on RNNs over multi-variable time series consisting of target and exogenous variables. RNNs trained over such multi-variable data capture nonlinear correlation of historical values of target and exogenous variables to the future target values.

In addition to forecasting, interpretable RNNs are desirable for gaining insights into the important part of data for RNNs achieving good prediction performance (Hu et al., 2018; Foerster et al., 2017; Lipton, 2016). In this paper, we focus on two types of importance interpretation: variable importance and variable-wise temporal importance. First, in RNNs variables differ in predictive power on the target, thereby contributing differently to the prediction (Feng et al., 2018; Riemer et al., 2016). Second, variables also present different temporal relevance to the target one (Kirchgässner et al., 2012). For instance, for a variable instantaneously correlated to the target, its short historical data contributes more to the prediction. The ability to acquire this knowledge enables additional applications, e.g. variable selection.

However, current RNNs fall short of the aforementioned interpretability for multi-variable data due to their opaque hidden states. Specifically, when fed with the multi-variable observations of the target and exogenous variables, RNNs blindly blend the information of all variables into the hidden states used for prediction. It is intractable to distinguish the contribution of individual variables into the prediction through the sequence of hidden states (Zhang et al., 2017).

Meanwhile, individual variables typically present different dynamics. This information is implicitly neglected by the hidden states mixing multi-variable data, thereby potentially hindering the prediction performance.

Existing works aiming to enhance the interpretability of recurrent neural networks rarely touch the internal structure of RNNs to overcome the opacity of hidden states on multi-variable data. They still fall short of aforementioned two types of interpretation (Montavon et al., 2018; Foerster et al., 2017; Che et al., 2016). One category of the approaches is to perform post-analyzing on trained RNNs by perturbation on training data or gradient based methods (Ancona et al., 2018; Ribeiro et al., 2018; Lundberg & Lee, 2017; Shrikumar et al., 2017). Another category is to build attention mechanism on hidden states of RNNs to characterize the importance of different time steps (Qin et al., 2017; Choi et al., 2016).

In this paper we aim to achieve a unified framework of accurate forecasting and importance interpretation. In particular, the contribution is fourfold:

We explore the structure of LSTM to enable variable-wise hidden states capturing individual variable’s dynamics. It facilitates the prediction and interpretation. This family of LSTM is referred to as Interpretable Multi-Variable LSTM, i.e. IMV-LSTM.

A novel mixture attention mechanism is designed to summarize variable-wise hidden states and model the generative process of the target.

We develop a training method based on probabilistic mixture attention to learn network parameter, variable and temporal importance measures simultaneously.

Extensive experimental evaluation of IMV-LSTM against statistical, machine learning and deep learning based baselines demonstrate the superior prediction performance and interpretability of IMV-LSTM. The idea of IMV-LSTM easily applies to other RNN structures, e.g. GRU and stacked recurrent layers.

Related Work

Recent research on the interpretable RNNs can be categorized into two groups: attention methods and post-analyzing on trained models. Attention mechanism has gained tremendous popularity (Xu et al., 2018; Choi et al., 2018; Guo et al., 2018; Lai et al., 2017; Qin et al., 2017; Cinar et al., 2017; Choi et al., 2016; Vinyals et al., 2015; Bahdanau et al., 2014). However, current attention mechanism is mainly applied to hidden states across time steps. Qin et al. (2017); Choi et al. (2016) built attention on conventional hidden states of encoder networks. Since the hidden states encode information from all input variables, the derived attention is biased when used to measure the importance of corresponding variables. The contribution coefficients defined on attention values is biased as well (Choi et al., 2016). Moreover, weighting input data by attentions (Xu et al., 2018; Qin et al., 2017; Choi et al., 2016) does not consider the direction of correlation with the target, which could impair the prediction performance. Current attention based methods seldom provide variable-wise temporal interpretability.

As for post-analyzing interpretation, Murdoch et al. (2018); Murdoch & Szlam (2017); Arras et al. (2017) extracted temporal importance scores over words or phrases of individual sequences by decomposing the memory cells of trained RNNs. In perturbation-based approaches perturbed samples might be different from the original data distribution (Ribeiro et al., 2018). Gradient-based methods analyze the features that output was most sensitive to (Ancona et al., 2018; Shrikumar et al., 2017). Above methods mostly focused on one type of importance and are computationally inefficient. They rarely enhance the predicting performance.

Wang et al. (2018) focused on the importance of each middle layer to the output. Chu et al. (2018) proposed interpreting solutions for piece-wise linear neural networks. Foerster et al. (2017) introduced input-switched linear affine transformations into RNNs to analyze the contribution of input steps, wihch could lead to the loss in prediction performance. Our paper focuses on exploring the internal structure of LSTM so as to learn accurate forecasting and importance measures simultaneously.

Another line of related research is about tensorization and decomposition of hidden states in RNNs. Do et al. (2017); Novikov et al. (2015) proposed to represent hidden states as matrices. He et al. (2017) developed tensorized LSTM to enhance the capacity of networks without additional parameters. Kuchaiev & Ginsburg (2017); Neil et al. (2016); Koutnik et al. (2014) proposed to partition the hidden layer into separated modules with different updates. These hidden state tensors and update processes do not maintain variable-wise correspondence and lack the desirable interpretability.

Interpretable Multi-Variable LSTM

In the following we will first explore the internal structure of LSTM to enable hidden states to encode individual variables, such that the contribution from individual variables to the prediction can be distinguished. Then, mixture attention is designed to summarize these variable-wise hidden states for predicting. The described method can be easily extended to multi-step ahead prediction via iterative methods as well as vector regression (Fox et al., 2018; Cheng et al., 2006).

Given $\mathbf{X}_{T}$, we aim to learn a non-linear mapping to predict the next values of the target series, namely $\hat{y}_{T+1}=\mathcal{F}(\mathbf{X}_{T})$.

Meanwhile, the other desirable byproduct of learning $\mathcal{F}(\mathbf{X}_{T})$ is the variable and temporal importance measures. Mathematically, we aim to derive variable importance vector $\mathbf{I}\in R^{N}_{\geq 0}$, $\sum_{n=1}^{N}\text{I}_{n}=1$ and variable-wise temporal importance vector $\mathbf{T}^{n}\in R^{T-1}_{\geq 0}$ (w.r.t. variable $n$), $\sum_{k=1}^{T-1}\text{T}^{n}_{k}=1$. Elements of these vectors are normalized (i.e. sum to one) and reflect the relative importance of the corresponding variable or time instant w.r.t. the prediction.

The idea of IMV-LSTM is to make use of hidden state matrix and to develop associated update scheme, such that each element (e.g. row) of the hidden matrix encapsulates information exclusively from a certain variable of the input.

Next, we provide the analysis about the complexity of IMV-LSTM through Lemma 3.1 and Lemma 3.2.

Given time series of $N$ variables, assume a standard LSTM and IMV-LSTM layer both have size $D$, i.e. $D$ neurons in the layer. Then, compared to the number of parameters of the standard LSTM, IMV-Full and IMV-Tensor respectively reduce the network complexity by $(N-1)D+(1-1/N)D\cdot D$ and $4(N-1)D+4(1-1/N)D\cdot D$ number of parameters.

In a standard LSTM of layer size $D$, trainable parameters lie in the hidden and gate update functions. In total, these update functions have $4D\cdot D+4N\cdot D+4D$ parameters, where $4D\cdot D+4N\cdot D$ comes from the transition and $4D$ corresponds to the bias terms. For IMV-Full, assume each input variable corresponds to one-dimensional time series. Based on Eq. 1, the hidden update has $2D+D^{2}/N$ trainable parameters. Equation set 3.1 gives rise to the number of parameters equal to that of the standard LSTM. Therefore, the reduce number of parameters is $(N-1)D+(1-1/N)D\cdot D$. As for IMV-Tensor, more parameter reduction stems from that the gate update functions in Equation set 3.1 make use of the tensor-dot operation as Eq. 1. ∎

For time series of $N$ variables and the recurrent layer of size $D$, IMV-Full and IMV-Tensor respectively have the computation complexity at each update step as: $\mathcal{O}(D^{2}+N\cdot D)$ and $\mathcal{O}(D^{2}/N+D)$.

Assume that $D$ neurons of the recurrent layer in IMV-Full and IMV-Tensor are evenly assigned to $N$ input variables, namely each input variable has $d=D/N$ corresponding neurons. For IMV-Full, based on Eq. 1, the hidden update has computation complexity $N\cdot d^{2}+N\cdot d$, while the gate update process has the complexity $D^{2}+N\cdot D$. Overall, the computation complexity is $\mathcal{O}(D^{2}+N\cdot D)$, which is identical to the complexity of a standard LSTM. As for IMV-Tensor, since the gate update functions in Equation set 3.1 make use of the tensor-dot operation as Eq. 1, gate update functions have the same computation complexity as Eq. 1. The overall complexity is $\mathcal{O}(D^{2}/N+D)$, which is $1/N$ of the complexity of a standard LSTM. ∎

Basically, Lemma 3.1 and Lemma 3.2 indicate that a high number of input variables leads to a large portion of parameter and computation reduction in IMV-LSTM family.

2 Mixture Attention

The idea of mixture attention mechanism as follows. Temporal attention is first applied to the sequence of hidden states corresponding to each variable, so as to obtain the summarized history of each variable. Then by using the history enriched hidden state of each variable, variable attention is derived to merge variable-wise states. These two steps are assembled into a probabilistic mixture model (Zong et al., 2018; Graves, 2013; Bishop, 1994), which facilitates the subsequent learning, predicting, and interpreting.

In particular, the mixture attention is formulated as:

In Eq. (3.2), we introduce a latent random variable $z_{T+1}$ into the the density function of $y_{T+1}$ to govern the generation process. $z_{T+1}$ is a discrete variable over the set of values $\{1,\cdots,N\}$ corresponding to $N$ input variables. Mathematically, $p(y_{T+1}\,|\,z_{T+1}=n,\mathbf{h}_{T}^{n}\oplus\mathbf{g}^{n})$ characterizes the density of $y_{T+1}$ conditioned on historical data of variable $n$, while the prior of $z_{T+1}$, i.e. $\Pr(z_{T+1}=n\,|\,\mathbf{h}_{T}^{1}\oplus\mathbf{g}^{1},\cdots,\mathbf{h}_{T}^{N}\oplus\mathbf{g}^{N})$ controls to what extent $y_{T+1}$ is driven by variable $n$.

Context vector $\mathbf{g}^{n}$ is computed as the temporal attention weighted sum of hidden states of variable $n$, i.e., $\mathbf{g}^{n}=\sum_{t}\alpha_{t}^{n}\mathbf{h}_{t}^{n}$. The attention weight $\alpha_{t}^{n}$ is evaluated as $\alpha_{t}^{n}=\frac{\exp{(\,\text{f}_{n}(\mathbf{h}_{t}^{n})}\,)}{\sum_{k}\exp{(\,\text{f}_{n}(\mathbf{h}_{k}^{n})}\,)}$, where $\text{f}_{n}(\cdot)$ can be a flexible function specific to variable $n$, e.g. neural networks.

For $p(y_{T+1}\,|\,z_{T+1}=n,\mathbf{h}_{T}^{n}\oplus\mathbf{g}^{n})$, without loss of generality, we use a Gaussian output distribution parameterized by $[\,\mu_{n},\sigma_{n}]=\varphi_{n}(\,\mathbf{h}_{T}^{n}\oplus\mathbf{g}^{n})$, where $\varphi_{n}(\cdot)$ can be a feed-forward neural network. It is free to use other distributions.

$\Pr(z_{T+1}=n\,|\,\mathbf{h}_{T}^{1}\oplus\mathbf{g}^{1},\cdots,\mathbf{h}_{T}^{N}\oplus\mathbf{g}^{N})$ is derived by a softmax function over $\{\text{f}(\,\mathbf{h}_{T}^{n}\oplus\mathbf{g}^{n})\}_{N}$, where $\text{f}(\cdot)$ can be a feedforward neural network shared by all variables.

3 Learning to Interpret and Predict

In the learning phase, the set of parameters in the neural network and mixture attention is denoted by $\Theta$. Given a set of $M$ training sequences $\{\mathbf{X}_{T}\}_{M}$ and $\{y_{T+1}\}_{M}$, we aim to learn both $\Theta$ and importance vectors $\mathbf{I}$ and $\{\mathbf{T}^{n}\}_{N}$ for prediction and insights into the data.

Next, we first illustrate the burden of directly interpreting attention values and then present the training method combining parameter and importance vector learning, without the need of post analyzing.

Importance vectors $\mathbf{I}$ and $\{\mathbf{T}^{n}\}_{N}$ reflect the global relations in variables, while the attention values derived above are specific to data instances. Moreover, it is nontrivial to decipher variable and temporal importance from attentions.

For instance, during the training on PLANT dataset used in the experiment section, we collect variable and variable-wise temporal attention values of training instances. In Fig 2, left panels plots the histograms of variable attention of three variables in PLANT at two different epochs. It is difficult to fully discriminate variable importance from these histograms. Likewise, in the right panel, histograms of temporal attentions at certain time lags of variable “P-temperature” at two different epochs does not ease the importance interpretation. Time lag represents the look-back time step w.r..t the current one. Similar phenomena are observed in other variables and datasets during the experiments.

In the following, we develop the training procedure based on the Expectation–Maximization (EM) framework for the probabilistic model with latent variables, i.e. Eq. (3.2) in this paper. Index $m$ in the following corresponds to the training data instance. It is omitted in $\mathbf{h}_{T}^{n}$ and $\mathbf{g}^{n}$ for simplicity.

The loss function to minimize is derived as:

The desirable property of this loss function is as:

The negative log-likelihood defined by Eq. (3.2) is upper-bounded by the loss function Eq. (9) in the EM process:

$-\log\prod\limits_{m}p(y_{T+1,m}\,|\mathbf{X}_{T,m}\,;\Theta)\leq\mathcal{L}(\Theta,\mathbf{I})$

The proof is provided in the supplementary material.

Therefore, minimizing Eq. (9) enables to simultaneously learn the network parameters and importance vectors without the need of post processing on trained networks.

During the training phase, network parameters $\Theta$ and importance vectors are alternatively learned. In a certain round of the loss function minimization, we first fix the current value of $\Theta$ and evaluate $q_{m}^{n}$ for the batch of data. Then, since the first two terms in the loss functions solely depend on network parameters $\Theta$, they are minimized via gradient descent to update $\Theta$. For the last term, fortunately we can derive a simple closed-form solution of $\mathbf{I}$ as:

, which takes into account both variable attention and predictive likelihood in the importance vector.

As for temporal importance, it can also be derived through EM, but it requires a hierarchical mixture. For the sake of computing efficiency, the variable-wise temporal importance vector is derived from attention values as follows:

This process iterates until convergence. After the training, we obtain the neural networks ready for predicting as well as the variable and temporal importance vectors. Then, in the predicting phase, the prediction of $y_{T+1}$ is obtained by the weighted sum of means as:

Experiments

PM2.5: It contains hourly PM2.5 data and the associated meteorological data in Beijing of China. PM2.5 measurement is the target series. The exogenous time series include dew point, temperature, pressure, combined wind direction, accumulated wind speed, hours of snow, and hours of rain. Totally we have $41,700$ multi-variable sequences.

PLANT: This records the time series of energy production of a photo-voltaic power plant in Italy (Ceci et al., 2017). Exogenous data consists of 9 weather conditions variables (such as temperature, cloud coverage, etc.). The power production is the target. It provides 20842 sequences split into training ($70\%$), validation ($10\%$) and testing sets ($20\%$).

SML is a public dataset used for indoor temperature forecasting. Same as (Qin et al., 2017), the room temperature is taken as the target series and another 16 time series are exogenous series. The data were sampled every minute. The first 3200, the following 400 and the last 537 data points are respectively used for training, validation, and test.

Due to the page limitation, experimental results on additional datasets are in the supplementary material.

2 Baselines and Evaluation Setup

The first category of statistics baselines includes:

STRX is the structural time series model with exogenous variables (Scott & Varian, 2014; Radinsky et al., 2012). It is consisted of unobserved components via state space models.

ARIMAX is the auto-regressive integrated moving average with regression terms on exogenous variables (Hyndman & Athanasopoulos, 2014). It is a special case of vector auto-regression in this scenario.

The second category of machine learning baselines includes:

RF refers to random forests, an ensemble learning method consisting of several decision trees (Liaw et al., 2002) and was used in time series prediction (Patel et al., 2015).

XGT refers to the extreme gradient boosting (Chen & Guestrin, 2016). It is the application of boosting methods to regression trees (Friedman, 2001).

ENET represents Elastic-Net, which is a regularized regression method combining both L1 and L2 penalties of the lasso and ridge methods (Zou & Hastie, 2005) and used in time series analysis (Liu et al., 2010; Bai & Ng, 2008).

The third category of deep learning baselines includes:

RETAIN uses RNNs to respectively learn weights on input data for predicting (Choi et al., 2016). It defines contribution coefficients on attentions to represent feature importance.

DUAL is an encoder-decoder architecture using an encoder to learn attentions and feeding pre-weighted input data into a decoder for forecasting (Qin et al., 2017). It uses temporal-wise variable attentions to reflect variable importance.

In ARIMAX, the orders of auto-regression and moving-average terms are set as the window size of the training data. For RF and XGT, hyper-parameter tree depth and the number of iterations are chosen from range $$and$$ via grid search. For XGT, L2 regularization is added by searching within $\{0.0001,0.001,0.01,0.1,1,10\}$. As for ENET, the coefficients for L2 and L1 penalties are selected from $\{0,0.1,0.3,0.5,0.7,0.9,1,2\}$. For machine learning baselines, multi-variable input sequences are flattened into feature vectors.

We implemented IMV-LSTM and deep learning baselines with Tensorflow. We used Adam with the mini-batch size $64$ (Kingma & Ba, 2014). For the size of recurrent and dense layers in the baselines, we conduct grid search over $\{16,32,64,128,256,512\}$. The size of IMV-LSTM layers is set by the number of neurons per variable selected from $\{10,15,20,25\}$. Dropout is selected in $\{0,0.2,0.5\}$. Learning rate is searched in $\{0.0005,0.001,0.005,0.01,0.05\}$. L2 regularization is added with the coefficient chosen from $\{0.0001,0.001,0.01,0.1,1.0\}$. We train each approach $5$ times and report average performance. The window size (i.e. $T$) for PM2.5 and SML is set to 10 according to (Qin et al., 2017), while for PLANT it is 20 to test long dependency.

We consider two metrics to measure the prediction performance. RMSE is defined as $\text{RMSE}=\sqrt[]{\sum_{k}(y_{k}-\hat{y}_{k})^{2}/K}$. MAE is defined as $\text{MAE}=\sum_{k}|y_{k}-\hat{y}_{k}|/K$.

3 Prediction Performance

We report the prediction errors in Table 1, each cell of which presents the average RMSE and MAE with standard errors. In particular, IMV-LSTM family outperforms baselines by around $80\%$ at most. Deep learning baselines mostly outperform other baselines. Boosting method XGT presents comparable performance with deep learning baselines in PLANT and SML datasets.

Insights. For multi-variable data carrying different patterns, properly modeling individual variables and their interaction is important for the prediction performance. IMV-Full keeps the variable interaction in the gate updating. IMV-Tensor maintains variable-wise hidden states independently and only captures their interaction via the mixture attention. Experimentally, IMV-Full and IMV-Tensor present comparable performance, though mixture on independent variable-wise hidden states in IMV-Tensor leads to the best performance. Note that IMV-Full and IMV-Tensor are of single network structure. Instead of composite network architectures in baselines, mixture of well-maintained variable-wise hidden states in IMV-LSTM also improves the prediction performance and empowers the interpretability shown below.

4 Interpretation

In this part, we qualitatively analyze the meaningfulness of variable and temporal importance. Fig. 3 and Fig. 4 respectively show the variable and temporal importance values during the training under the best hyper-parameters. The importance values learned by IMV-Full and IMV-Tensor could be slightly different, because in IMV-Tensor the gate and memory update scheme evolve independently, thereby leading to different hidden states to IMV-Full. IMV-LSTM is easier to understand, compared to baseline RETAIN and DUAL, since they do not show in global level importance interpretation like in Fig. 3 and 4.

Variable importance. In Fig. 3, top and bottom panels in each sub-fig show the variable importance values w.r.t. training epochs from IMV-Full and IMV-Tensor. Overall, variable importance values converge during the training and the ranking of variable importance is identified at the end of the training. Variables with high importance values contribute more to the prediction of IMV-LSTM.

In Fig. 3(a), for PM2.5 dataset, variables “Wind speed”, “Pressure”, “Snow”, “Rain” are high ranked by IMV-LSTM. According to a recent work studying air pollution (Liang et al., 2015), “Dew Point” and “Pressure” are both related to PM2.5 and they are also inter-correlated. One “Pressure” variable is enough to learn accurate forecasting and thus has the high importance value. Strong wind can bring dry and fresh air. “Snow” and “Rain” amount are related to the air quality as well. Variables important for IMV-LSTM are in line with the domain knowledge in (Liang et al., 2015).

Fig. 3(b) shows that in PLANT dataset in addition to “Irradiance” and “Cloud cover”, “Wind speed”, “Humidity” as well as “Temperature” are also high ranked and relatively used more in IMV-LSTM to provide accurate forecasting. As is discussed in (Mekhilef et al., 2012; Ghazi & Ip, 2014), humidity causes dust deposition and consequentially degradation in solar cell efficiency. Increased wind can move heat from the cell surface, which leads to better efficiency.

Fig. 3(c) demonstrates that variables “Humid. room”, “CO2 room”, and “Lighting room” are relatively more important for IMV-LSTM (“Humid.” represents humidity). As is suggested in (Nguyen et al., 2014; Höppe, 1993), humidity is correlated to the indoor temperature.

Temporal importance. Fig. 4 demonstrate the temporal importance values of each variable at the ending of the training. The lighter the color, the more the corresponding data contributes to the prediction.

Specifically, in Fig. 4(a), short history of variables “Snow” and “Wind speed” contributes more to the prediction. PM2.5 itself has relatively long-term auto-correlation, i.e. around $5$ hours. Fig. 4(b) shows that recent data of aforementioned important variables “Wind” and “Temperature” are highly used for prediction, while “Cloud-cover” is long-term correlated to the target, i.e. around $13$ hours. In Fig. 4(c) temporal importance values are mostly uniform, though “Humid. dinning” has short correlation, “Outdoor temp.” and “Lighting dinning” are relatively long-term correlated to the target.

5 Variable Selection

In this group of experiments, we quantitatively evaluate the efficacy of variable importance through the lens of prediction tasks. We focus on IMV-LSTM family and RNN baselines, i.e. DUAL and RETAIN.

Specifically, for each approach, we first rank variables respectively according to the variable importance in IMV-LSTM, variable attention in DUAL and contribution coefficients in RETAIN. Meanwhile, we add one more group of baselines denoted by IMV-Full-P and IMV-Tensor-P. The label “-P” represents that the Pearson correlation is used to rank the variables with the highest (absolute) correlation values to the target and the selected data is fed to IMV-LSTM.

Then we rebuild datasets only consisting of top $50\%$ ranked variables by respective methods, retrain each model with these new datasets and obtain the errors in Table 2.

Insights. Ideally, effective variable selection enables the corresponding retrained models to have comparable errors in comparison to their counterparts trained on full data in Table 1. IMV-Full and IMV-Tensor present comparable and even lower errors in Table 2, while DUAL and RETAIN have higher errors mostly. Pearson correlation measures linear relation. Selecting variables based on it neglects non-linear correlation and is not suitable for LSTM to attain the best performance. An additional advantage of variable selection is the training efficiency, e.g. training time of each epoch in IMV-Tensor is reduced from $\sim$16 to $\sim$11 sec.

Conclusion and Discussion

In this paper, we explore the internal structures of LSTMs for interpretable prediction on multi-variable time series. Based on the hidden state matrix, we present two realizations i.e. IMV-Full and IMV-Tensor, which enable to infer and quantify variable importance and variable-wise temporal importance w.r.t. the target. Extensive experiments provide insights into achieving superior prediction performance and importance interpretation for LSTM.

Regarding high order effect, e.g. variable interaction in data, it can be captured by adding additional rows into hidden state matrices and additional elements into importance vectors accordingly. This will be the future work.

Acknowledgements

The work has been funded by the EU Horizon 2020 SoBigData project under grant agreement No. 654024.

References

Appendix

For simplicity, we ignore the data instance index $m$ in the following proof.

The log-likelihood of the target conditional on input variables is defined as:

Since $\mathbf{I}\in R^{N}_{\geq 0}$, $\sum_{n=1}^{N}\text{I}_{n}=1$, it can parameterize a categorical distribution on $z_{T+1}$.

Then introducing Eq. 15 into Eq. 14, we can obtain

During the EM process, after the E-step, $2q^{n}\log q^{n}$ will be a constant and is not involved in the optimization process. In the M-step, minimizing the negative log-likelihood amounts to minimize the loss function as follows:

2 Experiments

In this part, we provide complementary experiment results as well as the insights from the results.

NASDAQ is the dataset from (Qin et al., 2017). It contains 81 major corporations under NASDAQ 100, as exogenous time series. The index value of the NASDAQ 100 is the target series. The frequency of the data collection is minute-by-minute. The first 35,100, the following 2,730 and the last 2,730 data points are respectively used as the training, validation and test sets.

2.2 Model Interpretation

In the following Table 4, 5, and 6, we list the full ranking of variables of the datasets by each approach. Variables associated with the importance or attention values are ranked in decreasing order.

3 Discussion

In this part, we summarize the insights from the experiments.

For multi-variable data, capturing individual variable’s behaviors and their interaction is the key for both prediction and interpretation. Conventional hidden states in standard LSTMs consume the data from all input variables at each step, while our IMV-LSTM family decomposes the hidden states by defining variable data flows for each hidden state element.

In the experiments, IMV-Full and IMV-Tensor outperform baselines using the traditional hidden states. Multi-variable data potentially carries different dynamics. Conventional hidden states mix the data of all input variables, thereby failing to explicitly capture individual dynamics. In the multi-variable setting, these opaque hidden states are a burden to both prediction and interpretation.

On the contrary, IMV-Tensor models individual variables and then uses mixture attention to capture the variable interaction by variable-wise hidden states. It achieves superior prediction performance and enables the interpretability on both temporal and variable levels.

For LSTM networks on multi-variable data, importance values inherently learned by the network are more suitable for retaining useful variables for predicting.

By choosing the variables based on the learned importance value, IMV-LSTM family mostly retains the prediction performance and presents lower prediction errors on two datasets. The importance value in IMV-LSTM is derived during the training and therefore it is able to effectively identify the variables used by IMV-LSTM to minimize the loss function, i.e. maximize the prediction accuracy.

Pearson correlation variable selection leads to the quality loss in prediction performance, i.e. higher errors. Pearson correlation measures the linear correlation and pre-selecting variables based on it neglects the potential non-linear correlation in data indispensable for LSTMs to capture.