Machine Translation Evaluation Meets Community Question Answering

Introduction and Motivation

In a community Question Answering (cQA) task, we are given a question from a community forum and a thread of associated text comments intended to answer the given question; and the goal is to rank the comments according to their appropriateness to the question. Since cQA forum threads are noisy (e.g., because over time people tend to engage in discussion and to deviate from the original question), as many comments are not answers to the question, the challenge lies in learning to rank all good comments above all bad ones.

Here, we adopt the definition and the datasets from SemEval–2016 Task 3 [Nakov et al., 2016] on “Community Question Answering”, focusing on subtask A (Question-Comment Similarity) only.SemEval-2016 Task 3 had two more subtasks: subtask B on Question-Question Similarity, and subtask C on Question-External Comment Similarity, which are out of our scope. However, they could be potentially addressed within our general MTE-NN framework, with minor variations. See the task description paper and the task websitehttp://alt.qcri.org/semeval2016/task3/ for more detail. An annotated example is shown in Figure 1.

In this paper, we tackle the task from a novel perspective: by using ideas from machine translation evaluation (MTE) to decide on the quality of a comment. In particular, we extend our MTE neural network framework from [Guzmán et al., 2015], showing that it is applicable to the cQA task as well. We believe that this neural network is interesting for the cQA problem because: (i) it works in a pairwise fashion, i.e., given two translation hypotheses and a reference translation to compare to, the network decides which translation hypothesis is better, which is appropriate for a ranking problem; (ii) it allows for an easy incorporation of rich syntactic and semantic embedded representations of the input texts, and it efficiently models complex non-linear relationships between them; (iii) it uses a number of machine translation evaluation measures that have not been explored for the cQA task before, e.g., Ter [Snover et al., 2006], Meteor [Lavie and Denkowski, 2009], and Bleu [Papineni et al., 2002].

The analogy we apply to adapt the neural MTE architecture to the cQA problem is the following: given two comments $c_{1}$ and $c_{2}$ from the question thread—which play the role of the two competing translation hypotheses—we have to decide whether $c_{1}$ is a better answer than $c_{2}$ to question $q$—which plays the role of the translation reference. If we have a function $f(q,c_{1},c_{2})$ to make this decision, then we can rank the finite list of comments in the thread by comparing all possible pairs and by accumulating for each comment the scores for it given by $f$.

From a general point of view, MTE and the cQA task addressed in this paper seem similar: both reason about the similarity of two competing texts against a reference text in order to decide which one is better. However, there are some profound differences, which have implications on how each task is solved.

In MTE, the goal is to decide whether a hypothesis translation conveys the same meaning as the reference translation. In cQA, it is to determine whether the comment is an appropriate answer to the question. Furthermore, in MTE we can expect shorter texts, which are typically much more similar. In contrast, in cQA, the question and the intended answers might differ significantly both in terms of length and in lexical content. Thus, it is not clear a priori whether the MTE network can work well to address the cQA problem. Here, we show that the analogy is not only convenient, but also that using it can yield state-of-the-art results for the cQA task.

To validate our intuition, we present series of experiments using the publicly available SemEval-2016 Task 3 datasets, with focus on subtask A. We show that a naïve application of the MTE architecture and features on the cQA task already yields results that are largely above the task baselines. Furthermore, by adapting the models with in-domain data, and adding lightweight task-specific features, we are able to boost our system to reach state-of-the-art performance.

More interestingly, we analyze the contribution of several features and parts of the NN architecture by performing an ablation study. We observe that every single piece contributes important information to achieve the final performance. While task-specific features are crucial, other aspects of the framework are relevant as well: syntactic embeddings, machine translation evaluation measures, and pairwise training of the network.

The rest of the paper is organized as follows: Section 2 introduces some related work. Section 3 presents the overall architecture of our MTE-inspired NN framework for cQA. Section 4 summarizes the features we use in our experiments. Section 5 describes the experimenal settings and presents the results. Finally, Section 6 offers further discussion and presents the main conclusions.

Related Work

Recently, many neural network (NN) models have been applied to cQA tasks: e.g., question-question similarity [Zhou et al., 2015, dos Santos et al., 2015, Lei et al., 2016] and answer selection [Severyn and Moschitti, 2015, Wang and Nyberg, 2015, Shen et al., 2015, Feng et al., 2015, Tan et al., 2015]. Most of these papers concentrate on providing advanced neural architectures in order to better model the problem at hand. However, our goal here is different: we extend and reuse an existing pairwise NN framework from a different but related problem.

There is also work that uses machine translation models as a features for cQA [Berger et al., 2000, Echihabi and Marcu, 2003, Jeon et al., 2005, Soricut and Brill, 2006, Riezler et al., 2007, Li and Manandhar, 2011, Surdeanu et al., 2011, Tran et al., 2015] e.g., a variation of IBM model 1, to compute the probability that the question is a possible “translation” of the candidate answer. Unlike that work, here we port an entire MTE framework to the cQA problem. A preliminary version of this work was presented in [Guzmán et al., 2016].

Neural Model for Answer Ranking

The NN model we use for answer ranking is depicted in Figure 2. It is a direct adaptation of our feed-forward NN for MTE [Guzmán et al., 2015]. Technically, we have a binary classification task with input $(q,c_{1},c_{2})$, which should output 1 if $c_{1}$ is a better answer to $q$ than $c_{2}$, and 0 otherwise. The network computes a sigmoid function $f(q,c_{1},c_{2})=\operatorname*{sig}(\mathbf{w^{T}_{v}}\phi(q,c_{1},c_{2})+b_{v})$, where $\phi(x)$ transforms the input $x$ through the hidden layer, $\mathbf{w_{v}}$ are the weights from the hidden layer to the output layer, and $b_{v}$ is a bias term.

We first map the question and the comments to a fixed-length vector $\left[\mathbf{x}_{q},\mathbf{x}_{c_{1}},\mathbf{x}_{c_{2}}\right]$ using syntactic and semantic embeddings. Then, we feed this vector as input to the neural network, which models three types of interactions, using different groups of nodes in the hidden layer. There are two evaluation groups $\mathbf{h_{q1}}$ and $\mathbf{h_{q2}}$ that model how good each comment $c_{i}$ is to the question $q$. The input to these groups are the concatenations $\left[\mathbf{x}_{q},\mathbf{x}_{c_{1}}\right]$ and $\left[\mathbf{x}_{q},\mathbf{x}_{c_{2}}\right]$, respectively. The third group of hidden nodes $\mathbf{h_{12}}$, which we call similarity group, models how close $c_{1}$ and $c_{2}$ are. Its input is $\left[\mathbf{x}_{c_{1}},\mathbf{x}_{c_{2}}\right]$. This might be useful as highly similar comments are likely to be comparable in appropriateness, irrespective of whether they are good or bad answers in absolute terms.

In summary, the transformation $\phi(q,c_{1},c_{2})=[\mathbf{h_{q1}},\mathbf{h_{q2}},\mathbf{h_{12}}]$ can be written as

We use $\tanh$ as an activation function, rather than $\operatorname*{sig}$, to be consistent with how the word embedding vectors we use were generated.

The model further allows to incorporate external sources of information in the form of skip arcs that go directly from the input to the output, skipping the hidden layer. These arcs represent pairwise similarity feature vectors between $q$ and either $c_{1}$ or $c_{2}$. In these feature vectors, we encode MT evaluation measures (e.g., Ter, Meteor, and Bleu), cQA task-specific features, etc. See Section 4 for detail about the features implemented as skip arcs. In the figure, we indicate these pairwise external feature sets as $\psi(q,c_{1})$ and $\psi(q,c_{2})$. When including the external features, the activation at the output is $f(q,c_{1},c_{2})=\operatorname*{sig}(\mathbf{w^{T}_{v}}[\phi(q,c_{1},c_{2}),\psi(q,c_{1}),\psi(q,c_{2})]+b_{v})$.

Features

We experiment with three kinds of features: (i) input embeddings, (ii) features from MTE [Guzmán et al., 2015] and (iii) task-specific features from SemEval-2015 Task 3 [Nicosia et al., 2015].

We used two types of vector-based embeddings to encode the input texts $q$, $c_{1}$ and $c_{2}$: (1) Google_vectors: 300-dimensional embedding vectors, trained on 100 billion words from Google News [Mikolov et al., 2013]. The encoding of the full text is just the average of the word embeddings. (2) Syntax: We parse the entire question/comment using the Stanford neural parser [Socher et al., 2013], and we use the final 25-dimensional vector that is produced internally as a by-product of parsing.

Also, we compute cosine similarity features with the above vectors: $\cos(q,c_{1})$ and $\cos(q,c_{2})$.

B. MTE features

We use the following MTE metrics (MTfeats), which compare the similarity between the question and a candidate answer: (1) Bleu [Papineni et al., 2002]; (2) NIST [Doddington, 2002]; (3) TER v0.7.25 [Snover et al., 2006]. (4) Meteor v1.4 [Lavie and Denkowski, 2009] with paraphrases; (5) Unigram Precision; (6) Unigram Recall.

BLEUcomp. We further use as features various components involved in the computation of Bleu: $n$-gram precisions, $n$-gram matches, total number of $n$-grams ($n$=1,2,3,4), lengths of the hypotheses and of the reference, length ratio between them, and Bleu’s brevity penalty.

C. Task-specific features

First, we train domain-specific vectors using word2vec on all available QatarLiving data, both annotated and raw (QL_vectors).

Second, we compute various easy task-specific features (Task_features), most of them proposed for the 2015 edition of the task [Nicosia et al., 2015]. This includes some comment-specific features: (1) number of URLs/images/emails/phone numbers; (2) number of occurrences of the string “thank”;When an author thanks somebody, this post is typically a bad answer to the original question. (3) number of tokens/sentences; (4) average number of tokens; (5) type/token ratio; (6) number of nouns/verbs/adjectives/adverbs/ pronouns; (7) number of positive/negative smileys; (8) number of single/double/triple exclamation/interrogation symbols; (9) number of interrogative sentences (based on parsing); (10) number of words that are not in word2vec’s Google News vocabulary.Can detect slang, foreign language, etc., which would indicate a bad answer. Also some question-comment pair features: (1) question to comment count ratio in terms of sentences/tokens/nouns/verbs/adjectives/adverbs/pronouns; (2) question to comment count ratio of words that are not in word2vec’s Google News vocabulary. Finally, we also have two meta features: (1) is the person answering the question the one who asked it; (2) reciprocal rank of the comment in the thread.

Experiments and Results

We experiment with the data from SemEval-2016 Task 3. The task offers a higher quality training dataset train-part1, which includes 1,412 questions and 14,110 answers, and a lower-quality train-part2 with 382 questions and 3,790 answers. We train our model on train-part1 with hidden layers of size 3 for 100 epochs with minibatches of size 30, regularization of 0.005, and a decay of 0.0001, using stochastic gradient descent with adagrad [Duchi et al., 2011]; we use Theano [Bergstra et al., 2010] for learning. We normalize the input feature values to the $[-1;1]$ interval using minmax, and we initialize the network weights by sampling from a uniform distribution as in [Bengio and Glorot, 2010]. We train the model using all pairs of good vs. bad comments, in both orders, ignoring ties.

At test time, we get the full ranking by scoring all possible pairs, and we accumulate the scores at the comment level.

We evaluate the model on train-part2 after each epoch, and ultimately we keep the model that achieves the highest accuracy;We also tried Kendall’s Tau ($\tau$), but it performed worse. in case of a tie, we prefer the parameters from an earlier epoch. We selected the above parameter values on the dev dataset (244 questions and 2,440 answers) using the full model, and we used them for all experiments below, where we evaluate on the official test dataset (329 questions and 3,270 answers). We report mean average precision (MAP), which is the official evaluation measure, and also average recall (AvgRec) and mean reciprocal rank (MRR).

Table 1 shows the evaluation results for three configurations of our MTE-based cQA system. We can see that the vanilla MTE system (MTEvanilla), which only uses features from our original MTE model, i.e., it does not have any task-specific features (Task_Features and QL_vectors), performs surprisingly well despite the differences in the MTE and cQA tasks. It outperforms a random baseline (Baselinerand) and a chronological baseline that assumes that early comments are better than later ones (Baselinetime) by large margins: by about 11 and 17 MAP points absolute, respectively. For the other two measures the results are similar.

We can further see that adding the task-specific features in MTE-CQApairwise improves the results by another 8 MAP points absolute. Finally, the second line shows that adapting the network to do classification (MTE-CQAclassification), giving it a question and a single comment as input, yields a performance drop of 0.6 MAP points absolute compared to the proposed pairwise learning model. Thus, the pairwise training strategy is confirmed to be better for the ranking task, although not by a large margin.

Table 2 presents the results of an ablation study, where we analyze the contribution of various features and feature groups to the performance of the overall system. For the purpose, we study $\Delta_{\hbox{MAP}}$, i.e., the absolute drop in MAP when the feature group is excluded from the full system.

Not surprisingly, the most important turn out to be the Task_Features (contributing over five MAP points) as they handle important information sources that are not available to the system from other feature groups, e.g., the reciprocal rank alone contributes about two points.

Next in terms of importance come word embeddings, QL_vectors (contributing over 2 MAP points), trained on text from the target forum, QatarLiving. Then come the Google_vectors (contributing over one MAP point), which are trained on 100 billion words, and thus are still useful even in the presence of the domain-specific QL_vectors, which are in turn trained on four orders of magnitude less data.

Interestingly, the MTE-motivated Syntax vectors contribute half a MAP point, which shows the importance of modeling syntax for this task. The other two MTE features, MTfeats and BLEUcomp, together contribute 0.8 MAP points. It is interesting that the Bleu components manage to contribute on top of the MTfeats, which already contain several state-of-the-art MTE measures, including Bleu itself. This is probably because the other features we have do not model $n$-gram matches directly.

Finally, Table 3 puts the results in perspective. We can see that our system MTE-CQA would rank first on MRR, second on MAP, and fourth on AvgRec in SemEval-2016 Task 3 competition.The full results can be found on the task website: http://alt.qcri.org/semeval2016/task3/index.php?id=results These results are also 5 and 16 points above the average and the worst systems, respectively.

This is remarkable given the lightweight task-specific features we use, and confirms the validity of the proposed neural approach to produce state-of-the-art systems for this particular cQA task.

Conclusion

We have explored the applicability of machine translation evaluation methods to answer ranking in community Question Answering, a seemingly very different task, where the goal is to rank the comments in a question-answer thread according to their appropriateness to the question, placing all good comments above all bad ones.

In particular, we have adopted a pairwise neural network architecture, which incorporates MTE features, as well as rich syntactic and semantic embeddings of the input texts that are non-linearly combined in the hidden layer. The evaluation results on benchmark datasets have shown state-of-the-art performance, with sizeable contribution from both the MTE features and from the network architecture. This is an interesting and encouraging result, as given the difference in the tasks, it was not a-priori clear that an MTE approach would work well for cQA.

In future work, we plan to incorporate other similarity measures and better task-specific features into the model. We further want to explore the application of this architecture to other semantic similarity problems such as question-question similarity, and textual entailment.

Acknowledgments

This research was performed by the Arabic Language Technologies (ALT) group at the Qatar Computing Research Institute (QCRI), Hamad bin Khalifa University, part of Qatar Foundation. It is part of the Interactive sYstems for Answer Search (Iyas) project, which is developed in collaboration with MIT-CSAIL.

References