Explaining Question Answering Models through Text Generation

Introduction

Language Models (LMs), trained on large amounts of data using self-supervised learning Peters et al. 2018; Devlin et al. 2018; Liu et al. 2019; Yang et al. 2019; Raffel et al. 2019, have been recently shown to encode substantial amounts of knowledge in their parameters Petroni et al. 2019; Jiang et al. 2019; Talmor et al. 2019. This has been demonstrated by their ability to answer questions that require common sense and world knowledge, without retrieving information from an external source Trinh and Le 2018; Zhou et al. 2019; Roberts et al. 2020; Ling et al. 2020. For example, the current top model for CommonsenseQA (CSQA) Talmor et al. 2018, https://www.tau-nlp.org/csqa-leaderboard a benchmark testing the ability to answer commonsense questions, answers questions using ALBERT Lan et al. 2019 only.

Despite these impressive results, most current models are based on end-to-end architectures, where it is difficult to know how the model reached its prediction. In fact, it has been repeatedly shown that often models obtain high performance through “shortcuts” rather than language understanding Tsuchiya 2018; Poliak et al. 2018; Gururangan et al. 2018; Geva et al. 2019. This has sparked interest in explainable models, where intermediate parts of the model can be inspected and interpreted by humans Lipton 2016; Ribeiro et al. 2016; Lundberg and Lee 2017; Camburu et al. 2018; Thorne et al. 2019; Rajani et al. 2019; Jain and Wallace 2019.

In this paper, we investigate explainable LM-based models for multi-choice question answering (MC-QA). Specifically, we address the question: What is the knowledge in the LM used for answering a question? Our approach consists of a generator and a classifier (see Figure 1). The generator takes a question and outputs a textual hypothesis, which consists of a few words in natural language. The classifier takes the hypothesis (and possibly the question) to make the final prediction. Unlike end-to-end models where reasoning is internal to the model, the hypothesis is an inspectable intermediate representation that exposes relevant knowledge extracted from the LM. To the best of our knoweldge, we are the first to propose this intermediate textual layer for as an explanation for a QA model.

Our setup can be viewed as an instance of controlled text generation. However, generation is not controlled by additional inputs Kikuchi et al. 2016; Ficler and Goldberg 2017; Keskar et al. 2019; Dathathri et al. 2019. Instead, we train from weak supervision – the generator is trained to produce hypotheses that are useful for the downstream QA application. We compare this approach to existing methods for generating explanations, where the explanation is provided as a target and the model is trained to generate it explicitly Camburu et al. 2018; Rajani et al. 2019.

Training from weak supervision raises several technical challenges. First, because the generator outputs discrete symbols, the loss is non-differentiable with respect to its parameters. We use the Gumbel-Softmax straight-through estimator Jang et al. 2016; Maddison et al. 2016 to overcome this difficulty. Second, the classifier can choose to ignore the generated hypothesis, or it can coordinate with the generator to change the original meaning of words. In such case, the hypothesis will not constitute a useful explanation. To encourage meaningful hypotheses, we (a) train the classifier along with an auxiliary similarity classifier that must use the hypothesis, (b) add loss terms that encourage the hypothesis to correspond to natural language, and (c) feed the classifier with multiple generator outputs.

We evaluate our approach in multiple setups. First, in a synthetic setup, where the model learns to output the hypernym of objects; Second, on CSQA, which focuses on commonsense knowledge; and third, on zero-shot transfer to QASC, a multi-choice QA task with an emphasis on scientific knowledge. We find that our approach reaches performance that is comparable to end-to-end models, while providing good hypotheses that are shown to be used by the classifier for prediction. We analyze the generated hypotheses and demonstrate that they shed light on different reasons for model error in different questions (e.g., missing world knowledge vs. language understanding difficulties), and help detect examples where the prediction of the model does not reflect its true knowledge. Our approach can be generalized outside of MC-QA to any scenario where we want to control the text generated from a LM using a signal from a downstream application. Our code and data is available at https://github.com/nika2312/qa_explaination.

Background

We consider the task of multi-choice question answering, where given a question $q$ and a list of candidate answers $\mathcal{A}=(a_{1},\dots,a_{n})$, our goal is to choose the correct answer $a^{*}\in A$. At training time, we observe question-candidates-answer triples $\{(q_{i},\mathcal{A}_{i},a^{*}_{i})\}_{i=1}^{N}$, from which we train a model.

A standard end-to-end approach for MC-QA is to first obtain a contextualized representation for the question and each answer candidate $a_{i}$, by passing it through a transformer-based pre-trained LM Vaswani et al. 2017: $h^{i}=\text{LM}(q,a_{i})$. Then, the contextualized representations are summarized to a single vector $g_{i}$. For example, in BERT Devlin et al. 2018 the summary vector is $g_{i}=h^{i}_{\texttt{CLS}}$, where $h^{i}_{\texttt{CLS}}$ is the representation of the special CLS token. Last, a weight vector $w$ is learned to compute the score $s_{i}=w^{\top}g_{i}$. The final output distribution is $\text{softmax}(s_{1},\dots,s_{n})$, and the entire model is trained to maximize the log-likelihood of $a^{*}$.

This simple setup has been successful in multiple MC-QA tasks that require commonsense reasoning Sap et al. 2018; Zellers et al. 2018; Chen et al. 2019b; Talmor et al. 2018; Wang et al. 2019. However, in end-to-end models, all reasoning is internal to the model, and it is difficult to know what knowledge inside the LM was used for prediction.

In this work, we develop a model that exposes more directly the knowledge inside the LM that is used, by having it generate this information in the form of language (Figure 1). Our goal is not to improve performance, as we assume backpropagation over a differentiable model is an effective method for distilling information from a pre-trained LM. Instead, we aim for explainability, aiming to help both users and practitioners understand what the model is doing. Recent work has shown Petroni et al. 2019; Jiang et al. 2019; Talmor et al. 2019 that it is possible to extract information encoded inside a LM by directly training it to output certain tokens in certain positions. Here, we make a weaker assumption that we do not know what words the model should generate, only that the text should be useful for the downstream application.

Model

At a high-level our model consists of two components, the generator and the classifier. The generator takes a question $q$ as input, and outputs a hypothesis, which is a sequence of tokens $c=(c^{1},\dots,c^{|c|})$. The classifier takes the question, generated hypothesis, and answer candidates $\mathcal{A}$, and predicts the answer.

We assume the generator is based on a large pre-trained autoregressive LM (encoding world knowledge in its parameters). The LM provides a distribution over the vocabulary $p_{\text{gen}}(x_{i}\mid x_{1\dots i-1})$, where $x_{1\dots i-1}=(x_{1},\dots,x_{i-1})$ is a sequence of observed tokens. Thus, the hypothesis $c$ is generated by decoding the hypothesis left-to-right, concatenating the decoded prefix to the question $p_{\text{gen}}(c\mid q)=\prod_{i=1}^{|c|}p_{\text{gen}}(c_{i}\mid[q;c_{1\dots i-1}])$. In this work, we utilize two well-known LMs, GPT-2 Radford et al. 2019 and XLNet Yang et al. 2019 (see §4).

The classifier is very similar to the end-to-end architecture described in §2. It takes the question $q$, the hypothesis $c$, and an answer candidate $a_{i}$ as input, and outputs a score $s_{i}$. The final distribution is $p_{\text{QA}}(a\mid q,c)=\text{softmax}(s_{1},\dots,s_{n})$. We discuss possible forms for the classifier and the effect on performance and explainability in §3.2.

Our framework raises a few technical challenges. First, because we sample discrete symbols from the generator, the log-likelihood of the correct answer $a^{*}$ is not differentiable with respect to the parameters of the generator $p_{\text{gen}}$. While this indeed makes optimization challenging, we overcome this by using the straight-through Gumbel-softmax estimator Jang et al. 2016; Maddison et al. 2016, which is a standard approach in this setup (§3.1). Second, if the classifier is strong enough, for example, another instance of a large pre-trained LM, it can easily answer the question directly, completely ignoring the hypothesis $c$ generated by the generator. To overcome this, we train a much simpler similarity classifier (possibly jointly with a more complex classifier), which provides incentive for the generator to produce useful hypothesis (§3.2). Last, how do we make sure that the hypothesis is a useful explanation for humans? We experiment with both loss functions and decoding mechanisms that improve the quality of the hypothesis (§3.3).

Our goal is to obtain a good MC-QA model, and thus we maximize the expected log-likelihood of the correct answer:

Computing the expectation exactly is intractable, and so we approximate it by sampling $\hat{c}$.

Training the classifier is trivial, since the loss is differentiable with respect to its parameters. However, the loss is not differentiable with respect to the parameters of $p_{\text{gen}}$. To overcome this difficulty, we use the straight-through (ST) Gumbel-softmax (GS) estimator, which has been shown to produce better gradient estimates compared to REINFORCE Williams 1992. The Gumbel-softmax trick provides a continuous relaxation for the categorical distribution over the vocabulary, making the model fully differentiable. However, it results in a mismatch between training and test time, as at test time we want discrete hypotheses. The straight-through estimator solves that by using argmax in the forward pass and softmax in the backward pass. For details on the GS-ST estimator, please see Bengio et al. 2013; Jang et al. 2016; Maddison et al. 2016; Yin et al. 2019.

We note that because the loss is backpropagated through the inputs of the classifier to the outputs of the generator, the vocabularies of the generator and classifier must be tied. In practice, we use XLNet (or GPT-2) for both generation and classification.

2 Classifier Expressivity

We would like to have hypotheses that reflect knowledge in the LM generator that is useful for the MC-QA task. However, if the classifier has the same knowledge as the generator (for example, if they are initialized with the same parameters), it has no incentive to use the hypothesis at all, as it can extract the same knowledge from its own weights. Moreover, even if the classifier is weaker than the generator, once it has enough capacity, it can coordinate with the generator in arbitrary ways, and make the tokens decoded by the generator lose their original meaning. Indeed, in §4 we show that in such cases hypotheses are meaningless.

In this work, we consider a simple similarity classifier $p_{\text{QA}}^{\text{sim}}$ whose only parameters are word embeddings (excluding the generator parameters). Specifically, given the sequence of word embeddings for the context $E_{c}=(e_{c}^{1},\dots,e_{c}^{|c|})$ and the sequence of word embeddings for an answer candidate $E_{a_{i}}=(e_{a}^{1},\dots,e_{a}^{|a_{i}|})$, we define the score for the candidate answer as:

This model does not utilize the question at all and hence must use the hypothesis $c$ to answer the question. This pushes the generator towards generating interpretable hypotheses that are more similar to the true answer compared to the distractors.

The similarity classifier $p_{\text{QA}}^{\text{sim}}$ encourages the generator to produce meaningful hypotheses, but this can come at a great cost to performance. Thus, we propose to train $p_{\text{QA}}^{\text{sim}}$ jointly with a more expressive LM-based classifier $p_{\text{QA}}^{\text{LM}}$. Specifically, this classifier is another large pre-trained LM almost identical to the end-to-end model described in §2. The only differences are that (a) it takes as input the concatenation of the question, answer candidate, and the generated hypothesis, (b) the summary vector $g_{i}$ encoding the input is the last hidden state of the contextualized representation. The classifiers share word embeddings and receive the same generator’s output. The modified objective is hence:

We empirically show in §4 through ablation tests that this LM-based classifier indeed uses the hypotheses $c$ generated by the generator, even when it is given the question as input.

Last, we note that $p_{\text{QA}}^{\text{sim}}$ pushes the generator to produce hypotheses that are similar to the answer. In other setups, we might want the generator to produce hypotheses that complement a different source of information. For example, if the question is accompanied by a paragraph that contains relevant information. In this case, the classifier can be a reading comprehension (RC) model whose parameters are untrained, and the generator will be pushed to produce hypotheses that allow the RC model to answer the question.

3 Explainability

The similarity classifier encourages the generator to generate meaningful hypotheses. However, there is no guarantee that the meaning of words does not change during training. Moreover, the similarity classifier objective might lead to repetitions in the generated text. We now describe variants aimed at improving interpretability.

We add a KLD loss regularization term to the generator objective Jaques et al. 2016: given the original pre-trained LM $p_{\text{NL}}$, we minimize at each decoding step the KL divergence $D(p_{\text{gen}}||p_{\text{NL}})$ between the trained generator distribution and the original pre-trained LM distribution. This prevents drift of the generator distribution.

We adopt the unlikelihood objective from Welleck et al. 2019 to discourage repetitions in the generated text. For each decoded token $c_{i}$, we add to the loss function the term $\sum_{w\in\mathcal{W}^{t}}{\log(1-p_{\text{gen}}(w\mid[q;c_{1...i-1}))}$, where $\mathcal{W}^{t}$ is a set of “negative” tokens we want to penalize, that is, those that already appeared in $c_{1...i-1}$.

Sampling a single hypothesis $\hat{c}$ provides a narrow channel for the generator to pass information about its distribution to the classifier. To provide more information, we modify the ST estimator forward pass, and output the top-$K$ tokens before passing the hypotheses to the classifier. This produces a set of similar words that are guaranteed to be distinct from one another. To avoid the computational burden of running the classifier over a beam of hypotheses, we perform just one step of decoding, and concatenate the top-$K$ tokens with highest probability (thus, $|c|=K$).

4 Supervised Generator

Several recent works learned to generate answers or explanations McCann et al. 2018; Camburu et al. 2018; Rajani et al. 2019; Huang et al. 2019 using the standard supervised sequence-to-sequence objective Sutskever et al. 2014. Specifically, they encode a source sequence and maximize the log-likelihood of the target explanation (or answer) token-by-token using cross-entropy loss over the vocabulary. We adapt this approach to our setup, and encode the question $q$ as usual, but use the correct answer $a^{*}$ as the target. Here, the model obtains supervision in every decoding step, unlike weak supervision where the loss is obtained once for the entire hypothesis. We refer to this model variant in §4 as SupGen. Training SupGen jointly with the LM-based classifier resulted in poor performance, due to optimization issues.

Experiments

We evaluate on the synthetic task of hypernym extraction, on CSQA, and on transfer to QASC.

We investigate whether our model can extract hypernyms, which have been shown to be encoded well in LMs Richardson and Sabharwal 2019.

We automatically create a dataset of 7,625 questions of the form “What is a [hyponym]?”, where [hyponym] is a slot filled by words, such as “dog”, and the possible answers are six disjoint hypernym categories: (a) “plant”, (b) “bird”, (c) “fish”, (d) “mammal”, (e) “reptile”, and (f) “bacteria”. Hyponym-hypernym pairs are harvested from ConceptNet Speer et al. 2016. We create a development set by randomly sampling 20% of the data, and use the rest for training.

2 MC-QA Experiments

We evaluate our models on CSQA Talmor et al. 2018, a multi-choice commonsense reasoning QA benchmark consisting of 12,102 questions with 5-choice answers, and on zero-shot transfer to QASC Khot et al. 2019, a dataset containing science questions that require external knowledge, consisting of 9,980 examples with 8-choice answers (Figure 2). QASC contains a corpus of scientific facts for information retrieval (IR) purposes, but we do not use it in our setup.

All models are trained for 20 epochs, using BertAdam optimizer with a learning rate of $2e^{-5}$, batch size of 8, and dropout of 0.1. Training was done using AllenNLP Gardner et al. 2018, on top of models from Hugging Face Wolf et al. 2019.

We start by examining the performance and explainability of models trained using a XLNet-based similarity classifier only (Eq. 1), and examine the effect of the variants from §3.

The aforementioned End2End baseline (§4.1) obtains an accuracy of 71.0 on the development set of CSQA. However, it has two advantages compared to our approach that we wish to disentangle: (a) it is fully differentiable (this is what we wish to isolate), but also (b) its architecture is different: it is given both the question and answer candidate as inputs to a transformer. Thus, it can model arbitrary interactions between them at the token level. This is in contrast to our model, where the generator only observes the question, and generates a hypothesis that is used to deterministically select the answer. Thus, we introduce the NoInteraction baseline, which is fully differentiable, but has limited interaction between the question and answer candidates.

In NoInteraction, we feed only the question $q$ as input to the pre-trained LM, and represent it with the last contextualized representation $g_{\text{final}}$. Then, the score for an answer candidate is computed just like in the similarity classifier: $s_{i}=\text{avg}(g_{\text{final}}^{\top}E_{a_{i}})$. This results in an end-to-end model without a hypothesis, where the question and answer candidates have limited interaction. NoInteraction obtains 63.7 accuracy on the development set, showing that with this limited interaction performance drops by 7-8 points. We view this performance as an upper bound on the accuracy of the similarity classifier.

Table 1 shows the performance of different generator models, where we compare different hypothesis lengths ($|c|$), use both the KLD and repetition (REP) loss functions, and the ST estimator with Top-$K$ sampling. Table 4 presents examples of hypotheses generated by the different generators that highlights the differences in explainability between them.

Our best model is Top-$K=3$ ST that reaches 58.0 accuracy, and by design outputs three different words in one decoding step. This is 5.7 points lower than NoInteraction, and we attribute the lower performance to the narrow channel between the generator and classifier (words instead of vectors), and to the difficult optimization.

Models with multiple decoding steps tend to produce repeated words, due to the nature of the similarity classifier, which encourages generating words that are similar to the correct answer. Adding the KLD and especially the repetition loss reduces repetitions dramatically, but also decreases accuracy. Table 4 shows that the generated text is reasonable, even when different from the gold answer (we evaluate and analyze the quality of hypotheses more explicitly below). Surprisingly, KLD loss did not yield more coherent phrases, perhaps due to the known tendency of LMs to generate repetitions Holtzman et al. 2019; Welleck et al. 2019.

We evaluated the supervised generator, SupGen, outputting exactly 3 tokens and then applying the similarity classifier. SupGen produces text that is more natural than our models, as evidenced by the examples in Table 6. However, because it is not optimized for QA, accuracy drops by 7.2 points.

2.2 LM-based classifier experiments

We now examine the performance and explainability of joint training of the LM-based classifier and the similarity classifier (Eq. 2). To show the importance of joint training, we evaluate NoSimClassifier, where the model is trained without the similarity classifier. When training jointly, we start with the similarity classifier only, as a warm-up for the generator, and then add the LM-based classifier.

Table 2 shows the results. The Q+C column shows results when the classifier is trained and tested using the question and hypothesis. The Q column shows results when training with the question and hypothesis as usual, but at test time only the question is passed, and the hypothesis input is zeroed out. This indicates whether the LM-based classifier uses the hypothesis or ignores it.

We observe that NoSimClassifier gets high accuracy, comparable to End2End. However, because the classifier is strong, it can ignore the hypothesis, and generated hypotheses are meaningless. This is also evident by the fact that zeroing out the hypothesis does not change performance.

All weakly-supervised generators reach roughly the same accuracy, and their performance dramatically drops in lieu of the hypothesis, showing that the LM-based classifier uses it. Top-$K=5$ ST performs best – almost the same as End2End. However, models differ in terms of how much they rely on the hypothesis (columns Q and $\Delta$). Performance drops by more than 13% for both Top-$K=3$ ST and Top-$K=5$ ST, indicating that the LM-based classifier strongly relies on the generated hypotheses. Table 4 shows some generated hypotheses, which are similar to those obtained when training with a similarity classifier only (§4.2.1).

Tables 8 and 9 (Appendix A) present examples where zeroing out the hypothesis input (column Q) creates or fixes an error for Top-$K=3$ ST. We observe how the hypothesis sways the prediction of the model, and that often even when an error is caused, the hypothesis is reasonable.

We evaluate SupGen, which was trained independently as a sequence-to-sequence model and is not optimized for QA. We experiment with decoding a hypothesis of length $|c|=3$ and also of maximal length $|c|=30$. Results show that indeed performance is 4 points lower than our best model, but the output text is more natural. Moreover, the LM-based classifier relies on it for its prediction, especially when we decode very long hypotheses $|c|=30$. To summarize, there is a trade-off between the two types of supervision, where training from the QA signal leads to higher accuracy, but text that is “list-like” and not natural (Table 4), while training to generate the answer yields lower performance, but more natural text (Table 6).

To better understand the quality of the generated text, we perform a human evaluation. We randomly sample 50 examples from the development set, and generate hypotheses from four models: (a) $|c|=3$+KLD+REP, (b) Top-$K=5$ ST, (c) SupGen $|c|=3$ and (d) SupGen $|c|=30$. We randomly shuffle question-hypotheses pairs from all models, show them to six graduate students, and ask them to rate whether a hypothesis is reasonable, somewhat reasonable, or not reasonable. Each hypothesis is rated by 3 students, and the score for a model is the average across raters and examples.

Table 3 shows the results of this experiment. Top-$K=5$ ST achieved the highest score of 0.74. While SupGen models produce more natural texts, they are judged to be less reasonable in the context of the question.

We examine whether our model can generalize to the QASC dataset, where general knowledge is needed. An End2End model trained on CSQA obtains 38.9 accuracy (Table 2), showing that a model trained on CSQA transfers reasonably well to QASC without fine-tuning. Our hypothesis-generating models also generalizes without fine-tuning to QASC, where Top-$K=5$ ST reaches the highest accuracy, 39.2, while also providing the hypotheses as explanation. Table 7 shows examples for generated hypotheses for QASC questions.

We evaluated Top-$K=5$ ST on the test set of CSQA and obtained 63.5 accuracy. As a point of reference, the leaderboard of CSQA reports one model that uses XLNet-large, which obtains 66.9 accuracy, but also uses external documents retrieved from Wikipedia with IR. DREAM entry on https://www.tau-nlp.org/csqa-leaderboard.

3 Explainability Analysis

We analyze how the textual hypotheses provide insights onto the abilities of the LM beyond what is possible with an end-to-end architecture. We analyze our joint model Top-$K=5$ ST, and present examples in Table 5.

First, we look at cases where both the similarity and LM-based classifiers answered correctly (49% of the cases). We manually annotated whether the hypotheses provide a reasonable answer for a random sample of 100 hypotheses from the development set, assigning 1 point to a reasonable hypothesis, 0.5 point to a somewhat reasonable hypothesis, and 0 points to an unreasonable hypothesis. The score was 0.94, indicating that when both classifiers are correct, the hypotheses are reasonable, and we can have confidence that the model does not “cheat”. In the few cases where the hypothesis is unreasonable, it reveals a shortcoming in the model’s knowledge. For example, for the question “The hostess was good at her job, she always had a smile when she would what?”, the model outputs the hypothesis “dinner eat serve food meals”, indicating that it does not distinguish a hostess from a waitress. However, the distractors are weak, and the model correctly chooses “welcome guests”.

Next, we look at cases where both classifiers were wrong (23% of the cases). Annotating 51 examples to estimate whether hypotheses are reasonable, we obtain a very low score of 0.21. By examining the hypotheses, we can understand the main reasons of error (examples in Table 5): (a) Missing knowledge: the question requires very specific knowledge that the model does not generate, suggesting that the knowledge is missing. (b) Semantic errors: the model ignores parts of the question and is misled by surface clues (ignoring negation in the first example, and a restrictive relative clause in the second). These two reasons cover 54% of the unreasonable hypotheses produced, showing how the hypotheses provide important information for “debugging” the model.

We also analyze cases where the LM-based classifier was correct but the similarity classifier was wrong (22% of the cases). Annotating 50 such examples produces a score of 0.55, showing that many of the hypotheses were actually reasonable. We observe that in two-thirds of these cases, the error is related to the inability of the generator to consider the distractors themselves, and while the hypothesis was reasonable, it was more similar to a distractor than to the gold answer (Table 5). Finally, transferring to QASC (Table 7) shows that the model performs well on general knowledge questions, which are somewhat similar to those in CSQA, but fails on more scientific questions that require encyclopedic knowledge.

We observe that joint training leads to a model that has comparable performance to an end-to-end model, but also provides a window to the information inside the LM. Specifically, using the hypotheses we can detect cases where the model was right, but lacks the necessary knowledge, cases where the model was wrong, but provides reasonable answers, and analyze different types of failures related to knowledge and language.

Discussion and Related Work

There has been substantial effort recently to collect human-generated explanations and train models to generate or select them. Rajani et al. 2019 presented the CoS-E dataset, where human-generated explanations to commonsense questions are used to train a LM. Wang et al. 2019 created a dataset for evaluating whether a model can choose the right explanation for its decision. Huang et al. 2019 crowd-sourced a multi-choice reading comprehension dataset, where answer candidates are human-generated explanations, and showed that a trained generative model can produce semantically consistent explanations. Sap et al. 2019 created a dataset for explaining social situations. All these approaches rely on human-generated explanations that are expensive to collect. Moreover, it is unclear whether generated explanations are actually used for predicting the correct answer.

There has been ample research recently on precisely defining what are explanations and what are their different facets (see Lipton 2016 and Wiegreffe and Pinter 2019 among others). Specifically, an important question is whether the explanation causes the prediction, that is, whether the model actually uses the explanation to reach its prediction. From this perspective, our similarity classifier provides hypotheses that strongly influence the prediction, as the question is not even passed to the classifier. Our LM-based classifier can potentially choose to ignore the hypothesis, but we show experimentally that ablating the hypothesis results in a performance drop. Thus, it is more similar to attention as an explanation Serrano and Smith 2019; Jain and Wallace 2019; Pruthi et al. 2019, where the attention structure does not necessarily reveal what tokens are used for prediction. Our supervised model is an even weaker form of explanation since it is trained independently from QA.

MC-QA has been a popular format for QA Lai et al. 2017; Zellers et al. 2018; Clark et al. 2018 mostly because it simplifies evaluation of free text answers. However, users in the real world ask questions in order to get an answer, and not to test the knowledge of a model or student Chen et al. 2019a. Thus, it is difficult to know whether models that succeed in MC-QA can actually answer the question, as they may take advantage of weaknesses in the way distractors were constructed. Our similarity classifier model can be viewed as an abstractive QA model that answers the question directly, and is evaluated with a multiple-choice format. In future work, our similarity classifier can be replaced by more complex untrained models that choose the right answer given the generated hypothesis.

Conclusion

In this work with propose a LM-based model for MC-QA, which generates natural language text that can be used to understand the the knowledge extracted from the LM, as an intermediate step. The performance of our model is on par with end-to-end models, while providing an inspectable layer for practitioners and users. Our approach is supervised from downstream application signal only, and thus can be generalized to any scenario where we would like to train a LM to generate text that is useful for a downstream application.

Acknowledgements

We thank Inbar Oren and Guy Tevet for their useful suggestions. This research was partially supported by The Israel Science Foundation grant 942/16, The Yandex Initiative for Machine Learning and the European Research Council (ERC) under the European Union Horizons 2020 research and innovation programme (grant ERC DELPHI 802800).

References

Appendix A Examples of the Impact of the Hypothesis on Prediction

Appendix B Hypernym Extraction

We report results on the synthetic hypernym extraction task with and without the Gumbel-softmax trick and the ST estimator. Table 10 shows the results. We observe that the ST estimator is crucial even on such a simple task, which aligns with prior observations Havrylov and Titov 2017 that ST helps overcome the discrepancy between training time and test time. GS improved results without ST, but had little effect with ST.