The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations

Introduction

The Second Workshop on Evaluating Vector Space Representations for NLP (RepEval 2017) features a shared task competition meant to evaluate natural language understanding models based on sentence encoders—that is, models that transform sentences into fixed-length vector representations and reason using those representations. Submitted systems are evaluated on the task of natural language inference (NLI, also known as recognizing textual entailment, or RTE) on the Multi-Genre NLI corpus (MultiNLI; Williams et al. 2017). Each example in the corpus consists of a pair of sentences, and systems must predict whether the relationship between the two sentences is entailment, neutral or contradiction in a balanced three-way classification setting.

We selected the task of NLI with the intent to evaluate as directly as possible the degree to which each model can extract and manipulate distributed representations of sentence meaning. In order for a system to perform well at natural language inference, it needs to handle nearly the full complexity of natural language understanding,Entailment notably does not require a system to ground its representations of sentence meaning to any outside representational system, for better or worse. For related discussion of entailment and natural language understanding see Chierchia and McConnell-Ginet (1991), Dagan et al. (2006), and MacCartney (2009). but its framing as a sentence-pair classification problem makes it suitable as an evaluation task for a broad range of models, and avoids issues of sequence generation, structured prediction, or memory access that can complicate evaluation in other settings.

The shared task includes two evaluations, a standard in-domain (matched) evaluation in which the training and test data are drawn from the same sources, and a cross-domain (mismatched) evaluation in which the training and test data differ substantially. This cross-domain evaluation tests the ability of submitted systems to learn representations of sentence meaning that capture broadly useful features.

This paper briefly introduces the task and dataset, presents the rules and results of the competition, and analyzes and compares the submitted systems. All the submitted systems are broadly similar, and incorporate bidirectional recurrent neural networks as a key component. We find that all systems performed fairly well, outperforming a simple bidirectional LSTM (BiLSTM; Hochreiter and Schmidhuber, 1997) baseline. To our surprise, no system performed dramatically worse on the mismatched evaluation than on the matched evaluation, and all systems performed reasonably consistently across examples representing a range of linguistic phenomena, suggesting that all were able to produce systems for semantic representation which, while not perfect, were effective and not tightly adapted to any particular style of language or set of constructions.

Dataset

MultiNLI (Williams et al., 2017) consists of 393k pairs of sentences from a broad range of genres of written and spoken English, balanced across three labels. Each premise sentence (the first sentence in each pair) is derived from one of ten sources of text, which constitute the ten genre sections of the corpus. Each hypothesis sentence and pair label was composed by a crowd worker in response to a premise. MultiNLI was designed and collected in the style of the Stanford NLI Corpus (SNLI; Bowman et al. 2015), but covers a broader range of styles of text, rather than the relatively homogeneous captions used in SNLI.

Testing and development sets are available for all genres, with 2000 examples per set per genre. Only five genres have accompanying training sets. So, for the matched development and test sets, models are tested on examples derived from the same sources as those in the training set, while for the mismatched sets, the text source is not represented in the training data.

Table 1 presents example sentences from the corpus and Table 2 presents some key statistics. For a detailed discussion of the corpus, refer to Williams et al. (2017).

Shared Task Competition

The purpose of the shard task is to evaluate techniques for training and using sentence encoders. To this end, we require that all models create fixed-length vectors for each sentence with no explicitly-imposed internal structure. Alignment strategies like attention that pass information between the two encoders handling the two input sentences in a pair are not allowed. Memory models that represent sentences as variable-length sets or sequences of vectors are also not permitted. While systems that use methods like attention and structured memory are effective for NLI (Rocktäschel et al., 2016; Wang and Jiang, 2016; Chen et al., 2017a; Williams et al., 2017, i.a.), much of the variation across models of this kind lies in the way that they explicitly or implicitly align related sentences, rather than the way that they extract representations for sentences. As a result, we expect that focusing our evaluation on a restricted subset of models will yield conclusions that are more generally applicable to work on natural language understanding than would have been the case otherwise.

We provide competitors with labeled training and development sets, and unlabeled test sets for which they must submit labels. The development sets are meant to be used for hyperparameter tuning and model selection, and training on the development sets is not allowed. We place no limits on the use of outside training data and resources except that they be publicly available. We specifically encourage the use of the SNLI training set. Multiple submissions from the same team are allowed, up to a limit of two per day during the two-week evaluation period. Individual participants (i.e., PIs) are permitted to join multiple teams within reason, but only when each team reflects a fully independent engineering effort and each team has a different lead developer.

Evaluation

Competitors had approximately ten weeks, starting with the release of the MultiNLI training and development sets, to develop their systems and two additional weeks—the evaluation period—to run their systems on the unlabeled test sets and submit results. The shared task evaluation was hosted through the Kaggle in Class platform using two competition pages—one each for the matchedhttps://inclass.kaggle.com/c/multinli-matched-evaluation and mismatchedhttps://inclass.kaggle.com/c/multinli-mismatched-evaluation sections of the corpus. The public leaderboard, which was displayed during the evaluation period, showed results on a random 25% of the test set labels, and the final results were computed by evaluating the two best systems from each competitor (chosen from the public leaderboard) on the remaining hidden 75% of the test set labels.

Results and Leaderboard

The competition results are shown in Table 3. All evaluated systems beat the BiLSTM baseline reported in Williams et al.. Furthermore, there is only a marginal gap between accuracy on matched and mismatched test sets for all systems.

The best performing single model is by Nie and Bansal, who achieve the best result on the matched competition and tie with Chen et al. in the mismatched competition. The Nie and Bansal model architecture uses stacked BiLSTMs with residual connections and, unlike the other high performing models, does not use within-sentence attention. The best performing system overall is an ensemble by Chen et al., which is based closely on the Enhanced Sequential Inference Model (ESIM; Chen et al., 2017a) but with attention only within each sentence, rather than between the two.

Looking toward the future, we also made available non-time-limited Kaggle in Class competition pagesMatched: https://inclass.kaggle.com/c/multinli-matched-open-evaluation Mismatched: https://inclass.kaggle.com/c/multinli-mismatched-open-evaluation to allow for further fair evaluations on the MultiNLI test sets. Note that since these evaluation sites report results on 100% of the test set, rather than the 75% used in the shared task, numbers reported on that site may differ slightly from those seen in the competition.

Model Comparison

All of the submitted systems are based on bidirectional LSTMs, but each system uses this core tool in a somewhat different way. This section surveys the key differences between systems, and the Model Details column in Table 3 serves as a summary reference for these differences.

Chen et al. and Nie and Bansal use three-layer bidirectional RNNs, while others only used single-layer RNNs. This likely contributes significantly to their good performance, as it is the most prominent feature shared only by these two top systems. They both use shortcut connections between recurrent layers to ease gradient flow, and Nie and Bansal find in an ablation study that using shortcut connections improves their performance by over 1% on both development sets.

Embeddings

Systems vary reasonably widely in their approach to input encoding. Yang et al. and Chen et al. use a combination of GloVe embeddings (Pennington et al., 2014, not fine tuned) and character-level convolutional neural networks (Kim et al., 2016) to extract representations of words. Balazs et al. also use pre-trained GloVe embeddings without fine tuning, but report (contra Chen et al.) that an added character-level feature extractor does not improve performance.

Vu et al. use pre-trained GloVe word embeddings augmented with additional feature vectors. They create embeddings for part-of-speech (POS), character level information, and the dependency relation between a word and its parent, and concatenate these with the embedding for each word. They find that this supplies a small but nontrivial improvement to their development set performance, especially in the mismatched setting.

Nie and Bansal use the simplest strategy, initializing embeddings with GloVe vectors and fine-tuning them.

Pooling

Vu et al. make a surprisingly effective change to the baseline BiLSTM model, motivated by Conneau et al.’s (2017) findings, by using max pooling rather than mean pooling when collecting the hidden states of the bidirectional LSTM for use as a sentence representation. They find that this yields an improvement of over 2.5% on both development sets.

While Vu et al. show that the choice of pooling strategy is quite important, Balazs et al. do not find a substantial effect in a similar comparison. This may be because Balazs et al.’s model also makes use of intra-sentence attention following the pooling layer, which dramatically reduces the importance of pooling.

Intra-Sentence Attention

Chen et al. and Balazs et al. both use attention over the BiLSTM states of each sentence to compute a final representation for that sentence. Chen et al. in particular uses a novel gated attention formulation, in which the BiLSTM gate values supply the attention weights over hidden states according to

where $g_{i}$ is the BiLSTM input gate and $h_{i}$ is the output from the BiLSTM encoder. They find that their use of gated attention helps performance somewhat relative to an unspecified baseline, though only in the matched setting.

Sentence Pair Classifier

Every system but Yang et al.’s uses elementwise product and difference features, comparing the two sentence encodings as part of the input to the classifier MLP that predicts the final relation label. In an ablation study, Chen et al. find this to be highly important, yielding more than a 3% gain in performance on both development sets.

Data and SNLI

We observe relatively little variation in the training data used in submitted systems. All systems are trained only on labeled NLI data—either the MultiNLI training set alone, or the MultiNLI and SNLI training sets combined. While Williams et al. find that the combined training set yields somewhat better results on the MultiNLI test set, Chen et al. nonetheless reaches state-of-the-art performance without using it.

Interim Discussion

We were particularly struck by the effectiveness of the max pooling strategy as a simple and highly effective improvement to the baseline BiLSTM sentence encoder. Less surprisingly, depth and intra-sentence attention appear to be broadly effective, and product and difference features appear to be valuable when using sentence encoders for the task of NLI. The results surrounding embeddings and input encoding were less clear, though Nie and Bansal’s use of pre-trained GloVe embeddings with fine tuning appears to be a simple and effective approach.

Error Analysis

In the interest of better understanding both the corpus and the submitted models, we annotate a 1,000-sample subset of the development set. We also provide a set of probe sentences and ask participating teams to submit vectors for all sentences in the probe set and test set. This section surveys our methods findings.

The annotated subset of the development set was released to competitors during the model development period, and consists of 1,000 examples each tagged with zero or more of the following labels. Labels were assigned manually except where clear keyword-spotting techniques sufficed.

conditional: Whether either sentence contains a conditional. Example: P: Laser-cutting equipment must be totally enclosed to be safe for human operators. H: Even if the laser machine is fully contained within, there still exist some amount of risk for the workers in the close proximity.

active/passive: Whether there is an active-to-passive (or vice versa) transformation from the premise to the hypothesis. Example: P: Hani Hanjour, Khalid Al Mihdhar, and Majed Moqed were flagged by capps. H: Capps never flagged anyone.

paraphrase: Whether the two sentences are close paraphrases. Example: P: Uh, lets see. H: Let us look.

coref: Whether the hypothesis contains a pronoun or referring expression that needs to be resolved using the premise. Example: P: You and I, gentle reader, are accredited members of the guild. H: We are recognised as members of the guild.

quantifier: Whether either sentence contains one of the following quantifiers: much, enough, more, most, less, least, no, none, some, any, many, few, several, almost, nearly. Example: P: We have provided an invoice to facilitate your gift. H: There’s no invoice available for your gift.

modal: Whether either sentence contains one of the following modal verbs: can, could, may, might, must, will, would, should. Example: P: Conversely, an increase in government saving adds to the supply of resources available for investment and may put downward pressure on interest rates. H: The amount of resources available for investment increases when government savings are increased.

belief: Whether either sentence contains one of the following belief verbs: know, believe, understand, doubt, think, suppose, recognize, recognize, forget, remember, imagine, mean, agree, disagree, deny, promise. Example: P: I trust that this is a fillip of propaganda and not a serious query. H: I believe this is to get attention and not a real inquiry.

negation: Whether either sentence contains negation. Example: P: On reflection, the parts will hold together. H: The parts will not hold together.

anto: Whether the two sentences contain an antonym pair. Example: P: As united 93 left Newark, the flight’s crew members were unaware of the hijacking of American 11. H: As the flight United 93 left Newark the crew members were fully aware of the hijacking of American 11.

tense_difference: Whether the two sentences use different tenses on any verbs. Example: P: Does she like what she does? H: Does she like what she is doing?

quantity/time_reasoning: Whether understanding the pair requires quantity or time reasoning. Example: P: The vice chairman joined the conference shortly before 10:00; the secretary, shortly before 10:30. H: The secretary joined before the vice chairman.

word_overlap: Whether the two sentences share more than 70% of their tokens. Example: P: Let’s look for paua shells! H: Let’s look for sticks.

long_sentence: Whether the premise or hypothesis is longer than 30 or 16 words respectively. Example: P: As invested with its dignity, since the seventeenth century just as the crown has been used for the monarch, or the oval office has come to stand for the president of the United States. H: Nobody in Britain associates the crown with the monarchy.

Table 4 shows model results on tagged examples for the BiLSTM baseline and for the three systems for which we were able to acquire example-by-example development set results (submission of these results was optional). Among those tags that are frequent enough to yield clearly interpretable numbers, none indicates a subset of the corpus that is dramatically harder or easier for the submitted models than is the corpus overall. This suggests that—as is typical with neural network models—these models do not rely strongly on any particular structural properties of the input texts to the exclusion of others.

We note that the submitted systems that use intra-attention (the three shown) do relatively well on the long_sentence and negation tags. This technique likely helps the encoders to recover the structures of long sentences and to correctly identify the scope of instances of negation. We also note that all systems do relatively poorly on the quantity/time_reasoning section, suggesting that these simple sentence feature extractors are not well situated to learn quantitative reasoning in this setting.

The competition participants were asked to submit sentence vectors for all the premise and hypothesis sentences in the test sets. We randomly sample 1,000 sentences from the matched test set and, using cosine similarity, calculate their pairwise distances against all sentences in the matched test set. Table 5 shows the percentage of times the first nearest neighbor belongs to the same genre as the chosen sentence. All models score fairly highly on this metric, suggesting that the learned representations are not genre-agnostic, despite their effectiveness in unseen genres. The models with higher percentage accuracy on the NLI task (see Table 3) show better performance on this metric as well, suggesting that this genre clustering property correlates with the overall quality of the metric space that each model uses to represent sentences.

The better models are also more interpretable. Table 6.1 shows example sentences and their three nearest neighbors for all models. It appears that entity identity is important for the Nie and Bansal model, though not it a way that is tied to syntactic position. For the Critics loved Merchant-Ivory example, we see matches to critics. In the Students love the rich culture example, we similarly see many matches to school and love. Since for each premise sentence in the MultiNLI corpus, there are 3 associated hypothesis sentences, it’s not surprising to see that the first nearest neighbor is often one of these associated sentences, like in the Critics example where the first nearest neighbor for all systems is the premise sentence. We found that for some examples, the better performing systems like Nie and Bansal’s had all three associated sentences as their top three nearest neighbors.

Probe Sentences

During the competition, we additionally provided a set of automatically generated probe sentences meant to aid error analysis. These probe sentences are produced to vary along dimensions relevant to probing for semantic role and negation information. We asked submitting teams to supply vectors for these sentences in addition to those in the test set. Figure 1 shows the cosine similarity between a subset of these sentence vectors rendered by Nie and Bansal’s (2017) system. We find that all systems (except that of Balazs et al., who did not submit these vectors) show similar behavior on these sentences, and we do not observe a clear correlation between behavior here and model performance. Perhaps unsurprisingly, we observe that sentences tend to be more similar to one another the more structural features they have in common. We observe this clearly for negation, identity of the subject, and tense, though continuous tenses are not reliably differentiated from others.

We find that BiLSTM-based models with max pooling or intra-sentence attention represent a popular and effective strategy for sentence encoding, and that systems based on this technique perform very well at the task of NLI.

We note that all submitted systems performed reasonably well across the many subsets of the data reflected by our supplementary tags, suggesting that none of these models exploit any particular narrow feature of the task or data to perform well. We also note that model performance does not vary much between the matched and mismatched sections of the test set. This means that submitted systems are likely capturing reasonably general strategies for extracting representations of meaning from text. As the systems get better, and fit the training data more closely, the disparity between matched and mismatched sets may appear. Both of these findings, though, bolster our expectation that the best of the submitted systems represent some of the best general-purpose architectures for sentence encoding available.

However, the task of NLI is far from being solved, and no submitted system approaches human performance, suggesting that there is ample room for further research on both the task and on the more general problem of sentence representation learning. Since many of the examples in MultiNLI require substantial commonsense background knowledge to solve fully, we suspect that the use of large outside datasets and resources (labeled or otherwise) will be crucial to making substantial further progress in this setting.

This work was made possible by a Google Faculty Research Award to Sam Bowman and Angeliki Lazaridou, and was also supported by a gift from Tencent Holdings. Allyson Ettinger contributed the supplementary probe sentences. We also thank George Dahl and the organizers of the RepEval 2016 and RepEval 2017 workshops for their help and advice.