Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge

Introduction

An open problem in Artificial Intelligence is quantifying the extent to which algorithms exhibit intelligent behaviour (Levesque, 2014). In Machine Learning, a standard procedure consists in estimating the generalisation error, i.e. the prediction error over an independent test sample (Hastie et al., 2001). However, machine learning models can succeed simply by recognising patterns that happen to be predictive on instances in the test sample, while ignoring deeper phenomena (Rimell and Clark, 2009; Paperno et al., 2016).

Adversarial examples are inputs to machine learning models designed to cause the model to make a mistake (Szegedy et al., 2014; Goodfellow et al., 2014). In Natural Language Processing (NLP) and Machine Reading, generating adversarial examples can be really useful for understanding the shortcomings of NLP models (Jia and Liang, 2017; Kannan and Vinyals, 2017) and for regularisation (Minervini et al., 2017).

In this paper, we focus on the problem of generating adversarial examples for Natural Language Inference (NLI) models in order to gain insights about the inner workings of such systems, and regularising them. NLI, also referred to as Recognising Textual Entailment (Fyodorov et al., 2000; Condoravdi et al., 2003; Dagan et al., 2005), is a central problem in language understanding (Katz, 1972; Bos and Markert, 2005; van Benthem, 2008; MacCartney and Manning, 2009), and thus it is especially well suited to serve as a benchmark task for research in machine reading. In NLI, a model is presented with two sentences, a premise $p$ and a hypothesis $h$, and the goal is to determine whether $p$ semantically entails $h$.

The problem of acquiring large amounts of labelled data for NLI was addressed with the creation of the SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2017) datasets. In these processes, annotators were presented with a premise $p$ drawn from a corpus, and were required to generate three new sentences (hypotheses) based on $p$, according to the following criteria: a) Entailment – $h$ is definitely true given $p$ ($p$ entails $h$); b) Contradiction – $h$ is definitely not true given $p$ ($p$ contradicts $h$); and c) Neutral – $h$ might be true given $p$. Given a premise-hypothesis sentence pair $(p,h)$, a NLI model is asked to classify the relationship between $p$ and $h$ – i.e. either entailment, contradiction, or neutral. Solving NLI requires to fully capture the sentence meaning by handling complex linguistic phenomena like lexical entailment, quantification, co-reference, tense, belief, modality, and lexical and syntactic ambiguities (Williams et al., 2017).

In this work, we use adversarial examples for: a) identifying cases where models violate existing background knowledge, expressed in the form of logic rules, and b) training models that are robust to such violations.

The underlying idea in our work is that NLI models should adhere to a set of structural constraints that are intrinsic to the human reasoning process. For instance, contradiction is inherently symmetric: if a sentence $p$ contradicts a sentence $h$, then $h$ contradicts $p$ as well. Similarly, entailment is both reflexive and transitive. It is reflexive since a sentence $a$ is always entailed by (i.e. is true given) $a$. It is also transitive, since if $a$ is entailed by $b$, and $b$ is entailed by $c$, then $a$ is entailed by $c$ as well.

Consider three sentences $a$, $b$ and $c$ each describing a situation, such as: a) “The girl plays”, b) “The girl plays with a ball”, and c) “The girl plays with a red ball”. Note that if $a$ is entailed by $b$, and $b$ is entailed by $c$, then also $a$ is entailed by $c$. If a NLI model detects that $b$ entails $a$, $c$ entails $b$, but $c$ does not entail $a$, we know that it is making an error (since its results are inconsistent), even though we may not be aware of the sentences $a$, $b$, and $c$ and the true semantic relationships holding between them. ∎

Our adversarial examples are different from those used in other fields such as computer vision, where they typically consist in small, semantically invariant perturbations that result in drastic changes in the model predictions. In this paper, we propose a method for generating adversarial examples that cause a model to violate pre-existing background knowledge (Section 4), based on reducing the generation problem to a combinatorial optimisation problem. Furthermore, we outline a method for incorporating such background knowledge into models by means of an adversarial training procedure (Section 5).

Our results (Section 8) show that, even though the proposed adversarial training procedure does not sensibly improve accuracy on SNLI and MultiNLI, it yields significant relative improvement in accuracy (up to 79.6%) on adversarial datasets. Furthermore, we show that adversarial examples transfer across models, and that the proposed method allows training significantly more robust NLI models.

Background

In NLI, in particular on the Stanford Natural Language Inference (SNLI) (Bowman et al., 2015) and MultiNLI (Williams et al., 2017) datasets, neural NLI models – end-to-end differentiable models that can be trained via gradient-based optimisation – proved to be very successful, achieving state-of-the-art results (Rocktäschel et al., 2016; Parikh et al., 2016; Chen et al., 2017).

Given two sentences $a,b\in\mathcal{S}$, the goal of a NLI model is to identify the semantic relation between $a$ and $b$, which can be either entailment, contradiction, or neutral. For this reason, given an instance, neural NLI models compute the following conditional probability distribution over all three classes:

Several scoring functions have been proposed in the literature, such as the conditional Bidirectional LSTM (cBiLSTM) (Rocktäschel et al., 2016), the Decomposable Attention Model (DAM) (Parikh et al., 2016), and the Enhanced LSTM model (ESIM) (Chen et al., 2017). One desirable quality of the scoring function $\operatorname{score}_{\Theta}$ is that it should be differentiable with respect to the model parameters $\Theta$, which allows the neural NLI model to be trained from data via back-propagation.

Model Training.

where $\hat{y}_{i,k}=p_{\Theta}(y_{i}=k\mid x_{i})$ denotes the probability of class $k$ on the instance $x_{i}$ inferred by the neural NLI model as in Eq. 1.

In the following, we analyse the behaviour of neural NLI models by means of adversarial examples – inputs to machine learning models designed to cause the model to commit mistakes. In computer vision models, adversarial examples are created by adding a very small amount of noise to the input (Szegedy et al., 2014; Goodfellow et al., 2014): these perturbations do not change the semantics of the images, but they can drastically change the predictions of computer vision models. In our setting, we define an adversary whose goal is finding sets of NLI instances where the model fails to be consistent with available background knowledge, encoded in the form of First-Order Logic (FOL) rules. In the following sections, we define the corresponding optimisation problem, and propose an efficient solution.

Background Knowledge

For analysing the behaviour of NLI models, we verify whether they agree with the provided background knowledge, encoded by a set of FOL rules. Note that the three NLI classes – entailment, contradiction, and neutrality – can be seen as binary logic predicates, and we can define FOL formulas for describing the formal relationships that hold between them.

In Section 4 we propose a method to automatically generate sets of sentences that violate the rules outlined in Table 1 – effectively generating adversarial examples. Then, in Section 5 we show how we can leverage such adversarial examples by generating them on-the-fly during training and using them for regularising the model parameters, in an adversarial training regime.

Generating Adversarial Examples

In this section, we propose a method for efficiently generating adversarial examples for NLI models – i.e. examples that make the model violate the background knowledge outlined in Section 3.

We cast the problem of generating adversarial examples as an optimisation problem. In particular, we propose a continuous inconsistency loss that measures the degree to which a set of sentences causes a model to violate a rule.

where $a_{1}$ and $a_{2}$ are two clause atoms.

In this work, we cast the problem of generating adversarial examples as an optimisation problem: we search for the substitution set $S=\{X_{1}\mapsto s_{1},\ldots,X_{n}\mapsto s_{n}\}$ that maximises the inconsistency loss in Eq. 4, thus (maximally) violating the available background knowledge.

2 Constraining via Language Modelling

Maximising the inconsistency loss in Eq. 4 may not be sufficient for generating meaningful adversarial examples: they can lead neural NLI models to violate available background knowledge, but they may not be well-formed and meaningful.

For such a reason, in addition to maximising the inconsistency loss, we also constrain the perplexity of generated sentences by using a neural language model (Bengio et al., 2000). In this work, we use a LSTM (Hochreiter and Schmidhuber, 1997) neural language model $p_{\mathcal{L}}(w_{1},\ldots,w_{t})$ for generating low-perplexity adversarial examples.

3 Searching in a Discrete Space

As mentioned earlier in this section, we cast the problem of automatically generating adversarial examples – i.e. examples that cause NLI models to violate available background knowledge – as an optimisation problem. Specifically, we look for substitutions sets $S=\{X_{1}\mapsto s_{1},\ldots,X_{n}\mapsto s_{n}\}$ that jointly: a) maximise the inconsistency loss described in Eq. 4, and b) are composed by sentences with a low perplexity, as defined by the neural language model in Section 4.2.

The search objective can be formalised by the following optimisation problem:

where $\log p_{\mathcal{L}}(S)$ denotes the log-probability of the sentences in the substitution set $S$, and $\tau$ is a threshold on the perplexity of generated sentences.

For generating low-perplexity adversarial examples, we take inspiration from Guu et al. (2017) and generate the sentences by editing prototypes extracted from a corpus. Specifically, for searching substitution sets whose sentences jointly have a high probability and are highly adversarial, as measured the inconsistency loss in Eq. 4, we use the following procedure: a) we first sample sentences close to the data manifold (i.e. with a low perplexity), by either sampling from the training set or from the language model; b) we then make small variations to the sentences – analogous to adversarial images, which consist in small perturbations of training examples – so to optimise the objective in Eq. 5.

When editing prototypes, we consider the following perturbations: a) change one word in one of the input sentences; b) remove one parse sub-tree from one of the input sentences; c) insert one parse sub-tree from one sentence in the corpus in the parse tree of one of the input sentences.

Note that the generation process can easily lead to ungrammatical or implausible sentences; however, these will be likely to have a high perplexity according to the language model (Section 4.2), and thus they will be ruled out by the search algorithm.

Adversarial Regularisation

We now show one can use the adversarial examples to regularise the training process. We propose training NLI models by jointly: a) minimising the data loss (Eq. 2), and b) minimising the inconsistency loss (Eq. 4) on a set of generated adversarial examples (substitution sets).

More formally, for training, we jointly minimise the cross-entropy loss defined on the data $\mathcal{J}_{\mathcal{D}}(\Theta)$ and the inconsistency loss on a set of generated adversarial examples $\max_{S}\mathcal{J}_{\mathcal{I}}(S;\Theta)$, resulting in the following optimisation problem:

In Eq. 6, the regularisation term $\max_{S}\mathcal{J}_{\mathcal{I}}(S;\Theta)$ has the task of generating the adversarial substitution sets by maximising the inconsistency loss. Furthermore, the constraint $\log p_{\mathcal{L}}(S)\leq\tau$ ensures that the perplexity of generated sentences is lower than a threshold $\tau$. For this work, we used the $\max$ aggregation function. However, other functions can be used as well, such as the sum or mean of multiple inconsistency losses.

For minimising the regularised loss in Eq. 6, we alternate between two optimisation processes – generating the adversarial examples (Eq. 5) and minimising the regularised loss (Eq. 6). The algorithm is outlined in Algorithm 1. At each iteration, after generating a set of adversarial examples $S$, it computes the gradient of the regularised loss in Eq. 6, and updates the model parameters via a gradient descent step. On line 6, the algorithm generates a set of adversarial examples, each in the form of a substitution set $S$. On line 9, the algorithm computes the gradient of the adversarially regularised loss – a weighted combination of the data loss in Eq. 2 and the inconsistency loss in Eq. 4. The model parameters are finally updated on line 11 via a gradient descent step.

Creating Adversarial NLI Datasets

We crafted a series of datasets for assessing the robustness of the proposed regularisation method to adversarial examples. Starting from the SNLI test set, we proceeded as follows. We selected the $k$ instances in the SNLI test set that maximise the inconsistency loss in Eq. 4 with respect to the rules in $\mathbf{R_{1}}$, $\mathbf{R_{2}}$, $\mathbf{R_{3}}$, and $\mathbf{R_{4}}$ in Table 1. We refer to the generated datasets as $\mathcal{A}_{\text{m}}^{k}$, where $m$ identifies the model used for selecting the sentence pairs, and $k$ denotes number of examples in the dataset.

For generating each of the $\mathcal{A}_{\text{m}}^{k}$ datasets, we proceeded as follows. Let $\mathcal{D}=\{(x_{1},y_{i}),\ldots,(x_{n},y_{n})\}$ be a NLI dataset (such as SNLI), where each instance $x_{i}=(p_{i},h_{i})$ is a premise-hypothesis sentence pair, and $y_{i}$ denotes the relationship holding between $p_{i}$ and $h_{i}$. For each instance $x_{i}=(p_{i},h_{i})$, we consider two substitution sets: $S_{i}=\{X_{1}\mapsto p_{i},X_{2}\mapsto h_{i}\}$ and $S_{i}^{\prime}=\{X_{1}\mapsto h_{i},X_{2}\mapsto p_{i}\}$, each corresponding to a mapping from variables to sentences.

We compute the inconsistency score associated to each instance $x_{i}$ in the dataset $\mathcal{D}$ as $\mathcal{J}_{\mathcal{I}}(S_{i})+\mathcal{J}_{\mathcal{I}}(S_{i}^{\prime})$. Note that the inconsistency score only depends on the premise $p_{i}$ and hypothesis $h_{i}$ in each instance $x_{i}$, and it does not depend on its label $y_{i}$.

After computing the inconsistency scores for all sentence pairs in $\mathcal{D}$ using a model $m$, we select the $k$ instances with the highest inconsistency score, we create two instances $x_{i}=(p_{i},h_{i})$ and $\hat{x_{i}}=(h_{i},p_{i})$, and add both $(x_{i},y_{i})$ and $(\hat{x}_{i},\hat{y}_{i})$ to the dataset $\mathcal{A}_{\text{m}}^{k}$. Note that, while $y_{i}$ is already known from the dataset $\mathcal{D}$, $\hat{y}_{i}$ is unknown. For this reason, we find $\hat{y}_{i}$ by manual annotation.

Related Work

Adversarial examples are receiving a considerable attention in NLP; their usage, however, is considerably limited by the fact that semantically invariant input perturbations in NLP are difficult to identify (Buck et al., 2017).

Jia and Liang (2017) analyse the robustness of extractive question answering models on examples obtained by adding adversarially generated distracting text to SQuAD (Rajpurkar et al., 2016) dataset instances. Belinkov and Bisk (2017) also notice that character-level Machine Translation are overly sensitive to random character manipulations, such as typos. Hosseini et al. (2017) show that simple character-level modifications can drastically change the toxicity score of a text. Iyyer et al. (2018) proposes using paraphrasing for generating adversarial examples. Our model is fundamentally different in two ways: a) it does not need labelled data for generating adversarial examples – the inconsistency loss can be maximised by just making an NLI model produce inconsistent results, and b) it incorporates adversarial examples during the training process, with the aim of training more robust NLI models.

Adversarial examples are also used for assessing the robustness of computer vision models (Szegedy et al., 2014; Goodfellow et al., 2014; Nguyen et al., 2015), where they are created by adding a small amount of noise to the inputs that does not change the semantics of the images, but drastically changes the model predictions.

Experiments

We trained DAM, ESIM and cBiLSTM on the SNLI corpus using the hyperparameters provided in the respective papers. The results provided by such models on the SNLI and MultiNLI validation and tests sets are provided in Table 3. In the case of MultiNLI, the validation set was obtained by removing 10,000 instances from the training set (originally composed by 392,702 instances), and the test set consists in the matched validation set.

As a first experiment, we count the how likely our model is to violate rules $\mathbf{R_{1}},\mathbf{R_{2}},\mathbf{R_{3}},\mathbf{R_{4}}$ in Table 1.

In Table 4 we report the number sentence pairs in the SNLI training set where DAM, ESIM and cBiLSTM violate $\mathbf{R_{1}},\mathbf{R_{2}},\mathbf{R_{3}},\mathbf{R_{4}}$. In the $\left|{\mathbf{B}}\right|$ column we report the number of times the body of the rule holds, according to the model. In the $\left|{\mathbf{B}\land\lnot{\mathbf{H}}}\right|$ column we report the number of times where the body of the rule holds, but the head does not – which is clearly a violation of available rules.

We can see that, in the case of rule $\mathbf{R_{1}}$ (reflexivity of entailment), DAM and ESIM make a relatively low number of violations – namely 0.09 and 1.00 %, respectively. However, in the case of cBiLSTM, we can see that, each sentence $s\in\mathcal{S}$ in the SNLI training set, with a 23.76 % chance, $s$ does not entail itself – which violates our background knowledge.

With respect to $\mathbf{R_{2}}$ (symmetry of contradiction), we see that none of the models is completely consistent with the available background knowledge. Given a sentence pair $s_{1},s_{2}\in\mathcal{S}$ from the SNLI training set, if – according to the model – $s_{1}$ contradicts $s_{2}$, a significant number of times (between 9.84% and 46.17%) the same model also infers that $s_{2}$ does not contradict $s_{1}$. This phenomenon happens 16.70 % of times with DAM, 9.84 % of times with ESIM, and 46.17 % with cBiLSTM: this indicates that all considered models are prone to violating $\mathbf{R_{2}}$ in their predictions, with ESIM being the more robust.

In Section A.2 we report several examples of such violations in the SNLI training set. We select those that maximise the inconsistency loss described in Eq. 4, violating rules $\mathbf{R_{2}}$ and $\mathbf{R_{3}}$. We can notice that the presence of inconsistencies is often correlated with the length of the sentences. The model tends to detect entailment relationships between longer (i.e., possibly more specific) and shorter (i.e., possibly more general) sentences.

1 Generation of Adversarial Examples

In the following, we analyse the automatic generation of sets of adversarial examples that make the model violate the existing background knowledge. We search in the space of sentences by applying perturbations to sampled sentence pairs, using a language model for guiding the search process. The generation procedure is described in Section 4.

The procedure was especially effective in generating adversarial examples – a sample is shown in Table 6. We can notice that, even though DAM and ESIM achieve results close to human level performance on SNLI, they are likely to fail when faced with linguistic phenomena such as negation, hyponymy, and antonymy. Gururangan et al. (2018) recently showed that NLI datasets tend to suffer from annotation artefacts and limited linguistic variations: this allows NLI models to achieve nearly-human performance by capturing repetitive patterns and idiosyncrasies in a dataset, without being able of effectively capturing textual entailment. This is visible, for instance, in example 5 of Table 6, where the model fails to capture the hyponymy relation between “male” and “man”, incorrectly predicting an entailment in place of a neutral relationship. Furthermore, it is clear that models lack commonsense knowledge, such as the relation between “pushing” and “carrying” (example 1), and being outside and swimming (example 2). Generating such adversarial examples provides us with useful insights on the inner workings of neural NLI models, that can be leveraged for improving the robustness of state-of-the-art models.

2 Adversarial Regularisation

We evaluated whether our approach for integrating logical background knowledge via adversarial training (Section 5) is effective at reducing the number of background knowledge violations, without reducing the predictive accuracy of the model. We started with pre-trained DAM, ESIM, and cBiLSTM models, trained using the hyperparameters published in their respective papers.

After training, each model was then fine-tuned for 10 epochs, by minimising the adversarially regularised loss function introduced in Eq. 6. Table 3 shows results on the SNLI and MultiNLI development and test set, while Fig. 1 shows the number of violations for different values of $\lambda$, where regularised models are much more likely to make predictions that are consistent with the available background knowledge.

We can see that, despite the drastic reduction of background knowledge violations, the improvement may not be significant, supporting the idea that models achieving close-to-human performance on SNLI and MultiNLI may be capturing annotation artefacts and idiosyncrasies in such datasets (Gururangan et al., 2018).

We evaluated the proposed approach on 9 adversarial datasets $\mathcal{A}_{\text{m}}^{k}$, with $k\in\{100,500,1000\}$, generated following the procedure described in Section 6 – results are summarised in Table 5. We can see that the proposed adversarial training method significantly increases the accuracy on the adversarial test sets. For instance, consider $\mathcal{A}_{\text{DAM}}^{100}$: prior to regularising ($\lambda=0$), DAM achieves a very low accuracy on this dataset – i.e. $47.4\%$. By increasing the regularisation parameter $\lambda\in\{10^{-4},10^{-3},10^{-2},10^{-1}\}$, we noticed sensible accuracy increases, yielding relative accuracy improvements up to $75.8\%$ in the case of DAM, and $79.6\%$ in the case of cBiLSTM.

From Table 5 we can notice that adversarial examples transfer across different models: an unregularised model is likely to perform poorly also on adversarial datasets generated by using different models, with ESIM being the more robust model to adversarially generated examples.

Furthermore, we can see that regularised models are generally more robust to adversarial examples, even when those were generated using different model architectures. For instance we can see that, while cBiLSTM is vulnerable also to adversarial examples generated using DAM and ESIM, its adversarially regularised version cBiLSTMAR is generally more robust to any sort of adversarial examples.

Conclusions

In this paper, we investigated the problem of automatically generating adversarial examples that violate a set of given First-Order Logic constraints in NLI. We reduced the problem of identifying such adversarial examples to an optimisation problem, by maximising a continuous relaxation of the violation of such constraints, and by using a language model for generating linguistically-plausible examples. Furthermore, we proposed a method for adversarially regularising neural NLI models for incorporating background knowledge.

Our results showed that the proposed method consistently yields significant increases to the predictive accuracy on adversarially-crafted datasets – up to a 79.6% relative improvement – while drastically reducing the number of background knowledge violations. Furthermore, we showed that adversarial examples transfer across model architectures, and the proposed adversarial training procedure produces generally more robust models. The source code and data for reproducing our results is available online, at https://github.com/uclmr/adversarial-nli/.

We are immensely grateful to Jeff Mitchell, Johannes Welbl, Sameer Singh, and the whole UCL Machine Reading group for all useful discussions, inputs, and ideas. This work has been supported by an Allen Distinguished Investigator Award.

References

Appendix A Supplementary Material

In the following, we report the accuracy of DAM on several adversarial datasets $\mathcal{A}_{\text{m}}^{k}$, with $k=100$ and $m\in\{\textrm{DAM},$ $\textrm{ESIM},$ $\textrm{cBiLSTM}\}$. In the following, we report the accuracy of ESIM on several adversarial datasets $\mathcal{A}_{\text{m}}^{k}$. In the following, we report the accuracy of cBiLSTM on several adversarial datasets $\mathcal{A}_{\text{m}}^{k}$.

A.2 Adversarial examples

In Table 7 we report inconsistent results produced by DAM on the SNLI training set, which violate rules $\mathbf{R_{2}}$ and $\mathbf{R_{3}}$ outlined in Table 1. In Table 8, we report inconsistent results yield by DAM on examples generated using the procedure described in Section 4.3.

A.3 Background Knowledge Violations

In the following we report the number of violations (%) to rules in Table 1 made by DAM, ESIM, and cBiLSTM on the SNLI test set.

A.4 Optimisation algorithms

In Algorithm 2 we describe our algorithm for generating adversarial examples by perturbing sentences in a dataset, and using a language model for constraining the generation process. In Algorithm 3 we describe our adversarial training algorithm: it solves a minimax problem, where first a set of adversarial examples is generated by maximising the inconsistency loss $\mathcal{J}_{\mathcal{I}}$. Then, the model is trained by jointly minimising the data loss $\mathcal{J}_{\mathcal{D}}$ and inconsistency loss on the generated adversarial examples.